This paper introduces a controller design methodology on person following omni-directional vehicle robots. The robot that we use as a platform is a mecanum wheeled omni-directional vehicle. The system mainly consists of the vehicle and stereo vision sensor. The stereo vision sensor is mounted on the top of the vehicle. The stereo vision sensor is utilized in order to recognize position of target point as an alternative of a person and measure the relative position from the vehicle to target point. Furthuremore, in order to perform the presented controller in real-time control system, we build the controllers on RT-Linux operating system. This paper describes a simple structure of the control system and shows experimental results.