We have been trying to develop Assistive MObile robot System (AMOS). The aim of AMOS is to replace only simple assistive work that picking up, transporting and placing the daily use objects in designated indoor location semi autonomously to reduce the burden of the care taker. AMOS should avoid any kind of collisions in the environment Therefore, the amount of sensors and functionalities are increased. In addition, the system is becoming huge and complicated. To control the AMOS robustly, we invented the concept of Shared Responsibility and the control method. The concept is applied to AMOS. Within the concept, the functionality of AMOS can be divided to four, Vision Based User Interface, Navigation System, Object Localization and Grasping, and Software Framework And finally all of four elements are developed.