It's necessary to recover quickly after disasters occur. However, the workers can't work in seriously damaged area because there is a possibility of secondary disaster. Therefore, the remote controlled construction machines are used to restore the damaged area. But now, the operator of these machines needs to be helped by a person who operates camera. The camera is manually operated by the operator. So, there is a problem that some persons are assigned to the camera operators and a difference in work efficiency is caused depending on their skill level. In order to solve this problem, we are developing an automatic tracking camera control system for construction machine operators by using image processing. It works well like manual camera operation. However, the reproduction of the skilled camera operation is still not enough. Therefore, we aimed to improve the system. In this paper, to improve the system, we recognized the moving direction of the construction machine from the image by image processing. In addition, using this recognition results, we improved the system to display more suitable image to machine operator according to the recognized moving direction.