Visual surveillance of dynamic objects on roads has been developed to ensure road safety for people. Particularly, vehicle tracking is considered as a key technology for the road safety; studies on multi-object tracking (MOT) are being actively pursued. However, when MOT is performed, raw vision data are not always available because of the technical limitation or the privacy concern of the system; MOT needs to be performed only using the coordinates obtained from the object detector without using features extracted from raw image data such as color of vehicles, which degrades the accuracy of MOT to the unsatisfactory level for road safety. This paper proposes an MOT scheme for moving vehicles that is inspired by cell tracking using the Viterbi algorithm. The proposed scheme extends the Brownian motion model, which was used in the base scheme of cell tracking, by weighting probability transitions in accordance with the direction of travel of vehicles on the road. We evaluate the proposed scheme using simulated vehicle-traffic data and verify that the proposed scheme performs better than benchmark schemes in terms of the accuracy of MOT. We also demonstrate an example of how the proposed scheme works well for real vehicle-traffic data.