An active stereo vision system based on a model of neural pathways of human binocular motor system is proposed. With this model, it is guaranteed that the two cameras of the active stereo vision system can keep their lines of sight fixed on the same target object during smooth pursuit. This feature is very important for active stereo vision systems, since not only 3D reconstruction needs the two cameras have an overlapping field of vision, but also it can facilitate the 3D reconstruction algorithm. To evaluate the effectiveness of the proposed method, some software simulations are done to demonstrate the same target tracking characteristic in a virtual environment apt to mistracking easily. Here, mistracking means two eyes track two different objects separately. Then the proposed method is implemented in our active stereo vision system to perform real tracking task in a laboratory scene where several persons walk self-determining. Before the proposed model is implemented in the system, mistracking occurred frequently. After it is enabled, mistracking never occurred. The result shows that the vision system based on neural pathways of human binocular motor system can reliably avoid mistracking.