This study investigated whether normal adults were able to discriminate agency from the perceived traces or trajectories of past actions such as handwritings. Subjects wrote two types of component parts of Chinese characters, either mastered and unmastered, and were later shown various handwritten strokes and judged whether each of them had been written by themselves or by someone else. We found that people tended to answer that the handwritings had been written by others when they saw unmastered types of strokes, while they tended to answer that the handwritings had been written by themselves when they saw mastered types of strokes. This finding suggests a tight interplay among perception, self-consciousness, and memorized action in the motor system and adds to our knowledge about a higher order representation level in the agency recognition. Possible cognitive neuroscientific implications and engineering applications of the finding are also discussed.