Why don't you possess me, too?

As has been mentioned before, Ryoutarou is a Singularity Point. What this means is that he can exist outside of time and he will not change it even when he goes into the past. This is very important, because otherwise that movie where he hung out with his ten-year old self in the Edo period? Would…