Ah, but there is a Principle of physics that argues against that. It's called the Grandfather Paradox. It states that if you could go back and change the past, then you could kill your grandfather. Then how could you exist in the first place? A paradox arises. Therefore, you cannot change the past.
The Principle of Self-Consistency was developed to allow backwards time-travel without violating the Grandfather Paradox.
By the way, all of this is not pure silliness (although it sure does sound like it). It follows from Einstein's Theory of General Relativity.
“then you could kill your grandfather. Then how could you exist in the first place? “
Grandmaw would know who the real father was. She’s the one you can’t kill!