Time speed manipulation isn't out of the realm of possibility. Going really fast through time relative to someone else is a matter of nearing the speed of light or a very heavy planet. Actually going back in time is the real point of time travel stories. You can go forward by many means, including freezing yourself and unfreezing later. But the other direction is where things get difficult.
Actually, I think speed is the first problem, but accurate mapping would be the second, for going both forward and back. You’d have to know where the earth (or whatever other place you were trying to trvel to) would be in relation to the universe.