Think not in terms of physical distance, but in terms of time.
We can't look back in time at nearby objects - the light we see comes from a short time ago.
Faraway objects (in any direction) are farther back in time, due to the finite speed of light.
The farthest you could possibly see would also be the oldest.
So, really some of the light from what appears to be stars has taken so long to reach my eyes that some of those stars have already burned out? So, perhaps what is going on in the farthest reaches of the universe (and closer) is something else entirely than we think because the light from those current celestial bodies hasn't had time to reach my eyes yet? Perhaps much more has burned out and come to an end than we might think.