And there you have a real scientific dilemma. For either the science of DNA is wrong. Or the theory of human organic evolution is wrong. Or both.
The Neanderthal is the only very tenuous missing link between man and lower level primates. Cromagnon man and others don't even come close to the tenuous level of the Neanderthal.
There are people who imagine earlier humans were tied to the land and didn't move around much. At the same time a few thousand Neanderthals could pretty much cover the entire landmass of Europe without much effort. They were pretty dangerous themselves and could kill, munch and eat their way across about any space they cared to.
Genes get passed in transit ~ that's the way it is. To think it didn't happen in the good old days is simply not realistic.