My understanding is that the line which produced modern humans and the line which produced Neanderthal are said to have split many hundreds of thousands, if not millions, of years prior to the emergence of Neanderthal.
The evidence has suggested for several years that there is no Neanderthal DNA found in modern humans. This suggests that if there was breeding between Neanderthal and Humans, infertile hybrids resulted, if anything.
There would seem to be a problem with that idea in that you'd figure any common ancestor of both modern man and the neanderthal would be further removed from us than the neanderthal. If we can't be descended from neanderthals because the genetic gap is too great, which is what we hear, then how could we be descended from something even further removed?
Realize that until a few years ago, most scientists still thought modern humans and neaderthals were closely related. This has completely changed especially with genetic studies.