For the sake of this discussion I am going to call just use "Homo Sapien" to denote our species.
The evidence may be a bit misleading depending on how the DNA of the two species (if Neanderthal and Homo Sapiens are two species) interact. For instance the lion and the tiger have at least that half million years of evolution separating them and they can still successfully interbreed (although it doesn't happen in the wild). We could force interbreed Humans and chimps in the lab today and it wouldn't be that much of a "force" with current technology.
The question is did Humans and Neanderthals co-mingle. I think the Answer is probably yes. Humans are very promiscuous and even if it was just rape on one side or the other it most likely happened much more than once in the thousands of years of Human Neanderthal interaction.
Did these co-minglings result in children. I think we can say we don't know. Your assertion that the children must have grown up in Human communities is a bit misleading. If mixed children were being born into Neanderthal society they probably developed at least some Homo Sapien cognitive ability and so the Neanderthal tribe would quickly begin to leave cultural artifacts which resemble Homo Spain artifacts which did in fact happen when the two cultures lived near each other. It is possible that Neanderthal tribes died out not as a result of increased competition but as a result that their children quickly stopped being Neanderthals if 1/2 breed children were growing up in their midst and also seeking mates from nearby Homo Sapien populations. In fact the change over could be so rapid to leave very little in the way of fossil remains of the population that is interbred. (In fact some bones that have been found suggest half breeds did exist but they are still debatable)
Nothing can be said for sure at this point and you seem very certain and that is just not good science with the available data.
I am not 100% certain, to be sure. When I discuss this topic this is what's in the back of my head: Gosh, I'll look really silly if tomorrow they unearth strong evidence that Neanderthals and Cro-Magnon interbred.
But, yes, my personal evaluation of the evidence leaves me extremely confident that Cro-Magnons and Neanderthal did not interbreed, at least to any detectable extent. For one, there are so many lines of argument against such admixture (genetic barriers, cultural barriers, language barriers, physical barriers, etc). If you do knock down one then you run into the next and then knock down that one and run into another, etc., etc.
But, more importantly, if Cro-Magnon and Neanderthal could interbreed, and if they were at all inclined to do so, then I think the evidence would be patently clear and undeniable. For one, you should have a stark genetic imprint that distinguished European populations from more distant populations. That's what I think; you don't have to agree with me.
From a scientific standpoint, I don't think that this would even be a subject for debate if not for so many people wanting so badly for it to be true. So, they grasp at the flimsiest strands of alleged 'evidence' and combine that with a tangled web of conjecture and assumptions masquerading as fact in order to fabricate the scenario they wish to be true.
But sure, I could be wrong. If the evidence ever proves me wrong I'm sure that I'll be appropriately red-faced afterward..
For all we know Neanderthals were not promiscuous at all and only had sex when the female was 'in heat' as do all extant primates except humans and bonobos. Moreover, for all we know the impulse to "rape" was inconceivable to a Neanderthal. ]
Do you know? No. You are just typically imagining that Neanderthals are nothing but humans with a brow ridge.