They were out-competed. Genetic diversity is no guarrantee of evolutionary success.
The best comparison here...consider the neandertal like Russia in the 1930s/40s/50s/60s...almost that they developed or manufactured...came from designs or origins in Europe or the US. The Russians had no independent thinking or creativity. They were smart enough to copy but not surge ahead. The humans were able to develop, conquer, and surge ahead because of intense disire and ability to comprehend...while Norman the Neandertal sat there and tried to copy and found himself out-manuvered on ever plateau.
Add to this whole scenario....Norman was in a hostile environment...didn't readily adapt, or farm, or design hunting parties for maximum goal success. Norman was just out to hunt for the day and try to have enough to feed his clan in the cave. If bad times came along...Norman didn't readily pack up and move...which the humans did learn and adapt to. Norman slowly watched half his clan die of starvation...and then got the idea. Later...Norman was slow to realize measures to beat extreme cold...and a few of his clan died there. Like a hard drive with a 90 percent load and no defrag in four years...Norman simply could get ahead, and slowly lost all of his clan.
Henry the human got smart...went onto follow the game and movement of the buffalo or deer. Henry and his group learned how to farm...build transportable tents...develop fishing techniques...and vairous herbal remedies that saved members of his clan. Henry was able to get half of his clan up to the age 40 or more...simply by smart planning.
There is likely some point...in Europe...50-100,000 years ago, where the last of the Neandertals were standing there...no further generations...and no future. And the humans...were multiplying and doubling in every other generation. Simple statistics and strategies of today...easily explain the path of destruction and the path of success.