If the species adapts in the wrong direction, we should find evidence of it in the fossil records. We don't. All we find is evidence of species that adapted in the right direction.
Well, species that did not adapt have indeed gone extinct. That's the fate of 99% of all species ever to have lived and they comprises most of the fossil record.
So you are factually wrong. The remains of failed species are commonplace.
And, once again, I ask you to show me an example.
Can you point out a species that exists today via adaptation whose lineage can be traced back to fossils, some of which that adapted in the wrong direction?