My original question was to see if there was an explination in the Lost Tales or JRRT Letters. My theory is that they lost so many Elves (including Gil-galad) that, like Arnor, only worse, the Elf Kingdom could no longer sustain itself. The few survivors that didn't bug out to the West would have settled with Elrond or Cirdan. Lorien was home of the Silvan Elves and they 'imported' Galadriel (Noldor) and Celeborn (Sindar) to rule after Amroth was lost at sea.
The Elves did lose a lot of people, and then a lot more went home. I think after a while they missed Valinor, a lot of them. The ones who are left by LotR are mostly those who really love Middle-Earth. And Galadriel who's stubborn.