They apparently DID NOT sit on their thumbs for 20 years, and claims that they did are bogus. If they filed an ejectment or quiet title action against the judge, then I'll change my tune. Otherwise, given that the judge won his quiet title action, it seems to me that they sat on their thumbs.
What you people who are claiming that this is legal are in effect saying is that any American citizen who buys a piece of property as an investment, and wants to just sit on it, has no right to do so if someone else has other ideas for the property. And anyone purchasing land like that had better spend several years becoming an expert in real estate law before making the purchase.
This is just another example of where the modern American "legal" system has turned individual rights, and common sense, on their heads.
Get real!