That's pretty weak justification, if you ask me. If I own a couple of acres, and a corner of it is rather desolate and unused, does someone have the right to move in and build a house on it just because I'm not using it at the moment? After all, it's still my land, right?
Show us any literature or culture from inhabitants prior to the Jews 400 B.C.