.
Where there are shear cliffs, there is no need to build a fence. Down stream, maybe, but not where there are cliffs. Also, the are vast dessert lands that nobody would dare attempt to cross.
Everywhere else, yes, put the wall.
And you man & patrol!
Hadrian’s Wall & The Great Wall failed not because they were bad ideas but because various imperial governments refused to man & patrol!