When the priest is facing ad orientem, towards the East (from which Christ will come on the Last Day), he is leading his people towards the Lord. Moreover, when he offers the Holy Sacrifice, he is facing the Lord. All this symbolism is destroyed when you turn the altar around - the priest is no longer even facing the crucifix.
The practical difficulty is that the priest is no longer the leader of his people, he is 'on stage' facing an 'audience'. Instead of his personality being submerged in the common worship, he becomes the center of attention and feels the need to interact with his 'audience', becoming a performer or a sort of emcee instead of a priest, instead of concentrating on what he is doing.
There really is a big and observable change between ad orientem and versus populum.
Sounds to me like you’re reading attitudes you might have into other peoples minds.
Underscoring your point, the ad orientem posture is sometimes referred to as versus Dominum!