Besides compulsion there is also expulsion - which is what the New Testament seems to recommend.
You are correct for any specific organization. Those who disagree can leave and start their own group. And do. That’s largely the history of Protestantism.
The original poster seemed to be referencing maintaining a consistent body of doctrine for the whole society. And that obviously requires coercion.