When there is no tangible, moral consequence for leaving the church (aka the Body of Christ and Biblical doctrine), then why not leave? Of course what one does not see until much time passes is the spiritual consequences of the separation.
The church must unapologetically construct itself on the Gospel and sound doctrine. If the result of this is a smaller church, than so be it. We must never allow ourselves to redefine God based on who we wish Him to be. That is nothing more than self-worship.
What is funny is that so many people who say this have their own opinion of who God is and how we should worship Him and think anyone who doesn't follow suit is looking for ear-tickling and no-consequences sin.