Many churches - and this is true of Baptist churches as well - are just social clubs for the religous-minded. My Baptist pastor would say the same thing. Churches too often become social clubs instead of the Body of Christ.
And a social club can be easily replaced.
I don't see it any other way. A true church is a society. Leaving a church is choosing a different principle of association (e.g., commiseration at the local bar). The Body of Christ is a specific social group, or else you could exercise your belief by yourself.