Objectively, did pastor leave and a group follow?
In my fictional account: The Holy Spirit left with the Pastor and a group. I am an elder in the group.
If you are talking church building that is one of the idols the pharisee worship.