Right you are. Most sat on the sidelines, and many were loyalists. (British sympathizers)
" ... Loyalists were numerous and included small farmers as well as large landowners, royal officeholders, and members of the professions; they were to be found in varying strength in every colony. A large part of the population was more or less neutral, swaying to this side or that or else remaining inert in the struggle, which was to some extent a civil war. So it was to remain to the end ...
From here
ping