Many is not the same as most. The fact remains that most of the officers and men serving in the US military who were from the South chose to join the confederacy. This of course included Robert E. Lee, Stonewall Jackson, and others. Lee had served alongside many who were to become stars of the Union firmament. People he considered his friends. His loyalty was to Virginia first. To Virginia, not to slavery or even to Jefferson Davis (who fought in the Mexican War along with Lee, Grant, Jackson, and even Sherman). Like I said it was a different time.
BTW, the very fact that Washington made that speech indicates that he knew many felt strong loyalties to their state, stronger in many cases than those to the United States. Ours was a system of divided responsibilities.
Meant to add, before I got called away because the someone ran into my car with a rental truck, that these divided responsibilities did not have to lead to conflicting loyalties, if each level of government had stayed in it's own area of responsibility. The northern states had mostly eliminated slavery, such as was practiced there in the first place. Given the normal political process, and the fact that slavery had begun to outlive it's usefulness with the coming mechanization of agriculture. Slaves were expensive to buy and more expensive to maintain, as opposed to Irishmen and Chinese who were cheap and pretty much disposable. Each state could have found it's own way to ending the institution, and I believe would have in time. But of course the abolitionist do-gooders couldn't wait, they'd rather kill and destroy for their vision of justice. Or more properly, get others to kill and destroy for them. Very few, although some, Union soldiers cared much one way or the other about slavery, just as long as the Negroes stayed in the South, and didn't come north to compete for the industrial jobs, something which didn't much happen until WW-II. In so doing, they created a backlash, the effects of which are still being felt to one degree or another.
The situation was somewhat analogous to the liberals achieving their aims through the courts, rather than through the normal political process, with elections and all that messy stuff. But in this case it was having the federal government do the work that should have been done at the state level under the Constitution as it then existed.
And he made it clear just how wrong those opinions were.