The term came from the southern leaders, John C. Calhoun and Alexander Stephens in particular, who used the term peculiar in terms of “our own” as in our own (southern) institution. It was not a reference to peculiar as meaning strange. The states that did not secede were considered to be southern states. West Virginia seceded from Virginia and stayed with the union.
The term peculiar references that they were alone in their sanctioning slavery...which they were not...so you consider Deleware and Maryland to be a southern states.