A simple reading of the Constitution itself would prove just that.
The Founders put the service or Labor clause specifically under the authority of the States in the Fourth Article, not under the powers of Congress in the first Article.
The ninth and tenth amendments were supposed to further insure it was up to the States whether to have slavery or not.
Attempting to 'end slavery' by any branch of government was, in itself, a treasonous act.
You are exactly right, and the Congress and President at the time of secession knew that too.
This might reflect media opinion at the time:
Washington March 3, 1861-——Tomorrow President James Buchanan
leaves office having served eight years as the fifteenth Chief Executive of the
United States. Although the secession came during his administration,
despite the withdrawl.of seven southern states, the remaining 18 are intact.
He leaves the United States Constitution residing undisturbed in the capital
in Washington. The House of Representatives and Senate will reconvene as
scheduled in five months. Chief Justice Roger B. Taney will preside over the
next term of the Supreme Court.
The Secretary of War, Joseph Holt reports no military activity in any areas
adjacent to the mason Dixon line. Army Commander General Winfield Scott
reports that the military is billeted as normal.
New York and Boston shipping houses are noting declines in imports landing
at their wharfs, but report that dry goods, machinery, passenger and mail
service with southern ports continues as usual.
With one exception, all federal facilities have been peacefully evacuated with
federal employees, their health intact, returning north by rail or ship.
Confederate government representatives are meeting in Washington with
officials to arrange for payment for these facilities and other debts resulting
from their withdrawal from the Union.
Last November, President Buchanan and Attorney General Black announced
that it was not within the authority of the government to institute military force to
address secession. He is to be congratulated for his respect for the constitution
and the peaceful lives of citizens everywhere.
Secession has removed slavery as an issue for either the courts or the territories,
and no longer preoccupies the interests of the legislature or press.
The following quote is from Mr. Buchanan's most recent address to Congress and demonstrates his respect for the country's future:
“Self-preservation is the first law of nature, and has been
implanted in the heart of man by his Creator for the wisest purpose;
and no political union, however fraught with blessings and benefits
in all other respects, can long continue if the necessary consequence
be to render the homes and the firesides of nearly half the parties to
it habitually and hopelessly insecure."
His commitment is manifest in the armistice arranged for Charleston and Pensacola.
It is now the responsibility of the next executive to maintain this peace.