I find it helpful when faced with that argument to quote the Mississippi Declaration of Causes of Secession:
Our position is thoroughly identified with the institution of slavery-- the greatest material interest of the world. Its labor supplies the product which constitutes by far the largest and most important portions of commerce of the earth.
These products are peculiar to the climate verging on the tropical regions, and by an imperious law of nature, none but the black race can bear exposure to the tropical sun. These products have become necessities of the world, and a blow at slavery is a blow at commerce and civilization.
If the war was not about slavery, I wonder why the south didn’t make a treaty with England to sell cotton to them exclusively and free the slaves in exchange for military assistance.
That is the danger of quoting just one or two favorite, cherry-picked documents. If you look at South Carolina's call to the other Southern States, or the Texas Declaration of Causes, you get a much wider picture and a much longer list of grievances. South Carolina remembered the Tariff of Abominations and the Nullification Crisis of 30 years earlier, and Texas had border-security issues with Congress, which had refused to appropriate moneys for frontier defense in the face of Comanche raids and Mexican border raids.
It wasn't about just one thing; but if there were "one thing" that caused the Southern States to bail, especially the States of the "upper South" (Tennessee, Virginia, North Carolina), it was Northern hostility to the South across a range of issues, coupled with impending Northern preclusive control of the U.S. Government.