Goods bound for the American market as a whole could have gone to any port when tariffs were equal. Lower tariffs would draw off trade. This had been illustrated for years in New Orleans where the Baratarians ran a thriving smuggling business. The long border between a coastal Confederacy and its US neighbors would have been hard to police.
You can hardly compare the relatively small amount of goods smuggled in by Jean Lafitte with they hundreds of millions of dollars worth of goods that the North imported every year.