Here is a photo of Fort Delaware, built from 1836 to 1859 downstream of Philadelphia. Not exactly puny.
Construction began in 1825 on Fort Hamilton in New York. Robert E. Lee beefed up the batteries there in the 1840s. In fact, the biggest muzzle loading cannon ever cast in the US was placed at this fort. I guess it must have been bigger than the Swamp Angel used by the Union on Charleston civilians.
Construction began in 1829 on Fort Sumter. Still wasn't finished when the war started. Fort Warren in Boston was built between 1833 and 1869.