However, as a US citizen I disagree that Congress has the authority to remove my 4th Amendment rights simply because I’m passing through said port. My rights are inherent in my person and not subject to my geographic location.
Your passport and your rights apply to your person at a border crossing. Everything else, including the underwear you are wearing, is what you are importing, and all imports are subject to inspection or seizure.
This is a common law concept and secure in our borders is the constitutional reference to this common law.
Or to flip your argument around, do you think that packages that are shipped to the United States should be exempt from customs inspection if they are going to a US citizen absent a warrant enabling the inspection? The only difference is your proximity to the property being imported.