The Constitution does not require tariffs to affect people equally.
You are absolutely correct in that assessment. It only requires that indirect taxes (i.e. taxes applied to activities or events) be applied without any regard to location.