Absolutely. But there it is. Hoover won that battle 50 years ago. I don’t think he even used the commerce clause, he just managed to hammer through that if the crime “happened” in more than one state it must be federal. His primary excuse was gunning for the interstate flight guys, rob somebody in a town on one side of the border and drive to the other side so the state cops in the original state can’t chase you. But of course in the end it was Hoover, he just wanted power, and he had the blackmail ammo to get it.
Hard to imagine Jefferson, Madison, Monroe, etc,,arguing that the FEDERAL government should become involved in a quibble arising out of a man’s affair or visit to a prostitute.