In a sense he was right. The constitution is not self-executing. Moreover, I sympathize with the objection that there is no need to forbid the government from doing something that it was granted no power to do in the first place.
Nevertheless, the Bill of Rights exists, and it's up to us to make the most of it.
Madison came to see the senate of the states, vertical division of power as a far better defender of freedom than bills of rights. He was right. The 17th Amendment was an incredible mistake.