Do you even know in what context Madison made that statement?
Sure I do, he was vouching for the character of Mr. Smith. Look it up for your self.
http://press-pubs.uchicago.edu/founders/documents/a1_2_2s6.html