I think you are missing the essential point, which has to do with how the rights of the individual rightist are to be protected from the mass of humanity. Those who sat on the Right in the French Assembly wanted to be safe from the mob. So too did the Founding Fathers--although they might not have been so snobbish about it--but that is almost precisely what Madison's comments on Democracy were about.
The Great movements of the Left, Communism, National Socialism, and various other varieties of Socialism, all embrace the mob--the power of numbers.
Again, see Political Spectrum, to illustrate my argument.
Historical Origin of the Terms
The usage in Western politics of "right" and "left" to refer to political affiliation stems at least from the French National Assembly in 1789, during the French Revolution. There, the Second Estate, or nobility, sat to the right of the chamber, and the Third Estate, or common people (at the time the radicals) to the left. Thus, "right" generally meant conservative, upholding the existing social or political order, and "left" meant radical, attempting to change or overthrow the existing order. The usage may actually be earlier, from the pre-Revolutionary Estates-General, where right and left referred to supporters and opponents of the monarchy.