Yes, very widely known. The affair between his mother, a woman of high society, and her French tutor was a big scandal, which greatly hindered his early social status. I can't find any instance of his rivals, some of the greatest legal minds of the day, using this to claim he wasn't Constitutionally eligible due to his father's status.
If he wasn’t born in country, to two citizen parents...he wan’t eligible then, nor now per the intent and understanding of the men who wrote the requirement into the Constitution.