He didn't say men were smarter than women, only that men had larger brains.
Don't turn this into an uproar like the one that occurred after a D.C. official said the word "niggardly", which means stingy, and had to resign because some idiots thought it had something to do with the N-word.
Hey, it is all a tongue-in-cheek JOKE. Relax. The election will be over in a few days (or weeks, depending on the lawyers).