Voters want to feel that office-holders are conscientious about fulfilling their responsibilities to all citizens.
A candidate who appears to be drawing distinctions between classes of citizens based on some criterion of worth is going to lose votes.
Of course this, was a fake controversy. I don't believe Romney was any less conscientious or dutiful than other candidates or office-holders with respect to poorer or humbler citizens, but it's a matter of appearances, and Romney lost that battle.
When is the last time that happened?
I actually think the voters have a way of sniffing out a sense of who a candidate is as a person, even when they don't understand policy. Romney saying he "wasn't concerned about the very poor because they have a safety net" may have been a gaffe, but it was more in the vein of a Freudian slip. Romney was out-of-touch with the average American and it showed in so many things that he said and did.
This is exactly why he was such a heavy promoter of the welfare state in Taxachusetts. He doesn't relate to poorer people, thinks they're inherently disadvantaged as compared to him, and therefore thinks they need handouts. He consistently gave the impression he didn't believe in the class mobility inherent in the American dream. He was Democrat lite and a poor standard-bearer for the Republican party from day one.
Now, it's true that Obama believes the same things, even more passionately. But Obama didn't try to hide it, he ran on it. Romney was insincere and uneasy as he tried to balance his own liberal record on economics with what he thought the conservative base wanted to hear (such as that he was "severely conservative"). People trusted Obama more because Obama was actually being more honest than Romney about who he was. Swing voters don't understand the issues but they can sniff out the candidates' personal qualities, so they vote on those.
Again, Romney was an obviously bad candidate from the get-go. No intelligent, informed, conservative Republican had any excuse for nominating him.