I think he probably dislikes Romney because he knows Mitt has had many accomplishments in his life. Except being elected what has Obama ever done? Same goes for Clinton who spent his entire life in politics.
'You think you're so smart. There are a lot of smart people.' Sounds like the Obama 'you didn't build that' may have been inspired in part by his opponent. Obama doesn't like the comparison of the highly accomplished Romney and his resume to Obama's paper thin history.