You have issues of different pollsters making different assumptions about who to poll, when they are polling, where they are polling, and so forth. They may all poll the same basic spots but diverge in only a small fraction of their samples. The extra size then wouldn't give you greater confidence, it just would mean you have duplicated a lot of your polling. To add subsets together, they all must be done in exactly the same way at the same time and according to the same statistical plan that is designed to sample the entire diversity of the pool. That isn't the case when you average in, say, a Zogby poll and a Newsweek poll.
It's better to be ahead than behind, even within the margin of error, in all these polls. The 2000 election showed just how off the herd of pollsters can be, though.