The dirty little secret from the 2006 election was that it was mostley due to the homo Foley. We like to pretend that the big spending GOP was being punished...but if you look back at the articles and polls at the time (just before Foley) even with the war, even with the spending, there was talk of the GOP retaining control of both houses.
Don't forget about Bush's approach to immigration. They reluctantly voted to build a fence, and to this day it still isn't done. It isn't that conservatives punished the GOP by voting dim, they simply stayed home.