Historically speaking voters tend to reelect THEIR incumbent congressman. While they might blame or think poorly of the performance of one particular party they tend to think, “Yeah but our guy's okay”. In other words when it comes time to vote there is nobody named “Joe Generic” on the ballot.
Obviously this is not to say that incumbents never lose. Nor am I saying that the poll results are not good for what they are worth. It's just that they aren't worth much.