Exactly. And this is true on many other political issues as well. This is why professionals put very little stock in the "right track/wrong track" questions. They don't tell you why the voter is unhappy. It simply isn't true that an unhappy voter will vote against the incumbent or his party. Maybe he is unhappy because the other party is being obstructionist and getting away with it. A GOPer who thinks that his party is not conservative enough is not going to vote liberal Democrat.
--A GOPer who thinks that his party is not conservative enough is not going to vote liberal Democrat.--
You're absolutely right, but that conservative may stay home and not vote, which is basically +1 vote for the Democrat.