It's not that immigration lost J.D. the election, it's that it didn't help him at all, not even with the base. I've seen no electoral evidence immigration adds votes, even in Republican districts. It's one of those issues that shows well in polls, but it doesn't influence people in the booth.
In the end, what it really says is you can be the most personable guy on Earth, but if you get swept up in scandal of any kind and you are a Republican, you WILL pay for it at election time. I only wonder if the people left in DC that didn't run this time around (Senators) learned anything from this election.