No, the whole House votes for the Speaker. It's usually a party line vote, but it doesn't have to be. A faction of the majority party can bring down their party's leader by cutting a deal with the minority party (which almost happened to Newt once).
If the Republicans get such on offer, they should agree to it but make sure just enough Republicans vote Pelosi so she gets in anyway. That way the Democrats can keep themselves busy with reprisals.
You are correct, I looked it up after I posted.