I had to do some digging, but I believe this is the answer:
http://en.wikipedia.org/wiki/Jefferson%27s_Manual#U.S._House_of_Representatives
The House of Representatives formally incorporated Jefferson's Manual into its rules in 1837, stipulating that the manual "should govern the House in all cases to which they are applicable and in which they are not inconsistent with the standing rules and order of the House and the joint rules of the Senate and the House of Representatives."
This URL contains (among other stuff) "Jefferson’s Manual of Parliamentary Practice":
Constitution, Jefferson's Manual, and the Rules of the House of Representatives, 111th Congress
See page 3 of Section IX: Speaker:
§ 315. Removal of the Speaker: A Speaker may be removed at the will of the House, and a Speaker pro tempore appointed, 2 Grey, 186; 5 Grey, 134.
I believe that is the actual notes made by Jefferson, and this additional text is explanation that was added later:
A resolution declaring the Office of Speaker vacant presents a question of constitutional privilege (VI, 35), though the House has never removed a Speaker. It has on several occasions removed or suspended other officers, such as Clerk and Doorkeeper (I, 287–290, 292; II, 1417). A resolution for the removal of an officer is presented as a matter of privilege (I, 284– 286; VI, 35). The Speaker may remove the Clerk, Sergeant-at-Arms, and Chief Administrative Officer under clause 1 of rule II.
My interpretation is that the Speaker can be removed by a majority of all Representatives. So, that would mean it would require 218 Republicans to vote against their sitting Speaker.
It's interesting to think about the ramifications of having the R speaker elected via a deal with the Dems.
How would this affect the rest of the R members?
Heck we may be dreaming... we may not get to any of this but it's nice to ponder!