It's a constitutional office.
Boehner can be swept out on a single vote ~ quite comparable to a vote of confidence in a parliamentary system only we don't call it that.
Henry Clay had his terms cut short twice, Andrew Stephenson was cut short, MIchael Kerr, Henry Rainey, Joseph Byrns, William Bankhead, and even Sam Rayburn were cut short before the end of their terms.
I suppose some of this was because of death or disability. But there's nothing to stop the majority party from withdrawing its support and bringing down a Speaker.
Except they're all prostitutes. Their jobs are more important than the country.