If he was simply abstemious while Speaker and dropped half his pay (which was not immodest as these things are measured) into his Thrift Saving Account he'd easily have quadrupled his networth.
What he did later was in the tens of millions per annum and it wasn't by "investing".