No doubt Egypt had a lot of slaves. That they were used exclusively on the pyramids seems unlikely. As others have pointed out, there are long periods in the Egyptian year when the farmers (the vast majority of the people) were sitting around with nothing to do.
Why not apply a labor tax to them and get some good out of this otherwise wasted (from Pharoah’s perspective) time? Meanwhile his slaves, a capital investment, probably had other regular duties.
Maybe they imported Illegals. ;0)