>>Gravel alone would settle over centuries, and the pyramid would fall inward, or otherwise sag, <<
I know nothing about how pyramids are constructed but is it possible that the exterior stones work similar to the ice blocks of an igloo? The lower stones bear the strain of the constantly inward sloping upper ones?
We have two related issues here.
Firstly, the walls of a self-supporting arch (an igloo is just a revolved arch) should be somewhat parabolic, otherwise they do not survive. (In this case we have to assume that the exterior walls are self-supporting.) Flat, angled walls will fall inward. Vertical walls will fall outward. It takes some good eye, or tensor calculus, to find the solution that does not load the stones horizontally. Exterior walls of the pyramid are not capable of supporting themselves; they exert tremendous pressure inward, compressing whatever is inside. If that material inside were to be compressible, such as loose gravel (no vibration packers in Egypt!) the middle of the pyramid would fall inward. This is not observed; in fact, this is the most stable part of the pyramid.
Secondly, let's assume that the walls are self-supporting. You can see them in every medieval arch that there is. However there is a big catch. The arch is carefully balanced to counter forces that push on each side of each block. Normally, like in a castle, those forces are known to the architect, and they are not changing. But if the pyramid has loose material inside, over time it will create more dense and less dense areas, and that will unbalance the walls of the arch. Some stones would be pushed out, and other stones would be tilting and falling in. This is not observed.
There is a related thought. If you assume that the pyramid is filled with gravel, at that scale the gravel acts like a fluid. The law of pressure at depth will then apply: pressure = weight_density * depth. This immediately tells us that the lower layers of pyramid's walls would burst out from the pressure at the ground level, as all that gravel would be squirted out. At the same time, the pressure at the upper layers would be negligible. This is why a gravel-filled wall has to be self-supporting. Modern filled walls only need to contain vertical columns of light artificial ceramic, and the CMUs are designed for that.
It's even possible that as you fill the gravel up, in a column, at some point the weight of the gravel above will crush the gravel below, compacting it into a fine dust, and later into solid stone. This will result in significant reduction of volume - and then you get voids and cave-ins. Worse still, it may take decades and centuries to fully settle, so there is no chance of invoking the warranty.