The key to understanding the puzzle is in understanding to what level the “observer” is neutral.
That is, with sub-atomic particles, “watching” them is not the equivalent of a person watching a football game on a television, live. The observer in the latter case has practically no effect on the outcome of the game. However, in the former case, “observation” implies interference because there is no way of observing a sub-atomic particle without affecting the particle, directly. In other words, if you were to imagine the electron as a ball, in darkness, the very act of shining a hypothetical “light” to detect its reflection off of the “electron” has an effect on the electron itself. That is the scale we’re talking about here.
Which begs the question - what is the "light" we're shining?
Except that in the double-slit experiment we are observing the outcome of both cases in the exact same way.
In other words, our manner of interfering in order to observe the outcome of the experiment is no different in the case that results in the wave manifestation than in the case that results in the particle manifestation. In both cases, we are observing by measuring the impact of the electrons or photons on a screen after having passed through the slit or slits, in the same way.
So the different results cannot be explained by a difference in the way we are interfering in order to observe.