One downside is that the ball to some extent carooms its way out of the musket, decreasing its accuracy. And, of course, with the rifle, the angular momentum of the spinning ball helps to maintain accuracy.
The rifle-musket was used in the War of Northern Aggression (the Third War of Independence). It was made possible by the development of the Minie Ball (more properly, the Minie Bullet). The Minie ball, of course, is conical with a hollow base. The projectile does not have to forced down the barrel, and it has a hollow base. This base expands when the weapon is fired, causing the base to expand and the projectile to engage the rifling of the barrel. The rifle-musket, then, has roughly the same rate-of-fire as the musket with greatly increased accuracy. The Minie bullet was invented in the 1840s, I believe.
With black powder the barrel fouls rapidly, so that even with a musket, the ball must be rammed home. Loading a long rifle becomes even more difficult with powder foully.
I'm not entirely positive that I am correct, but I think that the term "musket" implies a reasonably large amount of windage between the projectile and the barrel
You are not correct.
When loading ANY muzzleloader the correct procedure is Powder.. Patch (wrapped around the ball)..Ball. This is then rammed down the barrel with the ramrod. Patches are lubricated and their purpose is to provide a gas/friction fit between the barrel walls and the ball.