Actually, there were lots of fatal horse accidents. Pretty fast speed, unprotected passenger.
Historical accidents include Pierre Curie being run over by a wagon, Barbarossa falling from his horse (though it was drowning in the river under weight of his armor that killed him), there was one prominent English politician who was killed when thrown from his horse during the early 19th cent., Cole Porter, Christopher Reeves, Theodore Roosevelt more than once--the list really gets quite long (Horse related sports are sports of kings because only the very wealthy can afford to be laid-up with broken bones often and for some time).
All in all, I didn't say one is immune from accident by riding a horse--only that if you're drunk, you'd be safer letting ol' dobbin take you home than taking yourself home in a car.