I also assume that when men wore hats, they removed their hats and bowed to a superior. Later that became just touching the brim of the hat; and from there, it evolved into various hand movements related to the same.
Lots of different beginnings to that from different cultures.