And yet according to Catholicism, the marriage was never consummated, which means Joseph was also a virgin.
Exactly. There’s no way to explain this implication so it’s ignored.
Well, either a virgin, or a widower. (Source: http://www.catholic.com/quickquestions/was-st-joseph-a-virgin-or-a-widower-with-children )