He thought the fish or the fish sellers were all female?
For some reason, my very nice Italian-American female in-laws love this joke! I’ve heard it for years, lol!