Not really. My family has all four wisdom teeth in all generations. There are thousands of us. My parents had 6 kids, each set of grandparents had 10 kids. Only our generation has less kids. The reasons aren't because of wisdom teeth.
But the same argument would hold for my family as well...altho I had wisdom teeth, neither my brother or parents did....and my great grandparents had 12 children that lived to adult hood, and their children had many children, and in my parents generation and in my generation we have had less children...I did not say that people with wisdom teeth had less children...
But it does make sense, that those people who did have wisdom teeth, ,before the advent of modern dentistry, and had those wisdom teeth become impacted and infected, would die earlier, and perhaps die before they had a chance to reproduce...I do know, ,that my wisdom teeth were impacted and were infected, and if a dentist had not removed them, I would have been very ill, due to that infection, and possibly could have died...
So, as was previously pointed out by another poster, this may have had some effect, tho a small one, as to how many with wisdom teeth, or without wisdom teeth went on to reproduce...
I dont believe that anectodal stories about your family or my family matter much in the scheme of things...