Men were pushed into ineffective PSA tests but although they were ineffective, they were easy. The prostrate exams are not fun (I have heard). Men (and also women) get the invasive kinds of colonoscopies where they slice and dice "polyps" that are more than 99% benign. Personally I am going to think long and hard about that one before letting someone subjectively analyze my polyps through a scope. I like my polyps (if they are benign), no reason to remove them. Removal is mainly driven by the pervasive myth that cancer follows some progression starting with abnormal growth. But it does not.
Actually, many men with very early stage prostate cancer opt to have surgery, too. It’s not just about the PSA test.
Like you, I’m cautious about medical testing and medicines myself. I used to mistrust them altogether. Once you’ve been diagnosed, though, you begin to realize it’s a life-or-death situation. As much as a person might not want to have surgery or testing or medication, sometimes we have to concede to those things if we want a chance at a long life.