Land's sakes man, in whose name do you think the priests are absolving????
"ego te absolvo a peccatis tuis in nomine Patris, et Filii, + et Spiritus Sancti. Amen."No one is saying the priest does it on his own....as if a man could forgive debts owed to God. God forgives through the priest."I absolve you of your sins, in the name of the Father, and of the Son, and of the Holy Ghost."
If my kid gets dirty and I wipe his face, is it the cloth that's cleaning him up? No, I'm cleaning him up by means of the cloth. When I go to Confession, God cleans me up by means of the priest.
Just like Christ used the Apostles (John 20).
Your mistake is in treating the priest as the principle actor in this, instead of just the instrument that God is acting through.