I do solemnly swear (or affirm) TO WHO??
“Affirm” was put into the Constitution specifically for people who did not want to “swear.” And John Quincy Adams took the oath of office with his hand on a law book, not a Bible.
I quoted the Founders’ words. If you have an argument with the wording, take it up with them.