This ministry has been exercised in different forms and ways by different men in different centuries, and perhaps that's why you don't think there was a "pope" or a "Catholic Church" before the 4th or 5th century. But in this you are wrong: because you don't see the development of a sacred institution as it responds to historical challenges and new situations.
You will not find in Peter and his immediate successors a "monarchical" papacy, but will find him to be exercising, in his circumstances, and in an early form, the same ministry.
While Peter was a good friend of Jesus, the fact that he often slipped into a very skewed view of the Gospel is spelled out plainly in Paul’s letter to the Galatians. He had a tendency to hide in his Jewishness and was soundly (and publicly) rebuked by Paul, the apostle sent to the Gentiles (that’s you and me). Check Paul’s letter to the Roman believers and see if you find any trace of Peter’s supremacy...I find only Jesus. But, if you need to have some sort of “worldly organization” to feel safe, I leave you to that. But, out here, out of the camp, with Jesus, there is freedom and light and safety.