READ what is in your book![ regarding who Jesus gave the keys of the kingdom to]!
I must admit this is an inventive way of reading the passage in question. That's perhaps the most charitable assessment I can make of this argument.
As you correctly state, there are no chapters and verses in the "originals" (really copies of course but that's an irrelevant point here). So let's place ourselves in this scene, by imagining ourselves as witnesses to this exchange between Jesus and His apostles. A good practice by the way the Church encourages (it's called "lectio divina", reading Scripture, and meditating on it, with the expectation that we are dialoging with God and wish to hear what He has to say). So let's do that.
Imagine the scene. Jesus, at a certain point when He entered the region took an opportunity to ask his disciples, (all of them present), "Who do people say the Son of Man is?" So He's talking to all of them at this point.
Then they all respond with various (incorrect) answers, and so He asks them all again, "Who do you say I am" meaning the entire group, who do they say He is.
At this point they all fall silent, because probably no one wants to look the fool. But at a certain point, Peter speaks up ("impulsively" as you say, rather amusingly, because this "impulse" of Peter clearly comes from God the Holy Spirit Himself), but he speaks up and says "You are the Christ, the Son of the Living God".
At this point, when I've read this passage, I always imagine Peter shouting this sentence at the top of his voice, as He is moved by the Holy Spirit in a profound way.
At any rate, we've now reached the critical juncture of the exchange, the point you say He then continued to speak to everyone, but Catholics say He only spoke to Peter.
I would only submit the following in reply to such a claim: Imagine again the scene. If someone asks a group a question, and then that group falls silent, for a time however long doesn't matter, but then one person in that group speaks up to answer the question what is the natural, human way such an interaction would proceed?
Would the questioner (here Jesus) then continue to address only the group, or would such a questioner (He) then start to address the only person who had spoken up?
I submit the latter is the natural, human progression of such an interaction. Think about a business meeting or any other kind of meeting between human beings (of which Christ is one). When the group reaches an impasse, but then one of the group speaks up with an idea or suggestion, what is the natural tendency of everyone else in that group at that point? To address the group again, or to address the person who spoke up?
It's to address the person who spoke up. That's just plainly evident to anyone who has ever had interaction in a group setting. You reply to someone who spoke up, not to the group again, even if your question was to the group originally, if at least the one who spoke up was correct or had a good idea, as Peter was correct. So the natural tendency would have been for Christ to then address only Peter at that point, since he was correct in what he said, and also what he said was very profound, very important, and thus deserving of some recognition. Such recognition is even recorded in verse 17.
If the entirety of the above is not convincing, then at least consider the next facts. One, in verse 17-18, we clearly have Jesus addressing only Peter. So are we to believe He addressed only Peter in verses 17-18, but then went back to addressing the group as a whole in verse 19? I submit that makes no sense, given the natural human tendencies as described above but more importantly in the Greek the "you" in verse 17-18 is the same "you" (same tense) as verse 19, the singular. Secondly, that throughout history at least, up until the "Reformation", no one ever denied that at this point, verse 19, Jesus was addressing Peter and Peter alone, not the group. But again, as I pointed out in my previous post, this shows yet again a need for a divinely led teaching authority, or else everyone's personal opinion is what rules instead of the truth.
We have a divinely led teaching authority and it is the word of God and the Holy Ghost...
The bible clearly says that all the disciples were given the ability of binding and loosing...