The keys are really considered to have been given the church.
Granted whether the rock is Peter or his faith changes that as well.
If one accepts that it's Peter's faith which is the rock, then the church which has maintained that faith is the rightful 'heir' or successor to holding the keys.
If one thinks it's tied to Peter's office it's still questionable seeing as Peter founded more than one church, and frankly more than one ancient see. That said the church fathers do indicate which See has primacy over the rest.
(Though I'd question exactly what that primacy entails, and whether that primacy is due to the keys being 'inherited by' that see alone).
The power to bind and loose, definitely, in Matthew 19. But the keys are not mentioned there; the renaming that came with the keys suggests a level of personal graft of Peter onto Christ. We can certainly seek the proper intended role of the Roman See's primacy. My opinion is that the imperial papacy is a unique product of the Latin particular Church, which extends to the other Churches sort of by osmosis. If you actually look at how the Papacy works with respect to the Catholic Oriental churches (so called Uniates), you will see a far greater consiliarity. But then, it is the Latin Church that was bearing the brunt of the Reformation heresy in the past several centuries.