I must admit, though, you take a tack I wasn't expecting. Usually, people demand someone to trace the doctrine back to the Apostles--the assumption being that it is a novelty. Correct me if I'm wrong, but you seem ok with the idea that the Roman Primacy is no novelty and the See may well have inherited this privilege from Apostolic times.
You only object that the privilege may not be a perpetual one. Rome was the seat of orthodoxy, but she is no longer (and she had presumably forfeited that title by the time of the Reformation). Am I stating your position accurately?
I myself would argue that Irenaeus is not arguing for the Bishop Of Rome per se, but rather for the Church Of Rome - and even then, Irenaeus is arguing in favor of it's then-present state, and not for some future-looking behavior.
I must admit, though, you take a tack I wasn't expecting. Usually, people demand someone to trace the doctrine back to the Apostles--the assumption being that it is a novelty. Correct me if I'm wrong, but you seem ok with the idea that the Roman Primacy is no novelty and the See may well have inherited this privilege from Apostolic times.
Let me try and be surgically precise with my answer on this. First, I tend to agree with the Orthodox position that the Roman Bishop does not/should not in a position of preeminence over the other Bishops of the Church, so long as the other Bishops are observant and obedient to the faith. Second, I would argue that scripture itself makes no demands for this ecclesiastical authority, on behalf of the Bishop of Rome, in perpetuity. None of this IMO challenges arguments regarding the papacy's temporal authority over the Catholic Church itself. Anyone who submits to an authority should be bound to it.
I have far fewer problems with the idea of Roman Primacy being "handed down" from apostolic times, or even with the (IMO false) notion that the papacy can be traced back in an unbroken line to Peter himself. All of those are statements regarding ecclesiastical authority and covenantal representation, not statements regarding ability (and please note that I said "fewer problems", not "no problems at all"). I think there's a general principle articulated by Scripture that if the officeholder and organization remain unblemished, then an argument of submission to the ecclesiastical authority and continuity of the organization (whether the Church of Rome or anyone else) could be advanced. Most critically, however, is that I believe the Roman Bishop (like any other bishop, priest, pastor, etc) can fall into error in regards to matters of doctrine. In other words, even doctrinal remarks made ex cathedra can be in serious error, and thus the claims of spiritual authority made for both specific organizations and offices can always be tested and (potentially) found wanting, no matter the historical pedigree. The possibility of apostasy (and repentance and restoration) exists for any and every Christian body, even for the Catholic Church. Christ said that He is able to raise up "sons of Abraham" (Matthew 3:9, cf Romans 9:8) from the stones themselves. I do not think Matthew 16:18 demands a single ecclesiastical organization to fulfill it's meaning.
You only object that the privilege may not be a perpetual one. Rome was the seat of orthodoxy, but she is no longer (and she had presumably forfeited that title by the time of the Reformation). Am I stating your position accurately?
I would say you're very close - I would argue over the exact nature of the phrase of "forfeited". Suffice it to say that I am not a Restorationist. I believe strongly in creedal continuity. I do not believe in, nor require a a Trail of Blood to explain "where the church was" in history.
FWIW, I got quite a chuckle when I realized that we've had a similar conversation on these things before!