Understanding your objection, what you are objecting to is not the teaching of the Catholic Church. The Pope is held to be infallible when he teaches Christians on a point of dogma or morals with an intent of binding them defniitively to his judgement. He is not infallible when he makes private theological speculations or declines to settle a controversy at some point in time.
Infallibility is also not localized in the Pope. While the Pope can speak infallibly on his own and definitively settle controversies, we believe an ecumenical council can also do likewise. And of course, we believe that all Bishops and Priests teach infallibly when they repeat the universally held beliefs of the Church in instructing us.
Lastly, infallibility is nothign more than a protection from error. It is not a positive power that gives the Pope especial knowledge, wisdom, eloquence, or understanding. The Pope cannot speak for the Church so as to lead the Church astray that's all. On the other hand, he might very well speak poorly, or perhaps not at all.
Lets look at it this way. You believe an ecumenical council is infallible in a way that the sermon of a Bishop in his cathedral is not. What essentially is the difference? Also, no ecumenical council has every included the participation of all the Bishops. Therefore, it seems like some fo the Bishops are being made more authoritative than all the Bishops. How is this different from your objection?
If the Pope states one thing and a council states something to the contrary, which statement can be affirmed? More to the point, can the Pope trump a Council?