Yes, this is definitely the crux of our differences. The idea that the Catholic Church is somehow an obstacle between God and man reminds me of the attitude of some kids who see their parents as an "obstacle" to having fun or living life the way they think it should be lived.
However, in reality, it's not an either/or situation. It's not a choice between "the Church" on the one hand and "the individual" on the other. Of course God can and does provide man with insight. This in no way negates the role of the Church, however. Spiritual inspiration comes to man every day but in the overall plan of salvation, God established a group of men, a structure, to keep his Word intact, preserve the truth and shelter us from the storm.
Our personal insight needs a point of reference unless we wish to claim infallibility for ourselves. We're happy to say that the Scriptures are infallible (and they are) but infallible Scriptures are of little use without an infallible interpreter (the Church). What use is a secure lock without a key to open it? What use is a clever piece of machinery in the hands of someone who is not familiar with it?
An infallible Word of God needs to be safeguarded by an infallible Church. The two go together.
I would suggest the interpreter has never been infallible as is shown in scripture. Peter, James and others were rebuked by Paul for being led astray. Paul and Barnabas got into an argument and so they parted company for a time. Apollos was teaching a "different" gospel, one that wasn't bad but wasn't quite correct. Paul didn't want anything to do with Mark but had a change of heart.
People are fallible as is illustrated by the different interpretation of the gospel. The Corinthians held some strange pagan beliefs mixed with their Christianity. The Galatians held to a strict form of Judaism and were condemned for it. Even within the early second and third century eastern and western church there arose fundamental doctrinal differences.
People are fallible as well as their interpretation. The only thing that has ever remained constant are the scripture. We trust in God to guide our steps to that true and correct path. He leads us besides still waters-not us.
Where you believe God guides through impersonal committees, I believe God guides us individually. Yet isn't this inconsistent with God appearing before saints, directing them to do miracles that have been recognized by the Church over the centuries? Why wouldn't He just speak through the Church cardinals?
No church is infallible. All have heresies.