Except the law doesn't allow Congress access to school records or medical records on request. It does for tax returns.
You are exactly correct, though I believe that the original intent was to allow legislators to see samples of returns related to the legislative purpose to hand. If tax returns become political fodder, people will take extraordinary measures to conceal their financial activities, threatening the voluntary income tax system.