When the IRS audited my MIL's estate tax return last year, they wanted about $4K for "Gift taxes from the 1960's". That's exactly how the explanation read, with no further details. How the heck is someone supposed to fight them over something that vague?
The IRS sucks big time. I hate that agency more then any other. If one of our candidates ran on getting rid of IRS and really mean it, I can see them doing well. Right now nobody has that as a priority. They may discuss it but it is not in the top 3. I know we have other huge problems but I hate the IRS so much.
My guess is you got a noob agent fishing, and would he mind respectfully checking with his boss on this, or I would get a tax attorney, and have them fight this on your behalf.