I was kind of threatened by an IRS revenue agent once; right up to the point I informed him I was recording the call. Haven’t from the IRS again, EVER.
If the "agent" called you, then it was a scammer pretending to be an agent. The IRS never calls. They send mail, often registered, or ask you visit their offices.