The plot thickens!
That’s an urban legend.
Travel to Pakistan by American citizens was not banned by the U.S. or Pakistan. It was so easy for Americans to travel to Pakistan in 1981 that Pakistan was issuing 30-day visas to Americans upon arrival at any airport in Pakistan.
I’ve posted (on FR) the relevant Travel Advisory issued by the U.S. State Department in 1981. It summarizes the above information and further indicates that if American citizens plan to stay in Pakistan longer than 30 days they must apply for a longer-term visa prior to travelling there.
You can verify this information yourself by searching FR for the travel advisory.