1) How was the easement created - deed, prior use, other?
2) Who was the the beneficiary of the easement - government, utility, adjacent property.
3) What was there, if anything, before the fence?
Take that info and the related documents to a local real estate attorney, and you should be able to fairly quickly get an answer.
Easement is recorded on the deed for ingress/egress only.
We put the fence on our property line with no objection 15 years ago.