What is the reason for the neighbor who uses the easement to object? He has use of the easement, so long as he is not impeded there should not be any issue as it remains your property. Has he been maintaining the area of the easement, mowing, landscaping, gravel or pavement? If yes, for how long?
Please don’t take the above as legal advice, just think it through for yourself, and answer if you’re comfortable doing so.
He is objecting because he wants ditches on each side of the road(supposedly but we doubt that too). We know he wants our property, but we’ve made it clear in the past that we have no desire to sell...ever.