The HOA was not concerned with the flag pole being dangerous. They decided that it was not aesthetically pleasing to them.
Aesthetically pleasing or not, its not his property. If my neighbor builds on an easement or on my property-I don’t care how pretty it looks-its not his property.
As stated above the simple solution is to sell this man the land the flagpole is illegally built on.