I suspect the security aspect is a pain the butt for neighbors.
I have been there many a time and it is a police peninsula once you cross over the bridge....most that live there (not what we the unwashed consider living in a house) rarely venture except to the restaurant or liquor store...hell I went to the ocean every time visiting there and never saw a soul...VERY nice beaches there with a quick drop off and usually good waves.
I was just by there yesterday. I didn't see any obvious security. It was very different when he was here as the Official President. Then you couldn't drive on the roads that roads that bordered the property at all. Ditto with boat traffic on the Intracoastal. There one could pass, probably at at distance (I don't know. I'm not a boater.), but armed Coast Guard putt-putts were all over the place.
ML/NJ