OK, there’s a guard shack on the only road to the island to keep things exclusive.
https://goo.gl/maps/J1tqxHSJp9dfxBRi6
You need to be invited by a resident if you want to go for visit (or have a reason to be there.)
Thurgood Marshall was once a busboy there.