It was the 1970s riot in Asbury Park that did it in (although parts of it have come back VERY strong. I’m a few minutes away.) Then some of the shore towns with large Victorian homes/hotels took in the Section 8s - literally taking them off the hands of more prosperous NJ towns. White flight ensued.
Part of the problem with the loonies in Asbury and Ocean Grove were that they WEREN’T taking their meds.
You should visit Asbury again. It’s doing pretty well. I’m not at all fond of the newly designed waterfront but it’s popular with visitors and residents alike.