That’s what used to happen in many small towns across the USA.
You would lock them if you went out of town just so the neighbors would know you were gone. The worst thing that happened is that one of them might mow your lawn because they thought it was a looking a little scruffy. Of course, a lot of those places weren't blessed with much "diversity."
We spent a few days vacationing in east central Tennessee in February. At one museum, I talked to an old black man who told me that the races there had always gotten a long quite well. He said even during the peak of agitation in the mid 1960s, the local blacks would tell the outside agitators to keep moving because they didn't want their kind making trouble. Some of his ancestors had even fought for the Confederacy, not because they thought slavery was a good thing, but they just didn't want outsiders bringing trouble to that part of Tennessee.