Once its known that you hold gold (physical possession; gold on paper aint worth the ink), you will be a target for sure.
Following the inference in the original post, when you use physical gold to make a purchase, you OBVIOUSLY have gold.
Now; those who see you purchasing, know who has gold. Simple enough.
I wouldn't spend a farthing in the Inner City of any major metro area, and expect not to become a target of the "Have-Nots", would you?