I say make the quip even if no one else hears it.
'When he asked politician Lord Taylor of Warwick, whose mum and dad are Jamaican, 1999: “And what exotic part of the world do you come from?” To which Lord Warwick replied: “Birmingham.”'
Lord Warwick had the right approach.