Yes, due to their accommodation arrangements the travellers aren't good targets for the threat of public retaliation. However, the cousins of Barras and their close relations are probably in on the death threats against Martin, so fingering them might deter the people who plan to do the deed.