I use a paper towel to touch the door handle of the rest room when leaving. If you don't, you cancel out your precaution. I have seen too many guys leave without washing their hands.
I don’t touch bathroom doors either. Plus I always use a disinfectant wipe at the grocery store to wipe off my cart handle. Nice that our store supplies them.