None of his business where I am coming or going.
You may say it is none of his business, but the law disagrees with you. He will follow the law so I’d advise you to as well or risk the consequences of your own making.