Yep, it's the pilot's job is to make the go, no-go call. The client does not have the knowledge or experience to evaluate the conditions and risks and can't order the pilot to do anything.
Unfortunately, if the client is high profile enough, and you REFUSE him, you may permanently, depending on circumstances, LOSE him as a client.
My point: It's hard to say "NO" to the man paying the bill, especially if he thinks he's "right" and even becomes demanding in a particular situation.