I avoid situations where I could be tempted
You might enjoy this true story:
Many pastors were in Philadelphia for a multi-day conference, with their evenings free. As the afternoon session ended and they moseyed out to the lobby, one said to another, "The Cardinals are playing the Phillies tonight. Wanna go?"
Before the other could even reply, another pastor who had overheard the question erupted: "You're going to a BASEBALL game? Pastors?! How can you even call yourselves Christians?"
Astonished, the two innocents questioned their interrogator, and they discovered he had once had a baseball obsession that nearly cost him his pastorate and his family--so now he avoided it at any cost and strongly discouraged others.
Of course, they calmed him down and 'splained things to him, and then they each grabbed dinner: hot dogs at the ballpark. And the third continued, like you, avoiding tempting situations.
That’s a problem both Paul and C. S. Lewis talked about. A suitable answer would be: “Excuse me gentlemen, please enjoy the game but the devil messed with me too much through baseball for me not to be hurt by it.” The problem ws not the sport itself but how the devil turned it into that man’s idol.