Baloney! Olympic class athletes are by definition some of the most healthy people in the world. Very, very low chance of infection or serious illness from the Wuhan Flu.
People ordinarily in good health can still catch and pass on the virus, even if they don't get serious problems with it themselves. If the virus is still around there will be pressure on athletes not to go there, get infected, and bring the virus home.
Plus, the athletes countries send to the Olympics will be determined months before. If heats and matches are cancelled due to COVID19, who's going to be going to Tokyo? I'm not saying the Games will definitely be cancelled, but there's still a chance that they might.