Well it could certainly seriously damage his value for advertising.
Michael Vick went from the penthouse to the big house in one off-season. Depending upon the amount of play this gets, it will total his advertising value, which is the only value an Olympian has.
Phelps behavior, though isn't all that unusual. The regime required to get as good as he is at what he does almost requires an unbalanced binge personality.
I'd never really realized this until I worked with a guy who was on the University of Texas Golf Team. The guy drove me crazy talking about golf (I don't play.) He took a part time minimum wage job at a country club because he got to use the course for free (membership was something like 20,000 a year.) I got to where I'd hit the timer on my watch whenever we had a conversation. We never got past 2:34 before he was talking about golf. He could start on the first hole and describe every stroke in an entire round, and would if you couldn't get away from him. I asked him why he didn't turn pro. He told me he didn't like golf enough to put in the effort necessary to be successful as a pro.