So you're basically telling us that it's possible to be a great athlete, and yet have no skills....
You realize that's not even remotely true. You're saying basically that "athletic" is the same as "physical," but we all know that great athletes have to be both physically gifted, and skilled at what they do.
In that case, then a surgeon is an athlete. Or a mechanic. A "skill" is something specific to the sport in question. Shooting a basketball is a skill. Throwing a javelin is much less so.
Actually, I'm saying "athletic" is more specific than "physicial". But, again, "non-athletic" is not a pejorative in my usage.