Was the SEAL team that took out Osama all white? A law-school friend of mine, who is black, is a former SEAL, so I have a hard time picturing a team of SEALs as lily white.
Statistically, SF are overwhelmingly white. While there certainly are members of pretty much every race in the most elite organizations, whites are (for lack of a better term) over-represented. In discussions at the end of the book Blackhawk Down, Mark Bowden discusses this at some length (there had been only one black member of TF Ranger). Whether you want to call it an excuse, conventional wisdom or fact, this disparity has largely been attributed to the stringent swimming/water survival qualifications of all the elite units from Army Rangers, SF, USAF Pararescue, and of course, Navy Seals. Blacks on average have leaner body masses and less bouyancy than whites, which in terms of all other things being equal, inhibits their ability to swim.
As you point out in the example of your friend, there are certainly those who are fully capable of making the cut, but I've heard the above explanation cited from more than one source regarding the under-representation of blacks.
He / she is speculating...