I spent several years in projects moving Linux from 64 CPUs to thousands of CPUs. I was personally running Linux on 1024 processor systems with terabytes of main memory a couple of years ago. Massive operating system work has been done on Linux, as well as major library work, over the last five or ten years to get it to work exceedingly well on very large systems.
See further details for example in the Forbes 2005 article Linux Rules Supercomputers.
Or see the Wikipedia File:Operating systems used on top 500 supercomputers.svg file for a graph showing which operating systems run on the worlds 500 most powerful supercomputers, over the years. It went from about 90% Unix (which would include Solaris, I suppose) in 2002 to about 70% Linux in 2006.
In what configuration? Sounds like you’re talking about something like the SGI Altix setup at NASA, yes?
Last I looked, SGI’s kernel changes aren’t yet certified for Oracle deployment. Oracle has their own versions of Linux which they recommend for deployment - based on Red Hat Enterprise distro’s from what I last know. And last I looked, the Linux crowd was pitching a fit about the license Sun had on ZFS being incompatible with the Stallmanesque theology of Linux. Now that Oracle owns Sun, perhaps ZFS’ license changes to allow incorporation into Linux, but that’s probably down the road a bit.