The reason is that if you had a tanned skin and non-calloused fingers, that meant that you didn't work with your hands, ergo, you were nobility
Just like the kimono in Japan or the toga in ancient Rome -- you couldn't work in these clothes, so it meant that you were the rich class who got others to work for you
The effect is the same.