Give me a specific example of an animal that responds to color without being able to see it? My first impression is that you are making this scenerio up.
Well, for one thing, if you wish to call "man" an animal, I will allow it for this discussion. (Seeing as I cannot know what an animal thinks)
I know a guy who is color-blind to blue. He knows a beautiful day when he sees it out his window though. He's reacting to the appearances without reacting to color. But how does he know its beautiful? It's gray to him!