Some Neanderthals had bigger brains than we do, and so do elephants and whales. So sure, size matters, but size is not everything.
When humans and Neanderthals first came in contact, their tools etc. were roughly the same. But then slowly something happened. Human tools became more complex, sophisticated -- hi-tech if you will. Neanderthals' did not. And Neanderthals never conceived of cave paintings or sculptures. Why? Was it just IQ? Or was it something else?
We don't really know. I suspect culture had something to do with it.
Elephant brains are less dess than humans.