I'm inclined to agree with the nature via nurture concept. Children predisposed to be highly intelligent can be impacted by such things as poor diet, abuse, lack of stimulation, etc.
Conversely, children predisposed to be average or below average in intelligence can overcome these challenges by having a supportive environment and learning the right coping skills.
Society runs into problems, however, when it expects that everyone will be capable of mastering certain learning skills. One either can, or cannot. That's the honest, ugly truth of the matter. Some children will never be brain surgeons, no matter how much support is provided.