Possibly because of his size, physical conformation. Not that that excuses the bullies, but might in some degree explain their actions. He was different, and by his custom of not responding belligerently may have caused his tormentors to think they could continue to bug him without consequences.
Probably. I just wondered what the back story was.