Not all elf representations are small and pixieish. I always saw it as the largest of the elves by which he was surrounded. The poem to me has always read as shorter than average but very plump and rotund.
In truth he never talks about his stature/height...only that he is broad and round. The only potential reference to height is in the very first mention where he is called a ‘little old driver’ (to me not much different than any reference to an older gentleman) and the term ‘elf’ (which as I noted can be of many different sizes).
Here is the complete description from Moore himself:
“With a little old driver, so lively and quick,
.......
He was dressed all in fur, from his head to his foot,
And his clothes were all tarnished with ashes and soot;
A bundle of toys he had flung on his back,
And he looked like a peddler just opening his pack.
His eyes — how they twinkled! his dimples how merry!
His cheeks were like roses, his nose like a cherry!
His droll little mouth was drawn up like a bow,
And the beard of his chin was as white as the snow;
The stump of a pipe he held tight in his teeth,
And the smoke it encircled his head like a wreath;
He had a broad face and a little round belly,
That shook, when he laughed like a bowlful of jelly.
He was chubby and plump, a right jolly old elf,”