Same thing happened to me in the 3rd grade, my Mom gave the same answer.
My 8 year old grandson reads at the 12th grade level. No, he does not understand all the references but he is keeping all the adults in his life busy answering his questions. When he started the school year in a new school in a gifted program they tested him before they would let him check out the books he wanted to read. He was so put out with all the testing.
I had my dictionary right next to me and learned as I went.
No time spent with a book is ever wasted in any way.
Actually I spent SO much time in the dictionary because on the way to finding a word, I'd see some other word that ‘WOW... I always wanted to know what that word meant...”
It really slowed me down.
So I made a game out of it and I'd hold my breath until I found the word I was looking for... that way I'd go back to my reading without being distracted too much.... Then I'd go to the pages with those other words I wanted to know about, too!
Yup...I lived with my face in a book for YEARS, growing up. NO computer in those days!