"Nephilim" does not necessarily refer to giants. According to Wikipedia (I know, it's not an authoritative source):
The Brown-Driver-Briggs Lexicon gives the meaning of Nephilim as "giants".[1] Many suggested interpretations are based on the assumption that the word is a derivative of Hebrew verbal root n-ph-l "fall". Robert Baker Girdlestone [2] argued the word comes from the Hiphil causative stem, implying that the Nephilim are to be perceived as "those that cause others to fall down". Adam Clarke took it as a perfect participle, "fallen", "apostates". Ronald Hendel states that it is a passive form "ones who have fallen", equivalent grammatically to paqid "one who is appointed" (i.e., overseer), asir, "one who is bound", (i.e., prisoner) etc.[3][4] According to the Brown-Driver-Briggs Lexicon, the basic etymology of the word Nephilim is "dub[ious]", and various suggested interpretations are "all very precarious".[5]
The majority of ancient biblical versions, including the Septuagint, Theodotion, Latin Vulgate, Samaritan Targum, Targum Onkelos and Targum Neofiti, interpret the word to mean "giants".[6] Symmachus translates it as "the violent ones"[7][8][9] and Aquila's translation has been interpreted to mean either "the fallen ones"[7] or "the ones falling [upon their enemies]".[9][10]
So, instead of giants, the nephilim could merely be "fallen ones", "violent ones", etc, rather than "giants" I do not think there is a definitive answer to this. I think some of the confusion comes from translations that are a tad too definite in their choice of words, using "there were giants in those days" as the KJV does, rather than the vague word Nephilim. I still think R.C.'s explanation makes sense, and it is what I believe; but there is not a lot to go on in the small passage we have to work with.
but there is not a lot to go on in the small passage we have to work with
There are many scriptures mentioning and describing the existence of giants...I see no reason to doubt the bible just because my level of understanding can't comprehend it...
Deu 3:11 For only Og king of Bashan remained of the remnant of giants; behold, his bedstead was a bedstead of iron; is it not in Rabbath of the children of Ammon? nine cubits was the length thereof, and four cubits the breadth of it, after the cubit of a man.