Actually the Bible does talk about the "giants" but doesn't say where they came from. And a lot of people think Neanderthals were humans so I don't see a problem.
Neanderthals were neither Homo sapien sapien nor giants.
The only 'giants' we have evidence of, 10 ft tall Gigantopithecus blacki may have been a relative of ours but was closer to the orangutan than to Homo erectus, an ancestor of ours who lived at the time of Gigantopithecus. Does this mean the giants in the Bible are just big Gorillas?