From the late fall of 1944 through the early spring of 1945, the Japanese launched more than 9,000 of these "fusen bakudan", or fire balloons, of which 300 were found or observed in the US.
http://www.japan-101.com/history/fire_balloons_or_balloon_bombs.htm
Just because you never heard of it doesn't mean it didn't happen. And just because they weren't very accurate doesn't mean that they were a very real threat.