There's now no doubt that several great extinctions of life on earth in the past were caused by asteroid hits. Granted, they don't happen often but when they do -- like eruptions of Vesuvius or other sleeping giants -- they can be devastating.
In the end we're all dead anyway but, for the living I don't think it's paranoid to invest some of our intellect and treasure into figuring out 1) how to demolish/deflect cosmic intruders once detected, then 2) how to detect them.