As a terror weapon, radiological bombs are pretty terrifying, mostly because the general public has been fed decades of over-hyped phobic reporting on the dangers of radioactivity. This is the same misinformation that has caused hysteria over depleted uranium weapons, or has blocked or slowed the use of irradiation on food to kill bacteria. Most people have no conception of the significance of various types or levels of radiation.
So yes, a dirty bomb will terrify a lot of people, and it will force huge evacuations, and it will destroy property values in affected cities to the tune of hundreds of billions of dollars even if the reality is that the radiation levels are so low as to be almost indistinguishable from background levels. More people will die of heart attacks than get sick from radiation exposure.
If terrorist want to push the U.S. economy into a recession, a dirty bomb exploded in a major city would be a great way to accomplish that. If terrorists want to kill a lot of people, then they should instead use true Weapons of Mass Destruction.