You don't even really need the explosive charge. All you need to do is have a big enough chunk of critical mass and it explodes all by itself. That's why these 'gun' type bombs are so easy to make. You just slam two pieces of Uranium together and boom.
A terrorist would not need all that bomb casing, fins and other stuff. Just a tube and two chunks of Uranium that would fit inside of a half gallon milk container.
If the terrorist had plutonium, and a bit more skills - he/she could build a very small bomb that would do a lot of damage.
HEU is the scariest. It's so dangerous that they have to be very careful in places where they store it, so that no storage rooms are adjacent to one another. If someone were to stack a bit of this in the corner of one room, and someone else were to stack another bit of HEU in an adjacent room where it were up against the same wall as the first stack... that could be enough to cause a nuclear explosion!
You cannot have too much of this stuff in the same place or it will just blow up on its own. Even if half the mass is on one side of a wall, and the other half is not seen to you on the opposite side of a wall.
If terrorists can get their hands on it, the explosion is easy to make.
They have already caught people with plutonium and uranium in Germany and the Czech Republic.
Sadly, it's probably only a matter of time.