Mandatory participation is the problem. It is incompatible with freedom. Compulsory is what makes it wrong. Nothing justifies the penalty for forfeiting freedom.
Try this for a canard: Those who forfeit freedom in exchange for security end up with neither. And the United States is well on the way.
I presume common defense justifies forfeiting some freedom. "Security" is ill-defined and probably has little to do with real security (protection from people who take lives or property). I agree it's not worth giving up freedom (e.g. 4th amendment) for such security (e.g. catch criminals more easily).
On the economic side, I think it's a tougher call. Some property is already taken for the common defense. Penalizing success through unemployment tax is the same necessary evil with some redistribution aspects (not a good thing) and some smoothing of economy over regions and over time (which I consider a good thing). Furthermore, the country's manufacturing base has defense value and is often hit harder in economic cycles because it is physical. Preserving that work force to some extent helps the country survive over the long run.