True, but aren't laws that DO affect the rest of us use, as precedent, narrow SC decisions?
People in government do not always act legitimately. It is alas very common for people in government to enforce rules that are, for various reasons, not laws. In many cases people harmed by such actions may not have a remedy, but that does not mean such actions are legitimate.
Indeed, I would posit that one of the major problems with the common-law concept of precedent is that while it occasionally recognizes that not every illegitimate action that harms someone justifies a remedy [generally when it's forced to, since an illegitimate harmful action clearly occurred, but no workable remedy can be devised], it largely fails to recognize a major implication of this: a finding that a harmful action does not justify a remedy in no way implies a finding that such action was legitimate.