The expansive view started elsewhere, in particular with the president and Congress. Get rid of SCOTUS, and the president and Congress will retain the same illegitimate, expansive views of their own power.
The power of government lies, in large part, with the public's acceptance of the government as legitimate. My personal point of view is to take as illegitimate, decisions and assertions of force that are based on demonstrable fiction. I think the bulk of the federal government is illegitimate.
The difference, of course, is that the president and the Congress are accountable to the people. They must face the ballot box. And their mistakes are much more easily reversed, through new legislation. Supreme Court decisions, and the reliance of the Court on stare decisis, are much more difficult to reverse. And the justices themselves are totally unaccountable. Therefore, if I get to choose who gets the final say, an unaccountable court, or an accountable legislature, I'll take my chances with the legislature.