Your assertion allows for all kinds of exceptions to the 1st amendment, depending on a judge's whim. If a judge rules that the word "irresponsible" can be "defamation per se," as in "Pelosi is irresponsible," that doesn't have anything to do with the 1st amendment?
Elements are generally
a. Defamatory language (Language adversely affecting one's reputation. It must be based on specific facts, and express allegations of facts are defamatory).
b. Of or concerning the plaintiff.
c. Publication (communication to somebody besides plaintiff - example, posting on internet)
d. Damages to reputation.
If the matter is of public concern plaintiff must prove falsity and fault.
There are differences in degrees of libel within states, but many (including California it seems) require libel per se which are statements generally (varies slightly between states):
A - Adversely reflect on one's conduct in business or profession.
B - One who has a loathsome disease. Historically, this was leprosy. Today, AIDS applies.
C - One who is guilty of a crime involving moral turpitude.
D - Unchaste woman (sometimes expanded to all sexual misconduct)
As far as Pelosi goes, Actual Malice is required before she could bring defamation action. The actual malice standard requires that the plaintiff in a defamation or libel case prove that the publisher of the statement in question knew that the statement was false or acted in reckless disregard of its truth or falsity. I hate using Wikipedia as a source, but that matches what I was taught in law school, although I'm not a defamation lawyer.
In this case, the question I have is whether the judge ruled on the term "criminal" as the article states or whether it was referring to a specific felonious assault. If it's the former, that judge has his head up his arse. If it is the latter, it was the right decision if he wasn't there.