Microsoft fails Slammer's security test (MS neglected to install their own MSSQL patches)

CNet News ^ | January 27, 2003

[HAL9000]

Microsoft's policy of relying on software patches to fix major security flaws was questioned Monday after a series of internal e-mails revealed that the software giant's own network wasn't immune from a worm that struck the Internet last weekend.

The messages seen by CNET News.com portray a company struggling with a massive infection by the SQL Slammer worm, which inundated many corporate networks Saturday with steady streams of data that downed Internet connections and clogged bandwidth.

"All apps and services are potentially affected and performance is sporadic at best," Mike Carlson, director of data center operations for Microsoft's Information Technology Group, stated in an e-mail sent at 8:04 a.m. PST Saturday to other members of Microsoft's operations groups. "The network is essentially flooded with traffic, making it difficult to gather details concerning the impact."

The messages put Microsoft in an awkward position: The company relies on customers to patch security flaws but the events of last weekend show that even it is vulnerable. In this case, Microsoft urged customers to fix a vulnerability in the SQL Server 2000 software, but it apparently hadn't taken its own advice. Moreover, despite its 1-year-old security push , the software giant still had critical servers vulnerable to Internet attacks.

"This shows that the notion of patching doesn't work," said Bruce Schneier, chief technology officer for network protection firm Counterpane Internet Security. "Publicly, they are saying it's not our fault, because you should have patched. But Microsoft's own actions show that you can't reasonably expect people to be able to keep up with patches."

For years, system administrators have complained about their inability to keep up with the steady stream of patches that have poured out of Microsoft and other software companies. In October, the software giant even raised the bar for what's considered a "critical" vulnerability, so that administrators wouldn't have to deal with so many patches that seemingly required immediate attention.

"Seems like every time I install a system patch, something else goes wrong with my system," said Frank Beier, president of Web design firm Dynamic Webs. The designer said many system administrators won't patch for many months, because they don't trust Microsoft to fix the problem without breaking some other function of the software.

"In most cases, I'm better off just playing Russian roulette with the hackers until our servers are broken into," he said.

In the case of SQL Slammer, it seemed that Microsoft had done it right. The company had informed customers six months earlier about a flaw and included patches in both a roll-up patch--a software update that includes all the latest patches--and in the company's latest service pack for Microsoft SQL Server 2000.

But even within Microsoft, something went wrong.

"At approximately, 10:00 p.m. (PST, Friday), traffic on the corporate network jumped dramatically, eventually bringing all services to a crawl," stated Carlson's memo. "The root cause appears at this time to be a virus attacking SQL."

On Saturday, the Microsoft's Windows XP Activation service was down, not because the servers were vulnerable, but because the company's internal network was inundated with junk data, Rick Devenuti, the chief information officer for the software giant, said in an interview Monday.

"We are not sure how the virus got into our network," he said.

That the company has SQL servers on the desktop is not surprising, he added. Many of its developers run the database on their PCs, and other test machines have vulnerable databases installed to replicate customer networks. Devenuti didn't know how the worm got into the system to affect those servers, however.

"It just takes one machine to get going," he said. "At any given point in time, it is hard to be 100 percent patched with any machine. We are working hard to make patch management easier. But 100 percent is a high bar and in this case we are not there."

[AFreeBird]

BWAHAHAHA... This is rich.

[Southack]

The overriding problem with MicroSoft's design philosophy (which controls their Architectural design, which dictates how software is built, which influences the quantity and quality of software patches, et al) is that it places top priority on "features" rather than on functionality.

In fact, most of the Microbots that I've dealt with can't even differentiate between features and functionality; they honest to God claim that they are one and the same!

But the difference isn't subtle. "Features" are new bells and whistles. Adding a browser to an Operating System is an example of a "feature". On the other hand, getting the operating system itself to work without having to reboot is an example of "functionality" (e.g. installing Microsoft Dot Net) or lack thereof.

So it is this overriding design emphasis on "features" that continues to plague programmers (and system administrators, and security staff, and help desks, oh yeah, and end users). By definition, placing an emphasis on new "features" means that true, working, robust "functionality" has taken 2nd (or lower) place in your design shop.

[supercat]

How hard would it be to use language tools that generate full automatic bounds-checking code all the time? Pointers would probably have to be eight bytes rather than four [every pointer would be a combination of a handle and an offset] but the performance penalties shouldn't be too bad. Use of such code generation tools would probably help to avoid many of these problems before the initial code release, instead of letting them slip by necessitating future patches.

Am I the only guy who remembers when software was just released once, and it simply worked?

[supercat]

How hard would it be to use language tools that generate full automatic bounds-checking code all the time? Pointers would probably have to be eight bytes rather than four [every pointer would be a combination of a handle and an offset] but the performance penalties shouldn't be too bad. Use of such code generation tools would probably help to avoid many of these problems before the initial code release, instead of letting them slip by necessitating future patches.

Am I the only guy who remembers when software was just released once, and it simply worked?

[Billthedrill]

It can happen - over and above the fact that Microsoft sells the product, they're a customer too, with exactly the same faults and limitations of any customer, some of which were cited. A patch rushed into production and distribution before it is tested on a wide enough variety of machines may have unforeseen consequences - the "ripple effect." Security patches, especially, inherently involve an urgency that lends to this tendency. Where one of these ripple effects is encountered people become hesitant to patch, and the more of them, the more hesitant.

It isn't the System Administrator. SAs are living gods striding over a supine and reverential user community and are incapable of error. Even a brilliant, a genius, a very paragon of systems administration can get caught. Myself, for example. [Humble coughing in background.] I had a contractor (strike one) who didn't keep his patch levels to spec (strike two) plug his system into the company network without it being checked for currency (big strike three) clear across the continent, and all of a sudden Saturday morning we were seeing packets zinging around like flak over Baghdad. No damage done except for a couple anxious hours on my part, so I let him live this morning.

But there you are - even the most careful can get blindsided, and so too with Microsoft. I do not think that this means that "patching" has any fatal flaws as a concept, but clearly some measure of review is required and I'm betting the lights will be on late in Redmond tonight. Not sure the ripple effect is much of an excuse for this one, though - this particular patch has been out for six months and by now if there were any ripples we'd have heard. That's why it's on my boxes.

Now I shall proceed to flog myself with barbed wire for defending Micro$oft. I deserve it.

[bvw]

I have devloped some embedded systems that have been running forvever -- patchless. The desktop -- especially after the infamous MS-Registry, not to mention the constant effervesence of common load modules (dll's etc) under MS-Win -- is another story. In embedded systems, a developer has total or almost total control of the platform -- in modern servers and desktops, there are inumerable variations.

Easier to deliver a solid product under the Unix delivery models than under MS's flavor of the month.

[glorgau]

Am I the only guy who remembers when software was just released once, and it simply worked?

It all comes down to greed and self-interest. When shipping software meant printing up a slew of floppies along with a manual and a box along with shipping costs, there was a _significant_ penalty to be paid for shipping faulty software.

Do it wrong and you have to go through the whole process - publically - again.

Now, all one has to do is type up an email, post a file, and walla! a product is born/renewed.

[X-FID]

The messages seen by CNET News.com portray a company struggling with a massive infection by the SQL Slammer worm, which inundated many corporate networks Saturday with steady streams of data that downed Internet connections and clogged bandwidth.

The local news called it the "sequel slammer", their ignorance was showing, I believe SQL is an acronym for "Standard Query Language" a database language, I could be wrong though :)

[Boundless]

> In this case, Microsoft urged customers to
> fix a vulnerability in the SQL Server 2000
> software, but it apparently hadn't taken
> its own advice.

That's less than half of the problem.

Another big part is leaving port 1434 open on
at least one of their systems, exposed to
entire world outside the firewall.

Another is not filtering port 1434 at the
firewall, assuming MS has a firewall, of course.

There's no sane reason to expose 1434, unless MS
offers play servers for developers to try their
SQL apps against, of for P2P web services with
known and trusted customers, and in either of
those cases, such hosts need to be very carefully
isolated from the internal MS networks.

Assuming that the release of this worm wasn't unplanned,
the timing was actually fortuitous (for everyone except
weekend admins-on-call).

Had it happened at the opening of the markets on Monday, Mr.Bill would have caught the level of unpleasant
publicity that he has clearly earned for apparently
having learned nothing from these periodic exploits
of MS operating systems and apps.

I had to laugh when at least one clueful reporter
pointed out that the freebie MySQL has a more
elegant patch process than the $30K MS SQL2K.

[Psycho_Bunny]

"Seems like every time I install a system patch, something else goes wrong with my system," said Frank Beier, president of Web design firm Dynamic Webs. The designer said many system administrators won't patch for many months, because they don't trust Microsoft to fix the problem without breaking some other function of the software.

True enough to bear repeating.  I've rolled back patches because of this.

[FReepaholic]

>>...For years, system administrators have complained about their inability to keep up with the steady stream of patches that have poured out of Microsoft and other software companies...."Seems like every time I install a system patch, something else goes wrong with my system,"...<<

Amen.

Patching my company's servers and workstations has become an almost full time job for me. Plus, the time it takes to fix problems the patch causes.

We're a small company. I can only imagine the horror of trying to manage thousands in a large company.

One thing about it...I've got job security.

[Psycho_Bunny]

"sequel" Is the correct slang.

That's what you were referring to, right?

[Fractal Trader]

BWAHAHAHA... This is rich.

According to Bush2000, the folks at Microsoft must be morons for failing to apply their own patches!

[TXnMA]

Am I the only guy who remembers when software was just released once, and it simply worked?

Certainly not! Many millions of us Macintosh users experience that happy state every day...

[SauronOfMordor]

How hard would it be to use language tools that generate full automatic bounds-checking code all the time?

There are languages that have automatic bounds-checking - Java being one. On mainframe, PL/1 had the compile option of generating code to check for bad array subscripts and such before using array indices (you were supposed to use this feature in debugging stage, but I generally left it in in production, since it wasn't a huge performance hit).

[HAL9000]

According to Bush2000, the folks at Microsoft must be morons for failing to apply their own patches!

Bush2000 will get bonus points just for showing up on this thread.

[chilepepper]

The overriding problem with MicroSoft's design philosophy...

MicroSoft is NOT a software company, it is a software MARKETING company. Its innovation, its pioneering work in the software field is in its development of the End User License Agreement, not in the product itself, which is slapped together in a most hap-hazard fashion...

It is the genius of Bill Gates to have a company whose product

1. is not even OWNED by the purchaser

2. absolves the company of ANY blame as to whether the product works or not...

So, the entire Internet is brought down by a Windows product?

Not to worry! MicroSoft is protected! The EULA is a free pass: MicroSoft can crank out any kind of crap, and they are immune from lawsuit or prosecution. The buck stops with Linus and Open Source, not in Redmond!

It is Linux and Open Source which have theoretical bugs!

Of course, it is MicroSoft that has the REAL bugs that cause companies to lose money, and create a

THREAT TO UNITED STATES NATIONAL SECURITY DURING WARTIME

[Poohbah]

According to Bush2000, the folks at Microsoft must be morons for failing to apply their own patches!

They are--as is anyone else who doesn't apply the patches.

[supercat]

"Seems like every time I install a system patch, something else goes wrong with my system," said Frank Beier, president of Web design firm Dynamic Webs. The designer said many system administrators won't patch for many months, because they don't trust Microsoft to fix the problem without breaking some other function of the software.

Correct me if I'm wrong, but in open-source systems, aren't most patches pretty explicit in their function (if not via documentation, then via examination of what the code actually does)?