Free Republic
Browse · Search
General/Chat
Topics · Post Article

Skip to comments.

How McAfee turned a Disaster Exercise Into a REAL Learning Experience for Our Disaster Team
SANS ^ | 4/22/10 | Deborah Hale

Posted on 04/22/2010 9:26:37 AM PDT by Battle Hymn of the Republic

Our community has a unified disaster system. We have several organizations, local government, county government, city government, hospitals, school district and businesses involved in Disaster Planning and Response. Because we are in the northwest corner of the state of Iowa with border neighbors in Nebraska and South Dakota we often have regional exercises. Several times a year we have Disaster Exercises where all of our teams "play together".

Today was one of those days. At 8AM this morning the team started to gather at the local event center to prepare for the arrival of the exercise "victims". The victims were made up of students from local high schools and colleges and a few "adult chaperone" victims. The scenario was to be a Bioterrorist event at a sold out concert at the local event center. All of the players arrived and were briefed on the activities of the day. At precisely 9AM the exercise began. The first call went out to our 911 Center to notify them that an event was unfolding at the local event center. Information was being relayed to the 911 operator that something was going on at the Event Center with approximately 130 victims exhibiting various breathing/respiratory symptoms. The 911 operator was going through their normal fact finding questions when about 3 minutes into the call the 911 operator indicated that her computer had just quit. She was about to transfer the call to another dispatcher when all of the computers in the 911 center began to power down. At this point they knew something was going on but just not sure what.

Our on scene team at first thought that this was someone's idea of adding a little twist to the exercise. The 911 operator assured us that it was not. A call was made to the IT department and the 911 center soon discovered that the problem was not limited to their computers but that computers all over the system were shutting down. The local county and city governments share the network, resources and support staff for the computer systems. They began getting calls from city and county employees from all areas, police, fire, emergency management, financial, HR, etc. The first thing that came to mind was that a worm/virus was wrecking havoc on the City/County network. They began an emergency shutdown of all equipment in the network to prevent spread and additional damage from being done.

About an hour into their investigation they discovered that the culprit for the shutdown was not a worm/virus but an update that was being pushed out for the McAfee Antivirus program. The IT staff will have a long night tonight getting all of the machines that were damaged repaired and ready to go for the morning startup. They expect to have 80% of the machines backup by tomorrow morning and 99% back up by lunch time tomorrow.

So you may assume that the loss of the 911 Center caused the Disaster Exercise to be called. After all, how can you have a Disaster without your 911 Operators, Right? Not us. When the 911 Center went offline at 9:05am we had to decide if we were to continue the exercise or call it due to the loss of 911. Our EMS Director for the County decided to continue the exercise. He began to do dispatch and communication using our 800Mhz shared radio system. We continued the exercise, decontaminated and transported roughly 120 people to the local hospitals. We successfully completed the exercise at 11 am.

While we were in the Hot Wash Debriefing we received a call letting us know that it was not a worm/virus but was the McAfee update that caused the entire City/County to come to a screeching halt. Many of the individuals in the debriefing grabbed cell phones to call back to the office with the news of what happened. For a few it was too late, the updates had already run and their organizations too were experiencing the same problems. For those that hadn't updated yet the updates were turned off. Others were relieved to find out that they were using the competitors AV and were not in any danger.

Thanks to McAfee we were forced to test our response to a Disaster while in the midst of a real "disaster". The positive that came out of the exercise is the fact that we had a successful exercise while using our "backup" communication system. It was a true test of our ability to adjust to and respond to a disaster in less than perfect circumstances. Isn't that really what our goal was? We all know that many "disasters" having multiple components and today we saw firsthand how true that is.


TOPICS:
KEYWORDS: emergencyresponse; mcafee
interesting follow-up from yesterday's debacle.
1 posted on 04/22/2010 9:26:37 AM PDT by Battle Hymn of the Republic
[ Post Reply | Private Reply | View Replies]

To: Battle Hymn of the Republic
Why ANYONE would still have McAfee on their computer puzzles me.

Way back in the dawn of PC’s, McAfee was about the only one. It ALWAYS messed up my computer. Always locked it up - and never did that good a job protecting either.

I got rid of that thing back in the ‘90’s

2 posted on 04/22/2010 9:30:30 AM PDT by maine-iac7
[ Post Reply | Private Reply | To 1 | View Replies]

To: maine-iac7
Why ANYONE would still have McAfee on their computer puzzles me.

It came free with the computer.

3 posted on 04/22/2010 9:31:42 AM PDT by madison10 (Current guy in the White House: If he breathes, he's lyin')
[ Post Reply | Private Reply | To 2 | View Replies]

To: Battle Hymn of the Republic

Because I had problems with Symantec on a previous computer.


4 posted on 04/22/2010 9:32:51 AM PDT by knittnmom ("...only dead fish 'go with the flow'". - Sarah Palin 7/09)
[ Post Reply | Private Reply | To 1 | View Replies]

To: maine-iac7
Why ANYONE would still have McAfee on their computer puzzles me.

Our corporate IT pushes various McAfee crap onto my work machines. I shut down the McAfee services and processes each morning, and turn them back on before leaving in the evening. It's the only way I can get any work done.
5 posted on 04/22/2010 9:34:35 AM PDT by AnotherUnixGeek
[ Post Reply | Private Reply | To 2 | View Replies]

To: Battle Hymn of the Republic

I wouldn’t call it a debacle at all. I’d call it an excellent example of real world action. And the director of the exercise should be commended for continuing it, providing that he dismissed anyone who was needed to deal with the actual computer crash. Real world problems are often made worse by unforseen complications. Somebody who could engineer a biologic attack at a concert could also cause the 911 system to crash.


6 posted on 04/22/2010 9:37:37 AM PDT by sig226 (Mourn this day, the death of a great republic. March 21, 2010)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Battle Hymn of the Republic
Lemons, lemonade. Shouldn't have happened but anything that encourages improvisation is money made.

I still have McAfee on one old laptop in a closet. Before it hits the network again, McAfee's toast. Too bad, too - once upon a time it was a great product, but that was when it was distributed on five-inch floppy disks.

7 posted on 04/22/2010 9:37:37 AM PDT by Billthedrill
[ Post Reply | Private Reply | To 1 | View Replies]

To: Battle Hymn of the Republic
Linux can prevent most disasters, just use 64 bit processors with the hardware ‘no execute’(NX bit) enabled — This segregates code from data automatically. About 95-98% of all of the virus code, due to buffer overruns, is prevented by this simple act. The big boy computers have known this for 30 years, all separate out data from code with hardware fencing.

The best is due April 29, Ubuntu 10.04.

What is left is easily fended off with proper computer management, things such as never run your computer in Admin mode, always use standard user mode, for users, as it was designed to do.

8 posted on 04/22/2010 9:37:47 AM PDT by Tarpon ( ...Rude crude socialist Obama depends on ignorance to force his will on people)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Battle Hymn of the Republic

This shut down the computer system at the major teaching hospital where my husband works. I haven’t heard yet how things are going today.


9 posted on 04/22/2010 9:38:17 AM PDT by stayathomemom (Beware of cat attacks while typing!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Battle Hymn of the Republic
McAfee is toast.

After this debacle, any IT administrator that doesn't immediately start a plan to migrate away from McAfee products should be fired.

10 posted on 04/22/2010 9:39:50 AM PDT by justlurking (The only remedy for a bad guy with a gun is a good WOMAN (Sgt. Kimberly Munley) with a gun)
[ Post Reply | Private Reply | To 1 | View Replies]

To: madison10
It came free with the computer.

And it's worth exactly what you paid for it.

11 posted on 04/22/2010 9:40:57 AM PDT by Lurker (The avalanche has begun. The pebbles no longer have a vote.)
[ Post Reply | Private Reply | To 3 | View Replies]

To: knittnmom

I paid for McAfee for years and thought I was doing pretty well, only getting about one virus a year. Then, in a short period of relative poverty I switched to the free AVG. That was a maybe 6 years ago and I have not had a virus since.


12 posted on 04/22/2010 9:42:02 AM PDT by arthurus ("If you don't believe in shooting abortionists, don't shoot an abortionist." -Ann C.)
[ Post Reply | Private Reply | To 4 | View Replies]

To: maine-iac7

We had McAfee installed on our pc’s at home. All was fine until we upgraded to 2010. Big mistake. Needless to say, I uninstalled and went with Kaspersky and it seems to be much better.


13 posted on 04/22/2010 9:42:03 AM PDT by Grumpybutt
[ Post Reply | Private Reply | To 2 | View Replies]

To: AnotherUnixGeek

smart plan


14 posted on 04/22/2010 9:46:30 AM PDT by maine-iac7
[ Post Reply | Private Reply | To 5 | View Replies]

To: AnotherUnixGeek

I don’t hold a brief for McAfee but we have used it for years in our 10,000 strong company without noticeable issue. It’s supposed to generate its own random update intervals - obviously that didn’t happen in this case. Maybe an admin’s mistake, not the software itself.


15 posted on 04/22/2010 9:49:00 AM PDT by agere_contra
[ Post Reply | Private Reply | To 5 | View Replies]

To: madison10
There are plenty of GOOD programs available FREE - of course, one would have to bestir themselves for a minute or two to install them.

And we have become a society that likes everything done for us and given to us - no matter how destructive it is in the long run

16 posted on 04/22/2010 9:50:04 AM PDT by maine-iac7
[ Post Reply | Private Reply | To 3 | View Replies]

To: Battle Hymn of the Republic
So you may assume that the loss of the 911 Center caused the Disaster Exercise to be called. After all, how can you have a Disaster without your 911 Operators, Right? Not us. When the 911 Center went offline at 9:05am we had to decide if we were to continue the exercise or call it due to the loss of 911. Our EMS Director for the County decided to continue the exercise. He began to do dispatch and communication using our 800Mhz shared radio system. We continued the exercise, decontaminated and transported roughly 120 people to the local hospitals. We successfully completed the exercise at 11 am

EMS Director made exactly the right call. You don't get to plan the date and/or time of an emergency. You also don't get to plan the *other* failures that inevitably follow.

The only way he could've done it any better is to say that 2 of the top 3 people in the system were out of town or at the concert and presumed dead.

17 posted on 04/22/2010 9:50:19 AM PDT by Terabitten ("Don't retreat. RELOAD!!" -Sarah Palin)
[ Post Reply | Private Reply | To 1 | View Replies]

To: justlurking

The most effective plan would be to migrate away from Windows.


18 posted on 04/22/2010 9:51:44 AM PDT by Goldsborough
[ Post Reply | Private Reply | To 10 | View Replies]

To: arthurus
AVG is EXcellent. It catches bugs and chews them up for lunch.

Kaspersky is good also

19 posted on 04/22/2010 9:52:11 AM PDT by maine-iac7
[ Post Reply | Private Reply | To 12 | View Replies]

To: Grumpybutt

Yes - Kaspersky is great - I use both it and AVG


20 posted on 04/22/2010 9:52:54 AM PDT by maine-iac7
[ Post Reply | Private Reply | To 13 | View Replies]

To: Battle Hymn of the Republic
This is a good example of why a big company needs to not have all their computers on one antivirus software.

It would cost more but if a company could divide up their systems between two or more AV vendors then something like this wouldn't take down the entire enterprise.

21 posted on 04/22/2010 9:53:49 AM PDT by FReepaholic (I'm in my head and can't get out.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: agere_contra
It’s supposed to generate its own random update intervals - obviously that didn’t happen in this case. Maybe an admin’s mistake, not the software itself.

This event took out my computer yesterday morning. My understanding from our IT guys, is that if you were scheduled for an update of virus definitions yesterday between a certain time period, it got you. Apparently, among the new virus definitions, it listed a critical windows file among them.
22 posted on 04/22/2010 9:56:11 AM PDT by ZX12R
[ Post Reply | Private Reply | To 15 | View Replies]

To: Battle Hymn of the Republic

Kudos to your EMS Director, especially for having a viable backup plan when 911 service goes away. Part of dealing with an emergency is the other failures that occur along the way. It’s what makes it fun, right?


23 posted on 04/22/2010 9:58:56 AM PDT by FourPeas (God Bless America)
[ Post Reply | Private Reply | To 1 | View Replies]

To: arthurus

I set McAfee to update, then scan, then shutdown if it did not find any virii last night. I have to wait until I get home to find out if my Vista pc still works.


24 posted on 04/22/2010 9:59:52 AM PDT by knittnmom ("...only dead fish 'go with the flow'". - Sarah Palin 7/09)
[ Post Reply | Private Reply | To 12 | View Replies]

To: AnotherUnixGeek
Our corporate IT pushes various McAfee crap onto my work machines.

Same here. At my company, nearly 3,000 computers lost network connections within 30 minutes. People were going around individually this morning with the patch file on a USB drive. McAfee uses up a lot of memory running in the background. Glad I use AVG at home.

25 posted on 04/22/2010 10:00:49 AM PDT by IndyTiger
[ Post Reply | Private Reply | To 5 | View Replies]

To: raven92876

ping


26 posted on 04/22/2010 10:05:08 AM PDT by onedoug
[ Post Reply | Private Reply | To 1 | View Replies]

To: stayathomemom

The good news is that I got rid of McCaffee on my home systems a week ago ;-)

I have some experience with creating, running, and participating in emergency drills. This is could have been part of the script of some of the drills - so it was as real as you can get because it WAS. At the same time it goes to show that a mono-culture is just as unhealthy for machines as it is crops. One “virus” can take out 100% of the crops or your computer systems.

Hopefully, part of the lessons learned is to change out half of the McCaffee served machines with a competitors product across the entire computer environment. This way the 100% scenario can’t hit them again.

I run Windows/Linux at home so that I don’t have a single point of failure. (Even have alternate Internet connections available if needed...)


27 posted on 04/22/2010 10:37:28 AM PDT by fremont_steve
[ Post Reply | Private Reply | To 9 | View Replies]

To: maine-iac7

I got to contribute to the list of alternate antiviruses.

NOD32 is a great one.


28 posted on 04/22/2010 10:40:13 AM PDT by benjibrowder (For Neda. May God bless those fighting for freedom.)
[ Post Reply | Private Reply | To 19 | View Replies]

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
General/Chat
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson