NY's Power Outage & Public Safety Radio Services

[Letitride]

As night fall, NY and the rest of N.E. went into complete darkness, and public safety services (police, fire & ems operations) went on high alert and major disaster/emergency prepareness mode, calling in backup personnel, mobilizing reserves & putting established contigency plans into operations. By and large, thanks to numerous drills & training, things went quite smoothly & we've survived. But, we also learned a few surprises, that a lightning strike or what not can triggered a wide spread system shutdown, effectively shutting down 50 million people - taking down cash registers, ATM terminals, airport operations, trains & subways, etc. when electricity is lost. This happened a day after the computer virus attack on many systems. By chance, we've noticed unusual phone network & internet issues about 2 hours prior to the massive power outage before 4:30PM yesterday.

In any event, public hospitals went on emergency generator power in the entire region & almost everyone transitioned smoothly. Like others, our medical center in southern Brooklyn activated the major incident/disaster code and proceeded as planned, and with power being restored around 3:40AM this morning, things are now back to normal - but on alert still.

At the height of last night's power outage and as darkness fell, NY's finest (police dept) radio frequencies reported major trouble with their Motorola trunked/repeater radio system with 911 dispatch & central command, warning all members of service (MOS) over the air that radio transmission could be lost at any time without notice, that they are to switch over to short range "point-to-point" radio communications, essentially that they are on their own & they will be out of touch with the rest of the borough (county) and city. EMS (ambulance) frequencies also had issues getting through. Fire dept. dispatched operated on VHF and apparently had less problems. Monitoring & listening to all this on our police scanners & radio systems. On top of that, public services also have Nextel PTT cell services and they failed too, reported as not working at all - for example, out completely in the Bronx & Staten Island - very unassuring. to say the least. Of course, the general public by & large were unawared and it was a fairly calm night, with police out in force, some heavily, heavily armed in anti-terrorist mode in full gear.

If this was another serious incident with mass casualties, the results could be as tragic as 9/11/01 - their vehicles were beginning to run out of gas & only a few gas stations were pumping. All I can say & recommend at this point is that, check that "Go & Run Kit" and prepare for another major incident, restock on batteries & edible food supplies plus drinking water, and check your personal & family contingency plans" - and have backup means of communicating in case regular & cell phone services are down or seriously affected. All it takes is a lightning strike - or is it that simple ??

[njsvrzncdma]

i hope its not that simple. If it is we are all in for it sooner or later. There are lightning storms all the time and if something as simple as that can wipe out 50 million people then tax payers money is going to the wrong places.

[MrFlashport]

As a radio system manager, I can say this: never put all your eggs in one basket. Our agency uses a 12 channel SmartNet II 800MHz simulcast trunked system, with five sites and around 2,000 radios. If you lose the central controller, your repeaters operate in a conventional mode called failsoft, where radios "fail" to a pre-determinded repeater. That's bad enough. You lose power or your sites completely- your screwed. Does it happen? Yes. We have issues with Nextel interference, and we are under capacity. When we go before our county commission, radio systems are the LAST things to get funding. We are still waiting for some of that 98 billion dollars that Mr. Bush promised us back in 2001 so we can upgrade to Project 25...hello Washington..WHERE'S THE MONEY?

The bottom line is: HAVE A BACKUP. I spearheaded a project to bring our old VHF system back online. Sure, it is 50 year old technology, BUT IT WORKS. We had an old 3 channel repeater system, old Micor Repeaters at that, and they are now back in service, along with the MT500 and HT600 portables (what didn't get auctioned off). The repeaters are on a generator backed American Tower site, so they will function if commercial power is lost. Most recently, I added the old VHF control lines back to the consoles, so they are ready if the 800 system fails.

But you would be amazed at how many major cities, especially those who invested in analog and digital trunking systems, abandoned their old conventional VHF and UHF radios, in alot of cases even surrendering their old licenses back to the FCC. BAD IDEA. Ask Washington DC how well their Astro P25 digital 800 system held up during 9-11. It barely works inside the Pentagon Mall right now, because of a government that only ordered a system for 80 percent coverage! (They didn't have this problem with their old GE UHF system mind you) But Motorola wanted to make a sale, so they sold them less than what was required. Lots of politics that goes into public safety communications. I say, follow the KISS rule. If it works don't fix it. I find it ironic how we are told by sales people and consultants (most of whom are on the payroll of Motorola or Nextel) to go digital, go trunked...spend money. Then, when all the proverbial poop hits the fan, the "new state of the art" digital radios and trunking systems fail and we are shouting "Can you hear me now?" into our 5,000 dollar XTS3000 portables, we end up going back to what always worked for over 50 years: conventional FM VHF and UHF radios.

Same thing with cellular/PCS. Most of the legacy carriers (like Verizon and Cingular TDMA) have old established systems, alot of their sites (at least with Verizon in Atlanta) are on backup power and still have their old microwave paths for standby. But don't count on it. Commercial systems will fail or become severely overloaded. The ultimate backup is amateur radio. I have an HF thru UHF station in a small box (a Icom 706 MK II) with a battery backup. It is completely self contained. It works no matter what, the only exception would be a nuclear disaster (which would render any solid state device useless anyway).

I say it again, HAVE A BACKUP. If you want a backup comm system, consider getting a ham license. Even FRS/GMRS radios are good for something, albeit limited range without repeaters.

[Letitride]

Local update: 80% of our county (borough of Brookly, NY) with 2 million people - twice the size of metro Los Angeles - still without reliable power (hello: Con Edison & Keyspan) and only 50% of NYC with electricity, 24 hours after the power outage. Our hospital security dept. just reported finishing checking all backup batteries, power generators, UPS, cellphone & 2 way radio batteries, etc. and prepared for a long weekend. From operations command on duty to key personnel - nobody going home without rechecking their assigned pagers (what's that, ha!) & given spare batteries, at no charge. Just added another page to our major disaster plan & distributed by fax & email along with hard copies to all department chiefs.

Well, 50 million American from multiple N.E. states participated in the largest post cold war disaster prepareness training & barely got a passing grade for this drill. The next one might be a lot more challenging. If this massive shutdown was the result of a simple lightning strike or computer virus and it triggered this kind of outage, we are in for serious, serious, serious trouble ahead. N.Y. Governor is mad as hell & wants heads to roll and someone to pay for this mess, hello - Office of Homeland Security & Mr. President, can you hear us know!

Even now, from my monitoring station, I can hear reports of the 800Mhz trunked system by Motorola not working at 99%, radio checks are sometimes coming in only 4 by 4, getting cutoff completely at other times. Part of the problem is that these fleet vehciles had only digital radio systems installed - the old UHF & portables were warehoused when they upgraded. Police & EMS managed to stay above water operating on hi-power UHF & VHF. Utilities & public services got about 4 hours to get more grids running before night fall - some places expected to be out for the weekend. In fact, many people simply stayed home instead of going into work and found that they couldn't work from their house or apartment cause lines are down and laptop batteries ran out on them. What a lesson!

Ham radios are great except most people lack the technicals, and the local 2-meter groups were (and still are) busy on backup power - doing the essentials. I've just checked batteries my own GMRS radios and reprogrammed the Motorola UHF portables to operate on one of the frequency in low power mode, just in case - for in house operations. Meanwhile, we are getting our notebook pc for portable wirless operations to backup our T1 and analog lines.

[MrFlashport]

When you speak of the 800MHz system, are you talking about the NYC Office of Emergency Management system? I thought FDNY migrated to UHF T-band Astro P25 and that NYPD were slowly converting to Astro UHF, but they were dead set against trunking (smart move IMO) like LA because they wanted to AVOID failures like this...

The OEM's Smartzone system has been plagued with problems since they got it. It was recently written up for being the latest "victim" of Nextel interference in MRT. Why they went 800 when everyone else (PD/FD/PAPD) was on UHF or VHF is yet another exampe of OVERSELLING NEEDLESS EQUIPMENT by Motorola and friends.

And my heart goes out to you guys up there. This is what can happen when we use a power distribution system designed in the 1950's to power a 21st century world. I think I am going to invest in a Generac, and plenty of diesel!

[njsvrzncdma]

im with you on the generator idea no matter what it will still work ! (provided it was maintained and has fuel ! )

[Letitride]

Being a public forum, let's not get too technical or into specifics. Suffice to say, the high tech system OEM & others put together didn't worked as envisioned. About 24 hours ago and then at sunrise this morning, warnings were issued from command to field units that they are to be prepared to operate in "stand alone" mode should communications failed. That is, good luck & you will be on you own soon, without warning. Gasoline was running low for fleet vehicles & fuel for backup generators also low this morning due to the extended power outage. Thank God, field units were able to find working landlines & call in to local station houses, still using 70's style old AT&T 5 button multi-line analog phones. Never mind those Merlin system or PBX exchanges. It's amazing how those old fashion low-volt line powered twisted pair wires can do on its own. Of course, media did not and will not report on these & the public will never learn about these serious gaps & plan failure. Conversion has been slow since Uncle Sam hasn't written the check in full - coupled with budget considerations.

Power backup and generators are good only if well maintained, inspected and stocked with fuel. A hospital in nearby suburb didn't have enough reserve to keep its b/u running because of budget issues. The head of engineering & chief operating officer, if they're working for our facility, would be rewarded with a long vacation on a one-way ticket. Just cannot believe that so many gaps still exist on this day & age. The disaster specialist, the American Red Cross was nowhere in sight until today, announcing that they've opened 5, that is FIVE cooling centers in the 5 boroughs (counties) of NYC. Last night, out of town visitors were thrown out of the Marriott and told to sleep on the sidewalks of NYC since electricity & water were out. Finally coming home tonight & 2/3 of the gas stations that I drove pass around town (that's about 25) had run out of gas and it's cash only - credit card terminals down, I'm told mostly. It made my colleague & I wondering, how much better & prepared we really are since 9/11/01.

[DeathRider]

well, according to this article, there's a list of who to blame:

http://gregpalast.com/printerfriendly.cfm?artid=257

[Letitride]

Well said (written) on the Texas connection. Deregulation means no regulation, plain & simple. Wall Street "experts" already sounded the forecast that gasoline cost will continue to rise, along with the price of natural gas & other energy costs this year. Isn't this great for stimulating the economy, creating jobs & why the tax cut is good for America??? No wonder those Hummer by GM are doing so well in sales. Really, did I miss something!

[MrFlashport]

Originally posted by: Letitride Being a public forum, let's not get too technical or into specifics. Suffice to say, the high tech system OEM & others put together didn't worked as envisioned. About 24 hours ago and then at sunrise this morning, warnings were issued from command to field units that they are to be prepared to operate in "stand alone" mode should communications failed. That is, good luck & you will be on you own soon, without warning. Gasoline was running low for fleet vehicles & fuel for backup generators also low this morning due to the extended power outage. Thank God, field units were able to find working landlines & call in to local station houses, still using 70's style old AT&T 5 button multi-line analog phones. Never mind those Merlin system or PBX exchanges. It's amazing how those old fashion low-volt line powered twisted pair wires can do on its own. Of course, media did not and will not report on these & the public will never learn about these serious gaps & plan failure. Conversion has been slow since Uncle Sam hasn't written the check in full - coupled with budget considerations. Power backup and generators are good only if well maintained, inspected and stocked with fuel. A hospital in nearby suburb didn't have enough reserve to keep its b/u running because of budget issues. The head of engineering & chief operating officer, if they're working for our facility, would be rewarded with a long vacation on a one-way ticket. Just cannot believe that so many gaps still exist on this day & age. The disaster specialist, the American Red Cross was nowhere in sight until today, announcing that they've opened 5, that is FIVE cooling centers in the 5 boroughs (counties) of NYC. Last night, out of town visitors were thrown out of the Marriott and told to sleep on the sidewalks of NYC since electricity & water were out. Finally coming home tonight & 2/3 of the gas stations that I drove pass around town (that's about 25) had run out of gas and it's cash only - credit card terminals down, I'm told mostly. It made my colleague & I wondering, how much better & prepared we really are since 9/11/01.

The bottom line is "homeland security" is nothing more than a dog and pony show meant to accomplish several goals: (not neccesarily in this order either) 1)- to get governments to spend more money (thus raise taxes), 2)-to pass new redundant laws, and 3)- to give it's citizens the APPEARANCE that we are all "doing something". With felons and non-citizens working for the TSA, illegal immigration at an all time high (which is BTW, where are "terrorism" problem came from in the first place) and all we get is a bunch of ear candy from Bush and his people about how these magical grants and new laws will "protect" us and assist us in providing for better first responders. Yeah, well WHERE'S THE MONEY? We have been waiting for over two years for our grant to come through. Our agency is not unlike the thousands across America, we operate on a shoe string budget, using outdated equipment and lack the funding to send everyone to Emmitsburg for those great FEMA EMI classes that cost thousands. Then who is gonna cover our territory while we are away? If most citizens knew just how close every single incident such as this grid failure is to crippling our public safety resources are, there would be alot more recalls than just Mr. Davis in California!

This may not be a forum for this, but I had to vent. I am just disgusted with Washington and it's perpetual load of B.S. being thrown in our faces. Some of us who work in your local government really DO care about our community; believe me, I don't do it for the money. I don't get wooed by lobbyists with 500 dollar a plate dinners, and I don't get a call girl service either. I work hard, I love radio and communications and made it my calling to put my skills to work for the community in which I live. Ever since 9/11/2001, the Feds have made a grand stand big deal out of how much local infrastructure needs to be fixed. Well thanks for noticing Mr. Ridge, Mr. Rumsfeld and Mr. Bush. It has only been this way for the last three decades. And you all have all the answers, right? I am really sick of it...it is really patronizing and insulting to those of us who do the REAL, ACTUAL WORK of keeping these systems alive from day to day to have to tolerate these scumbag politicians tell the public how hard THEY are working to provide us with funding and support for our infrastructure. If that were the case, the big blackout would NEVER HAVE OCCURED, and don't even get started on 9/11/2001- that could have been prevented had people been doing their jobs. If the public only knew...

To get back to the point, Letitride, you are right. Having equipment is only as good as the personnel who maintain it and have the training to use it properly. It kinda goes without saying that if one buys a generator system they should bother to do the work to maintain and operate it properly. But again, the training and equipment come with a cost, in both money and man hours. Most agencies can't afford it- bottom line- it's about the money. I encourage every citizen to do what you can to protect yourself. Take CPR and First Aid traning, it is free in alot of communities. Know what to do when in harms way. I am VERY amazed with how well citizens of NYC come together and KNOW WHAT TO DO when these things happen. I saw citizens directing traffic, providing shelters, and generally helping each other out. Now that simply amazes me. It goes to show you that the power IS in the people. I still believe we can overcome anything, if we as citizens stick together and work together. Who needs Washington and their used car salesmen.

Off of soapbox.

[Letitride]

Well, the EMS union blew it wide open on the COMPLETE failure of their dispatch & UHF radio system during the outage and of stories of being relying on FDNY radio - mix of VHF & UHF system to help protect lives, being stationed at local fire houses so that messages can be relayed & hi priority calls answered. (The N.Y. Newsday article is just another wakeup call). Response times were typically under 5 mintues for high priority / cardiac cases & they were held up in the system for 30 minutes - fortunately, only 1 live was lost. Mutual aid (private/voluntary ambulances, mostly BLS units) responded to the call for aid, just like 9/11 but simply did not have the UHF radios to properly communicate so they're useless in more than one way.

As cited in the article, the backup power generator at world famous Bellevue Hospital (city runned) and St. Vincent (voluntary) - 2nd. closest to WTC - in downtown Manhattan were both level I trauma center and they've not operating properly because of fiscal issues. If they're really called upon to triage & treat massive casualty on Thursday, I wondered what they would've done - declare a diversion to other ER, leaving the much smaller NYU Downtown Hospital to handle city hall & the Wall Street financial district - that place also in serious financial distress. Then, the state-owned & runned teaching hospital (SUNY Downstate Medical Center) also acknowledged its generator problem on Thursday, which covered one of the poorest neighborhood in Brooklyn of 2 million people. I'm still stunned of all the systems failure we experienced in the emergency response system as other reports begin to come in and in calls to my colleague at other health care facilities. The broader picture is just slowly emerging now among us. I serve on an ongoing task force on life & fire safety as the 7 Environments of Care, national standards set by the Joint Commission on Accreditation of Health Organizations or JACHO, and when we've done with our analysis, findings & recommendations, we'll make sure heads will be turned, if not rolled or get dropped.

http://www.nynewsday.com/nyc-ems0816...headlines-left

For the average, ordinary citizens in NY and in other cities & towns, it's just amazing how we pulled it all together in literally complete darkness. Thousands slept outdoor in the Penn (rail) Station. By 4:15PM, our medical center called the disaster code, having flipped on our portable radio & tv sets on battery power, issued gears & flashlights, and cleared the ER entrance - and amazingly, many staff on the day shift (off at 4:00PM) came back into the facility on their own as our roof top power generators fully kicked in. All of the training, drills, preparation & experiences learned from 9/11/01 and since then paid off. Let's pray what happened 2 days ago won't happen again cause they still don't know what & how it happened, and not expected to pinpoint the chain of events until sometime next week - it's now just finger pointing.

Oh, God Bless America!

[jones]

the pay phones worked all throughout the blackout.

[bobolito]

In the next few weeks all we're going to see is just a blaming game. Right now, all components of the system are simply entertaining Americans throwing the ball at each other while they'll figure out a way out of this mess. Of course, nobody wants to take the blame and hold the responsability of the weight of 50 million people that went into darkness and all the lawsuits that will result from all this mess. On Friday, the US was fingerpointing Canada, then Canada fingerpointed the US. Later on Canada blamed a lightning strike in a Niagara plant. Weather analysts in the US said they did not detect any storm activity in the Niagara area at shutdown time so they ruled out that possibility. Then, during the weekend all the eyes began to steer away from Niagara to focus on Ohio and the Midwest. Now, Ohio says it was not them, but somewhere else in the Midwest. So whoever/whatever the finger points next, I am sure they will be quick to find an excuse to wash their hands from this game. In the meantime, we are all simply being entertained watching this "ball game" so let's not expect any conclusions or outcomes in the near future. They need to plan this very carefully and figure out what they are going to tell the American people.

[Letitride]

NYC Mayor Bloomberg announced over the "weekend" that come "Monday morning" that he will appoint a(nother) task force to study what worked & what didn't, to understand & prevent failures from happening again in the future.

My colleagues at work & counterparts involving in life & fire safety are just glad that by & large, 50 million Americans passed the live disaster drills & training exercise last Thursday, and will be learning from it. Thank God, it wasn't a terrorist attack & that no major disaster or incidents occured where lives were lost or SERIOUSLY at risk. At least NYC is open & public about its shortcomings, initially only in the EMS system but then confessed to problems with the PD & 911 dispatch system, and about the backup generator power problems at quite a few hospitals. With massive power outage, hospital emergencies could've been called upon to handle mass casualties & trauma cases and discovered that the backup systems failed.

The deregulated utility companies have been profitable but have NOT invested in infrastructure building for 30 years, and now, we the American citizens & taxpayers are going to be paying for another round of price increases in our energy bills to pay for it. I guess we need the U.S. Energy Secretary to tell us that, huh! That's really, really responsible government at its best!

An interesting note - on my Saturday visit to the local Target store, all the available flashlights are gone from the shelves - including the designer series from Eddie Bauer. But at least they still have enough & variety of batteries in stock - time to resupply ourselves & be prepared on your own. Back to business as usual.