Tuesday, May 3, 2011

Terrible Storm and RTEMS Outage

Surely by now, you have noticed that the RTEMS Project appeared to drop completely off the face of the planet about 6pm CST April 27.  It was at this time that the third storm system moved through north Alabama and knocked out all major power transmission lines.  This blog is a first in a series to let you all know what happened and what is happening now.

The primary servers and lab machines for the RTEMS Project (.org and .com) are in Huntsville Alabama which was in the path of the storm April 27th 2011. This storm  killed 340+ across multiple states and left a huge trail of destruction.  I have heard reports of business signs being found 100 miles (160km) away.  The following is a nice weather summary without the heart wrenching photos of the death and devastation.


Huntsville had three systems pass over it that day. The first was nasty but no issues impacting the server or our home. The second system resulted in water getting into the rtems.info server area but we still had power. This allowed us to clean up and get the server back online until the third system hit. The third system was the killer. It was the devastating one that wiped out communities from Mississippi through Alabama and Georgia to points further north. It destroyed the major power transmission lines into north Alabama and Mississippi. Local utilities get power from the Tennessee Valley Authority (TVA) and TVA could not supply power to them. Their blog is here with details:


I was teaching an RTEMS class during this with the sole attendee being a wonderful fellow from the UK.  We spent much of Wednesday in a safe area inside OAR.  And once the storm had passed and we realized we were the lucky ones, we finished the class without power.  His hotel room was wet and without power but his bed was dry.  He could charge his laptop from the car.  We took a table and a couple of chairs from OAR and sat outside the door.  When the sun moved and we got hot,  we moved the furniture.  At one point, we were on the other side of the parking lot.  We had nothing else to do and a dusk to dawn curfew, so we followed the class material, chatted, drank soda, etc.. Just chilled and did RTEMS stuff.  Phillip deserves a big thank you for helping me make sure all was turned off Thursday and cleaning the fridge and freezer Friday. I sent him to Chattanooga for the weekend and I hope it was some nice relaxing site-seeing.

Friday, my family went to a hotel in a neighbouring city and waited for power to be restored. We came home Monday afternoon since our home had no apparent damage and power was restored.  We are near the main hospital so we usually get power early.  I started ensuring the rtems.info and elviscostellofans.com server came back up OK. Michele is cleaning the fridge and freezer out. If we didn't lose any electronics due to power spikes, then that's all I think we have. That makes us very lucky. Michele and I know people with deaths in their families or homes destroyed. Cleaning the fridge looks pretty tame in comparison.

We tried to stay in touch using our cell phones for email but the towers died about 12 hours in.  Plus if we didn't know you on Facebook or via our private emails, it looked like we disappeared.  I apologize for not remembering linkedin and the RTEMS facebook group.  I didn't even remember IRC until Friday when we got to the hotel.

Huntsville Utilities announced yesterday (Monday night) that they have done all they can.  Everywhere TVA has given them power has been passed on to residents.  Only 30% have power.  I believe that Redstone Arsenal, Marshall Space Flight Center, and Research Park will be the last in the area to be restored.  They consume a LOT of power between them and it is more important to get power back to houses.

We really appreciate the good karma that was sent our way. We are both tired and frazzled but that's no biggie.


  1. Hi Joel,
    Glad to hear you and your family made it through the storm OK! Been checking to see if any rtems sites back online but still not. Finally did a google search for "rtems tornado" and your blog popped up.


  2. I can't believe RTEMS and tornado was a google hit. :D

    I must admit that the gravity of the storm and power outage took a while to sink in. With rtems.org down, it never occurred to me that there are a few other ways to get news out. The topic on the IRC channel, this blog, facebook RTEMS group, and linkedin.

    I should have done that before the cell towers died here. I hope there isn't a next time but there are times when things are down and I will now remember how to deal with them.