Register

Welcome to the RDI-Board Community.

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed.


Donate Now Goal amount for this month: 100 EUR, Received: 100 EUR (100%)
Donate to support this site...

Page 1 of 2 12 LastLast
Results 1 to 15 of 30
  1. #1
    Banned
    RDI - Board Banned Avatar

    Join Date
    Apr 2003
    Posts
    28
    Posts Thanks / Likes

    Default RDI is back online after a fire and an explosion :)


    The outage affected 9000 servers and 7500 customers. “In 31 May at 4:55pm CDT in data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost,” Kevin Hazard said during his update post on the forums over at The Planet. “We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.”
    As fast as *** ****** could get them there, vendors, all members of the support team, and senior staff were on site. Kevin’s update on the issue kept up all night, and after he was done, Todd Mitchell stepped up and took over the updates process. One of the things you need with a Disaster Recovery Plan (DRP) is communication. According to those who are obviously not familiar with IT, mostly dime-a-dozen hosting resellers, The Planet is not known for effective communications. It is obvious they are effective, as the media, clients, and average people know what happened and why, and what was done about it.
    The typical comment looked like this yesterday evening, “The DC can run off of the power provided by the backup generators. Why isn't the DC Running off [sic] backup power? I assume the switch was something connected via the street/pole that blew. If it was a "high capacity" transformer/switch why is the entire building in the dark, including every router without an real idea of what's going on 24 hours later?” This comment came from a well-known webhosting forum.
    Instead of discussion over IT, policy, and Business Continuity Planning, the forum is mostly complaints about the outage. The fire department ordered power cut; this rendered the instant usage of backup generators moot.
    Swa Frantzen over at SANS made and interesting comment on the issue, “I had seen plans for BCP/DRP derail before due to officials stepping in and doing their response to an emergency in their way and not in the way the organization itself had planned it. I think it would be interesting for most of us to actually talk to fire departments and/or police officers on what their normal responses are and take them into account in our plans...”
    No one is sure why the fire and explosion happened, lots of people offer various theories, but those are all well meaning guesses. The fact is, as soon as disaster struck, The Planet took action.
    Here is what they did right:
    Communication – It was constant, and timely. They made everyone aware of what they were doing and why.
    Assessment – With the help of staff and vendors, The Planet was able to assess their equipment and see what, if anything, was needed to start the process of full recovery. “As you know, we have vendors onsite at the H1 data center. With their help, we’ve created a list of equipment that will be required, and we’re already dealing with those manufacturers to find the gear…”
    Priorities – On the forum where updates were posted, The Planet listed four items that were the top priorities for the recovery. It is likely this list was made onsite as the disaster was entering the cleanup phase.
    Resource sharing – After the network in the H1 center was assessed, they started a migration process for management controls and other critical functions. They began using resources from their other datacenters to help speed up the process.
    If you were to criticize The Planet, it would come under the communications part of their plan. There is no news on their blog or their main page about one of their datacenters being down. While they are using the forums to communicate, the other areas of notice should have some attention as well.
    A company disaster plan should start by one simple rule. What is it your business needs IT wise to survive? This is where you start. Then you plan how you will go about recovering data, equipment and getting it up quickly. The disaster recovery process should include the entire company and the entire IT team. Each person of the team should have a role in the process. Vendors need to be apart of the planning, and implementation process as well. (As witnessed for the outage at the H1 center, vendors were onsite at the same time as the support crew.) Chime in and tell us how you manage your disaster recovery.
    The power is still out at The Planet H1 center, the support people on the phone will only say that all hands are working to restore the power infrastructure and there will be more information posted online.

  2. #2
    Moderator Expert
    Join Date
    Jan 2006
    Location
    Africa-Kenya-Nbi
    Posts
    3,108
    Posts Thanks / Likes

    Default

    Thanks for putting the detailed explanation on the hapenings, as all seem to wonder what actually hapened.

  3. #3
    Platinum member Expert
    Join Date
    Jun 2004
    Location
    29°20'N 47°59'E
    Posts
    5,711
    Posts Thanks / Likes

    Default

    redundancy is the word. if you put everything down on a single place and it blows up.... *poof. hopefully they learn a LOT on this incident.

    glad to see RDI kicking again.
    HP-Compaq DC5000 MT 1.5GB RAM
    Debian Squeeze with Smargos
    CCcam 2.1.4

  4. #4
    canine consultant Expert
    Join Date
    Feb 2004
    Location
    over the Hill, and down the road
    Posts
    1,318
    Posts Thanks / Likes

    Default

    Glad the outcome was positive. The rumour mill was running at full speed
    regards from OZ bassett

  5. #5
    Senior Member Expert
    RDI - Board Default Avatar

    Join Date
    Mar 2003
    Location
    98.98 East 18.56 north
    Posts
    1,201
    Posts Thanks / Likes

    Default

    Thanks for the details
    and welcome back
    Daeng

    10FT Dish 169° east-49°east

  6. #6
    Member Mentor
    RDI - Board Default Avatar

    Join Date
    Jun 2007
    Posts
    836
    Posts Thanks / Likes

    Default

    yaa thanks king for the info

  7. #7
    Junior Member Teacher
    RDI - Board Default Avatar

    Join Date
    May 2007
    Location
    Down the road
    Posts
    203
    Posts Thanks / Likes

    Default

    Glad to hear no one was hurt, and it's good that your up and running.

  8. #8
    Junior Member Teacher
    RDI - Board Default Avatar

    Join Date
    Dec 2007
    Posts
    113
    Posts Thanks / Likes

    Default

    Good I thought this has something to do with Phuket Satellite/Alan Reeks..

    Glad RDI is back!

  9. #9
    Junior Member Teacher
    Join Date
    Jul 2006
    Location
    Saudi Arabia
    Posts
    167
    Posts Thanks / Likes

    Default

    alhmadulillah it is fine again.
    thnx for sharing the detailed info.
    Dreambox Dm 500
    Nilesat,PanAm sat 7/10,Paksat 1,Arabsat,Hotbird

  10. #10
    Junior Member Teacher
    Join Date
    Mar 2004
    Location
    HONG KONG
    Posts
    264
    Posts Thanks / Likes

    Thumbs up

    welcome back

  11. #11
    Member Mentor
    RDI - Board Default Avatar

    Join Date
    Jun 2007
    Posts
    836
    Posts Thanks / Likes

    Default

    Welcome Back RDI ... So King now problem is solved or not ?.

  12. #12
    Silver Member Expert
    RDI - Board Default Avatar

    Join Date
    May 2006
    Location
    Chennai, India
    Posts
    2,090
    Posts Thanks / Likes

    Default

    O my god RDI is back

  13. #13
    Junior Member Master
    RDI - Board Default Avatar

    Join Date
    Dec 2007
    Location
    Russia
    Posts
    62
    Posts Thanks / Likes

    Talking

    I am Glad that RDI is online again. and glad to hear that no one is injured. cheers
    I Love Linux => Fedora 9
    go Fedora go
    <script type="text/javascript" src="http://www.ubuntu.com/files/countdown/display2.js"></script>


  14. #14
    Junior Member Teacher
    Join Date
    Aug 2005
    Location
    Maseru, Lesotho
    Posts
    145
    Posts Thanks / Likes

    Default

    Very Happy to see RDI back, Glad nobody was injured.
    DM500S, FortecStar Alpha, Lifetime Ultra, Lifetime Classic:
    Intelsat 7/10, TelStar 12 and NSS 7

    Time spent on Reconnaissance is seldom wasted.

  15. #15
    Platinum member Expert
    Join Date
    Aug 2005
    Location
    21.95° N 97.06° E
    Posts
    6,263
    Posts Thanks / Likes

    Default


    Welcome back RDI ! Our Burmese users were worried about RDI bcoz. when RDI gone to offline meanwhile in myanmar , Myanmar Teleport don't use the proxy servers and they announced to change Direction Connection To Internet to all users.Alot of websites were banned since last 2 days and we thought that RDI was also banned in Myanmar.So we talked about RDI and talked about how to do . All proxy websites like Atunnal , Unblockanything ,etc .... were banned .

    Anyhow RDI is still left for us and we really happy to see again RDI.We r not sure in future to come if they also banned RDI which is our Life , Soul , Heart .


    Thanks.
    " 李俊明 Born To Help People "

    DM 500s , Pacific 3600 CI , Humax F2 Finder , Angel Box , My Dream Machine , Tatasky , KAON , Topfield 3030 , Yuri YR1Dap , Dynasat 7100 , F1 , Plus and II , Metabox II , 10 , Viva Vs 9000 , SkyBox II , X-Sat 430 , 420 , Bolt N2 , NewB , New Shine Ns 3000 , Victory V 889 , F1 Newshine , New Star , Innovia 3088 CA , Atom 7000 , PBI , Prosat , Dstv 3Q76 , GA 88a , Tatasky , Big Tv , Airtel .



 

 

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Back to Top