Subscribe to Collegiate Chronicles        RSS Feed
-----

Frankenstorm Meets Frankensystem

Icon 3 Comments
This week has been a massive. ball. of. stress.

Hurricane Sandy... Sandy... Honestly? That's the fiercest thing they could come up with? Hurricane Sabretooth wasn't an option? Hurricane Sucksalot? Hurricane Sleeplessness? Hurricane Steve the Velociraptor didn't cross anyone's mind? Though I have to give the media credit for latching on to the storm's Halloween proximity and dubbing it the Frankenstorm. Props to whomever coined that one.

Anyway, Hurricane Stupid Name nailed the Northeastern US and brought massive damages not only to the coast, but for miles and miles inland. I live in southwestern Pennsylvania. We were hammered with wet, heavy snowfall. Things are just now starting to regain some semblance of normalcy.

I left work early last Friday with a sore throat. I spent most of last weekend curled into a ball of moody, sleepy illness on the couch. Monday morning, I called in sick. My plan was to see my doctor for an antibiotic - assuming whatever had crawled into my throat was even treatable.

I didn't make it to the doctor's office.

Instead, my phone started ringing at 10AM. "We're closing at 11:45 today and we'll be closed tomorrow. Update the websites."

I grabbed a chair and my laptop and connected successfully to the VPN. From there, nothing worked. I tried accessing the database directly only to be met with error after error indicating that the connection was made and subsequently dropped. I tried accessing the CMS only to find a blank screen. I tried hitting the main pages of the sites I needed to update (6 of them total) and each and every one of them led to a white screen and a database error.

In a mad panic - thinking the issue was in my 3G wireless connection, I loaded up the kids (who didn't have school), the laptop, and myself and hauled butt to my mother's house where I could hop online with her cable connection. I encountered all the same errors.

My cell phone rang again. "We've been without power all morning. The sites are down. Fix them."

"Are the servers online?"

"I don't know. Call IT." I was given a cell number for our IT Director.

I called. "Are the servers online?"

"Why wouldn't they be?"

"I don't know. I was told there was a power outage and none of the sites are working."

"Let me look and I'll call you back."

Two hours later, the phone rang again. "The servers are back up. Try it now."

Nothing.

"We were running updates this morning when we started experiencing short power outages. I don't think any of the updates finished running. Let me see if I can sort it out. I might need to enlist your help when we all get back to campus later this week."

Running updates in a major storm? Sounds logical.

She called back. "Okay. I think we got it now."

The main site was functioning fine. I was able to establish the required VPN connection and connect directly to the database for updates. All the other sites were still down.

I called back. "Whatever you did to the main site server needs done to the auxiliary site server."

"Which server?" I gave her the server names. "Which ones are those?"

"Those are the servers that house all the auxiliary sites and their corresponding databases."

"Um... I'm at home in Virginia. I'll drive up later and let you know when they're up."

My phone started ringing off the hook from there. My boss, her boss, bosses from other departments... websites are down... email isn't working... campus phone system is non-functional... backup generator went down... key card door access has died... the electronic temperature control for the pool stopped giving a read-out. Fix it. Fix it. Fix it.

I thought my head would explode. The electricity is out, people. Electronic temperature controls won't work without electricity. Electronic key card readers won't work without electricity. If the generator is down, the servers aren't working... hence, no email, no websites, no VoIP phone systems.

It was almost 9PM when the IT Director called back. "The generator is running now. I reset all the servers. Everything should be up. I'm already in the car on my way back to Virginia. Call me if you need anything."

I hopped on Firefox. Again, the main site was functional... everything else was spewing database errors. I called her back. "Still getting the errors," I said.

"Yeah, I looked that up. It's a coding issue. You'll have to figure it out."

All night Monday, all day Tuesday, all morning Wednesday, I wracked my brain. I searched and searched and made changes and edits and uploaded and re-uploaded code. Every programming board on the net had some new tiny nugget of a suggestion on how to correct the errors. I tried them all. Then I went back and tried them all again. Absolutely nothing was having any impact at all.

I took a break, made a fresh cup of coffee and started digging through the cupboard for a lunch that wouldn't require any actual cooking.

As I sat down with my piping hot cup of ramen and my fresh-from-the-Keurig cup of black coffee, I refreshed the page. It was working. Double-you-tee-eff?

I checked them all. They were ALL working. IT WAS A HALLOWEEN MIRACLE!

I didn't know what had happened, but I decided against questioning it and angering the gods.

Two hours later, I received another call. "We're going to remain closed the rest of the week. Please update all the sites. They should all be up now. One of the database servers wasn't connected to the generator so it's been completely turned off for the last three days. Ha! Funny, isn't it?"

Yeah. Hysterical.

3 Comments On This Entry

Page 1 of 1

no2pencil Icon

01 November 2012 - 07:53 AM
They are called procedures. Your head of heads should look into them. Just saying.
3

.Aaron Icon

01 November 2012 - 03:34 PM

Quote

Two hours later, I received another call. "We're going to remain closed the rest of the week. Please update all the sites. They should all be up now. One of the database servers wasn't connected to the generator so it's been completely turned off for the last three days. Ha! Funny, isn't it?"


Hmmmm.. yea, I'd be ready to kill someone.. not literally, but pretty close.
2

Martyr2 Icon

01 November 2012 - 04:27 PM
I have had days just like that before. One rule I have come to learn in things like that, and I am sure you have just now learned it, when so many things are going wrong at the same time and you are stumped, it isn't you, it is someone else.

If 10 servers go down and you haven't even logged into 8 of them in the last several months, and they have been running fine until now, it usually means that someone else did something to make them go down. Here it would be someone not plugging in something. Or a storm knocked out the power and that is not a coders domain.

When I have been hit with such massive fail, I usually realize it is someone else doing it and then I call that person and tell them to fix it all. Now if you can prove to me that the errors fall in my realm of responsibility (actual code errors) then I will fix it. But when a storm is raging and all things tech are going down, you have to realize it is way beyond you and your superiors should know that too.

Great blog B9. :)
6
Page 1 of 1

September 2014

S M T W T F S
 123456
78910111213
14151617181920
2122 23 24252627
282930    

Recent Entries

Recent Comments

Search My Blog

0 user(s) viewing

0 Guests
0 member(s)
0 anonymous member(s)