Has anyone worked in a data center?
Discussion
Seems fun when things go wrong:
https://www.youtube.com/watch?v=LeFtwtvy4Wc
^ I wish I had a powerful relay switch like that on my desk every time the web apps or databases go down. That way when bugged by departments every 5 minutes asking "is it back up yet" they can watch me flip a switch, hear the loud bang and go away.
Good nightlife? .... I hear time dilation can occur if you spend long enough in there with no sunlight.
To tell you the truth I wouldn't mind trying it for a week, must be a totally different environment from an office.
https://www.youtube.com/watch?v=LeFtwtvy4Wc
^ I wish I had a powerful relay switch like that on my desk every time the web apps or databases go down. That way when bugged by departments every 5 minutes asking "is it back up yet" they can watch me flip a switch, hear the loud bang and go away.
Good nightlife? .... I hear time dilation can occur if you spend long enough in there with no sunlight.
To tell you the truth I wouldn't mind trying it for a week, must be a totally different environment from an office.
I used to be an engineering manager on a contract that had a couple of data centres. The guys on the ground were way more expert than me and the electrical systems were massively complex. 1000 battery UPS systems, mains synchronising 11kV generators etc.
I took a call late one afternoon, they were concerned that the power supply for the generator control system was failing. There had been an external power blip and the panel had gone blank rather than initiating then stopping the generator start procedure as expected. The control specialist was called, who diagnosed some parts needed but they were on a 24hr delivery (yes, should have had spares at site!) We opted to call on the contingency power contract and have an external generator delivered and connected up 'just in case'. 8 hr deployment time
Still just a fall back position, the site had not had an external power failure that caused a forced generator in ten years (short blips were managed by the UPS and we completed a full on generator load run for a day every three months)
It was now about midnight, so I went home and a shift pattern was arranged so there was at least two maintenance and two IT guys on site at a time (rather than one of each). Got back to site at 7 and they were just making the final cables off and completed a quick test run. All good, phew.
At 8.45 all the lights went out, lots of ACBs banging in and out like in that linked video and a million alarms went off, The guy had just arrived to start the control system repair and all eyes swivelled in his direction. "Nothing to do with me, I haven't even opened the panel!"
It was an external power failure. The guys manually started the contingency generator and closed the breakers so it then took the load (just under 1 MW) after about 10 mins on UPS
If the contingency power install had taken an hour longer, the data centre for a major financial insitution would have failed, the sort of thing that makes the national news.
My heart rate still inceases a little typing this out and it's ten years ago.
I still do engineering manager things, I don't manage data centres any more
I took a call late one afternoon, they were concerned that the power supply for the generator control system was failing. There had been an external power blip and the panel had gone blank rather than initiating then stopping the generator start procedure as expected. The control specialist was called, who diagnosed some parts needed but they were on a 24hr delivery (yes, should have had spares at site!) We opted to call on the contingency power contract and have an external generator delivered and connected up 'just in case'. 8 hr deployment time
Still just a fall back position, the site had not had an external power failure that caused a forced generator in ten years (short blips were managed by the UPS and we completed a full on generator load run for a day every three months)
It was now about midnight, so I went home and a shift pattern was arranged so there was at least two maintenance and two IT guys on site at a time (rather than one of each). Got back to site at 7 and they were just making the final cables off and completed a quick test run. All good, phew.
At 8.45 all the lights went out, lots of ACBs banging in and out like in that linked video and a million alarms went off, The guy had just arrived to start the control system repair and all eyes swivelled in his direction. "Nothing to do with me, I haven't even opened the panel!"
It was an external power failure. The guys manually started the contingency generator and closed the breakers so it then took the load (just under 1 MW) after about 10 mins on UPS
If the contingency power install had taken an hour longer, the data centre for a major financial insitution would have failed, the sort of thing that makes the national news.
My heart rate still inceases a little typing this out and it's ten years ago.
I still do engineering manager things, I don't manage data centres any more
I've done a lot of work in data centres. It's often a pain to get in due to security, a pain to get out due to size and layout of the buildings and security, extremely noisy, and either very hot or freezing cold. It's usually difficult to get stuff in and difficult to get stuff out, and they universally seem to be staffed by persons with an extremely surly attitude where any routine action on their part is doing you a massive favour. I go out of my way to be polite and friendly because they can go a long way to making your day either much easier or much harder than it might otherwise be, but I've met more DC staff who are very hard work to deal with than anywhere else I've ever been.
I'll never forget the first time I was sent to install some kit in one in my first proper IT infrastructure engineer job, I'd only started the week before and went with a colleague who showed me to the suite where I needed to be was, then he went off to another job in a different part of the DC.
I was up on some steps and plugged a network cable into a switch, and just as the cable clicked home there was a loud clunk and all the lights on the floor went off! I instinctively took the cable back out again, but the lights remained off. I got down from the steps and froze in a state of panic. It took me a minute to compose myself, but it then dawned on me all the kit was still powered on and there really couldn't be any link between the switch and the lighting. I called the colleague who pissed himself laughing and then told me the lights for the whole floor were on a switch by the entrance door but on a timer and you had to go and switch them back on every 20 minutes!
A lot of DC's are not as well maintained as the providers would have their customers believe. I've had supposed independent and redundant dual power feeds fail simultaneously in one very large central London DC on multiple occasions - they once told us they needed to take the A feed down for maintenance, when they did this it overloaded and tripped the main breaker for the B feed as all the load transferred! When poking around it's not uncommon to see all sorts of warnings and unresolved faults on things like UPS units and fire suppression left for weeks at a time. One place in Manchester I visited looking for some colo space had maintenance log stickers on their UPS's which when I read them had been recommending battery replacement for the last 3 years!
I'll never forget the first time I was sent to install some kit in one in my first proper IT infrastructure engineer job, I'd only started the week before and went with a colleague who showed me to the suite where I needed to be was, then he went off to another job in a different part of the DC.
I was up on some steps and plugged a network cable into a switch, and just as the cable clicked home there was a loud clunk and all the lights on the floor went off! I instinctively took the cable back out again, but the lights remained off. I got down from the steps and froze in a state of panic. It took me a minute to compose myself, but it then dawned on me all the kit was still powered on and there really couldn't be any link between the switch and the lighting. I called the colleague who pissed himself laughing and then told me the lights for the whole floor were on a switch by the entrance door but on a timer and you had to go and switch them back on every 20 minutes!
A lot of DC's are not as well maintained as the providers would have their customers believe. I've had supposed independent and redundant dual power feeds fail simultaneously in one very large central London DC on multiple occasions - they once told us they needed to take the A feed down for maintenance, when they did this it overloaded and tripped the main breaker for the B feed as all the load transferred! When poking around it's not uncommon to see all sorts of warnings and unresolved faults on things like UPS units and fire suppression left for weeks at a time. One place in Manchester I visited looking for some colo space had maintenance log stickers on their UPS's which when I read them had been recommending battery replacement for the last 3 years!
quinny100 said:
It took me a minute to compose myself, but it then dawned on me all the kit was still powered on and there really couldn't be any link between the switch and the lighting.
Similar, I was working in a large supermarket in my days on the tools and the comms room AC wasn't working. Found no power to it, the in-house maintenance guy wasn't around so I went and found the main incoming panels. Looked at the circuit charts and found the location of the breaker for the comms room AC, started to open the right cover and a very loud buzzer sounded. Jumped yards and quickly closed it. I then spent about 15mins wondering what to do next and looking to see if I could prevent the 'anti tamper' alarm from going off again so I could look at/reset the breaker.Suddenly the buzzer sounded again and one of the warehouse staff went to the loading bay and opened the roller door for the delivery driver who was ringing the yard door bell
Reset breaker, diagnosed problem, went home.
Built one in Raleigh NC, scared the kak out of me whilst watching USD$ 2M+ of kit being slapdashed across the loading dock, none of the networking kit was installed either (Thanks AT&T), got her up and running eventually; nearly as hard as deployment of kit in the Negev desert. But that's another story.
SVX said:
Built one in Raleigh NC, scared the kak out of me whilst watching USD$ 2M+ of kit being slapdashed across the loading dock, none of the networking kit was installed either (Thanks AT&T), got her up and running eventually; nearly as hard as deployment of kit in the Negev desert.]But that's another story.
When you wrote that post, did you intend to make yourself sound like Lord Flashheart from Blackadder, or is that actually how you usually converse?With all the cringey embellishments, I was half expecting to see "smoke me a kipper, I'll be back for breakfast - woof!" at the end.
Gassing Station | Jobs & Employment Matters | Top of Page | What's New | My Stuff