Data centre fire = Goodbye cloud data.

[peterperkins]

'Activate your disaster recovery plan.'

Hmm not exactly comforting when they want us to put everything into the cloud.

https://www.bbc.co.uk/news/business-56345589

I thought the cloud/online storage was massively backed up in different locations like a giant Raid 1 array.

It's useful as a backup and for remote access, but clearly that usb stick in your pocket, portable hdd drive etc and multiple copies are still vital.

Edited by peterperkins on Thursday 11th March 06:09

[rxe]

Really, it should be perfectly comforting.

It is perfectly possible to build a cloud application that is vulnerable to problems like this, but it is also perfectly possible to build something that is incredibly robust and spans continents. The last thing I build on cloud would probably survive the destruction of half the planet. Whether you could access it or not would be a different question. I’m working back “on prem” and it is a boat load harder to build something with similar levels of reliability.

[Crafty_]

peterperkins said:

I thought the cloud/onlines storage was massively backed up in different locations like a giant Raid 1 array.

Possible if you architect your solutions properly, but if you decide to be cheap and not bother, guess what could happen...

[oobster]

One of the services my employer offers to customers was affected by this fire, I don't know much about IT or data centres but how does a building like this go on fire?

I am assuming there would be a massive amount of electrical equipment on site but what else is there? Would a building like that have lots of combstibles, paper, carboard, furniture etc? Would there be anything else on site in a data centre that would provide fuel for a fire?

Also, would there not be some sort of inert gas fire suppresion system in place?

[dapprman]

The commercial cloud providers all multi-region capabilities, however you can then watch your costs start to rack up so people skimp, also fail over is not as automatic as people think meaning really you should be running across multiple regions at the same time with a high level load balancer, such as Akamai, to hand the network pointing for you. Again watch those costs rise rapidly.

As with everything it is a balance between cost and resilience with too many 'consultants' ignoring the need to support SLAs (as the big boys offer 4 or 5 nines, but with lots of caveats, including if you go for a single data centre or single region set up.

[dapprman]

oobster said:

Also, would there not be some sort of inert gas fire suppresion system in place?

If a fire gets to a wall and climbs up to the rafters then nolon (or what ever the present halon replacement is called) will do nothing as it is heavier than air.

[Spitfire2]

oobster said:

One of the services my employer offers to customers was affected by this fire, I don't know much about IT or data centres but how does a building like this go on fire?

I am assuming there would be a massive amount of electrical equipment on site but what else is there? Would a building like that have lots of combstibles, paper, carboard, furniture etc? Would there be anything else on site in a data centre that would provide fuel for a fire?

Also, would there not be some sort of inert gas fire suppresion system in place?

The really big Data centres are far too big for halon gas systems. Been in a rather huge one in Dublin with several halls each the size of a football pitch. In that example they didn't even have sprinklers - water could quickly do as much damage as fire.

[EK993]

It’s all down to cost. As has been stated, you can architect a system to run across multiple regions, either within a continent (US, EU), or spanning across different continents. The more resiliency, the more $$. I work for a global multi national (financial services) and the amount we spend in Azure is eye watering (we have very few physical data centers left).

[camel_landy]

oobster said:

One of the services my employer offers to customers was affected by this fire, I don't know much about IT or data centres but how does a building like this go on fire?

I am assuming there would be a massive amount of electrical equipment on site but what else is there? Would a building like that have lots of combstibles, paper, carboard, furniture etc? Would there be anything else on site in a data centre that would provide fuel for a fire?

All of the cables... They're wrapped in plastic and there are big bundles of them.

M

[camel_landy]

rxe said:

I’m working back “on prem” and it is a boat load harder to build something with similar levels of reliability.

Not if you've got the correct funding and the right people on the job it isn't...

Cloud hides all that messy stuff behind a number on an invoice.

M

[rxe]

camel_landy said:

Not if you've got the correct funding and the right people on the job it isn't...

Cloud hides all that messy stuff behind a number on an invoice.

M

Well that's the problem.... (Currently twiddling thumbs while the subcontractor's vendor racks some kit...because some muppet signed a 2 week SLA...)

They try very hard to make data centres non-flammable, but even low smoke cable burns, and a lot of data centres are surrounded by rather "hostile" neighbours. I worked at one that had a paint factory next to it some years ago, that was a disaster waiting to happen.

[WonkeyDonkey]

I'm surprised they don't have systems that suck all the oxygen out of a room if a fire is detected. Obviously would need to be unmanned.

[buggalugs]

oobster said:

One of the services my employer offers to customers was affected by this fire, I don't know much about IT or data centres but how does a building like this go on fire?

I am assuming there would be a massive amount of electrical equipment on site but what else is there? Would a building like that have lots of combstibles, paper, carboard, furniture etc? Would there be anything else on site in a data centre that would provide fuel for a fire?

Also, would there not be some sort of inert gas fire suppresion system in place?

I read somewhere that they’re replacing power cables in other data centres asap due to a potential insulation problem.

I thought cable sheaths were meant to be non combustible these days but not sure what the details are. LSOH, plenum rated etc.

[colin79666]

oobster said:

I am assuming there would be a massive amount of electrical equipment on site but what else is there? Would a building like that have lots of combstibles, paper, carboard, furniture etc? Would there be anything else on site in a data centre that would provide fuel for a fire?

Batteries (UPS).

[RicksAlfas]

oobster said:

I am assuming there would be a massive amount of electrical equipment on site but what else is there? Would a building like that have lots of combstibles, paper, carboard, furniture etc? Would there be anything else on site in a data centre that would provide fuel for a fire?

They print everything out so they have a paper back up.

[dapprman]

camel_landy said:

rxe said:

I’m working back “on prem” and it is a boat load harder to build something with similar levels of reliability.

Not if you've got the correct funding and the right people on the job it isn't...

Cloud hides all that messy stuff behind a number on an invoice.

M

When I worked in support then project work we always used to say "blame networks as there's a 99% chance you are right", however since moving in to architecture (and also with the benefits of an extremely good network engineer) I've seen just how many break points and independent dependencies there can be with cross site networking (never mind the costs when you find you need to dig a new cabling trench to update/upgrade your bandwidth on your of your external connections).

WonkeyDonkey said:

I'm surprised they don't have systems that suck all the oxygen out of a room if a fire is detected. Obviously would need to be unmanned.

This is how Halon and it's replacements (as Halon is cargegenic) effectively works - you get a blanket of gas dropping down and pushing the oxygen/air upwards and off the flame.

[Ian Geary]

So the cloud has gone up in smoke?

(I have nothing sensible to add)

[Oakey]

oobster said:

One of the services my employer offers to customers was affected by this fire, I don't know much about IT or data centres but how does a building like this go on fire?

I am assuming there would be a massive amount of electrical equipment on site but what else is there? Would a building like that have lots of combstibles, paper, carboard, furniture etc? Would there be anything else on site in a data centre that would provide fuel for a fire?

Also, would there not be some sort of inert gas fire suppresion system in place?

I read somewhere that you used to have around 15mins to escape a house fire before things really turned to st but thanks to so many synthetic based products in our homes you now have around 2 - 3mins. I imagine it's the same for a data centre where there's tons of cables and stuff.

[TTmonkey]

colin79666 said:

Batteries (UPS).

That would be a badly designed data centre. Ours don’t have UPS in the server racks, it’s all external to the room.


Lots of power though, and cables etc.

[Munter]

WonkeyDonkey said:

I'm surprised they don't have systems that suck all the oxygen out of a room if a fire is detected. Obviously would need to be unmanned.

I think it's usually a gas based system that removes heat/replaces the air with something inert. Thing about removing the oxygen is if it go's off accidentally and you have someone in there...it's considered bad form. So you have to try and not kill people, but still put the fire out without water preferably.

I would have thought you'd have 2 stages. Gas, if that fails a big ohfk turn on the sprinklers button.