BA systems down globally

[Laplace]

Publically, probably not. Within the UPS industry, yes.

[pushthebutton]

dmsims said:

You think ?

My money is on Cruz stuffing their mouths with gold so he is not found out

How will Cruz be found out?

[AW111]

Vaud said:

Yes and no. Outsourcing makes sense when it is not economically viable / you don't have the critical mass / investment budget etc to perform in-house.

It makes no sense, for example, to build your own computers from parts in a modern business. You outsource and buy a fully ready, supported product from Dell.

If you have one airconditioning unit, it makes no sense to have a dedicated air conditioning repair man on site for the fractional time they are used - you outsource it.

Ditto many IT services - cloud is attractive because of the sheer scale of investment they can bring (AWS/Azure) and PAYG pricing (opex intensity vs capital intensity). Owning your own data center doesn't make a lot of sense in may use case these days outside of a few cases. you may keep the application knowledge in house, but you outsource the "hassle of the asset"...

Small businesses outsource all the time - it's how many new companies (e.g. Uber) have become as big as they have, as fast as they have - no capital intensity and keeping to their core (platform code + brand identity + engagement) and outsource everything else.

I design & code software for a smallish company, and all our business IT is outsourced (locally). Despite occasional friction, I love it.
It was intriduced long ago, after the dev team started logging time spent sorting other people's computer problems

R&D have our own (virtual) server, we run the build system & repo. IT supplier is in charge of hardware, UPS, backups, etc.
More importantly, we don't get interrupted in our real work to fix email, sort out admin's problems etc.
We're too small to have a full time IT department alongside R&D, but part-time admin can be a huge time-sink.

[Du1point8]

AW111 said:

Vaud said:

Yes and no. Outsourcing makes sense when it is not economically viable / you don't have the critical mass / investment budget etc to perform in-house.

It makes no sense, for example, to build your own computers from parts in a modern business. You outsource and buy a fully ready, supported product from Dell.

If you have one airconditioning unit, it makes no sense to have a dedicated air conditioning repair man on site for the fractional time they are used - you outsource it.

Ditto many IT services - cloud is attractive because of the sheer scale of investment they can bring (AWS/Azure) and PAYG pricing (opex intensity vs capital intensity). Owning your own data center doesn't make a lot of sense in may use case these days outside of a few cases. you may keep the application knowledge in house, but you outsource the "hassle of the asset"...

Small businesses outsource all the time - it's how many new companies (e.g. Uber) have become as big as they have, as fast as they have - no capital intensity and keeping to their core (platform code + brand identity + engagement) and outsource everything else.

I design & code software for a smallish company, and all our business IT is outsourced (locally). Despite occasional friction, I love it.
It was intriduced long ago, after the dev team started logging time spent sorting other people's computer problems

R&D have our own (virtual) server, we run the build system & repo. IT supplier is in charge of hardware, UPS, backups, etc.
More importantly, we don't get interrupted in our real work to fix email, sort out admin's problems etc.
We're too small to have a full time IT department alongside R&D, but part-time admin can be a huge time-sink.

Thats onshore outsourcing... like using a Dublin team for a London project, so is generally not bad.

Offshore outsourcing is completely different and you would hate it.

[CaptainSlow]

Countdown said:

robinessex said:

Accountants, they know the cost of everything, the value of nothing !

In some cases Accountants are told by the FD/CEO that "We want to do Option X, please make X the most viable/profitable option".

Quite recently I've provided costings that show outsourcing Payroll will be 25% more expensive for a more basic service. However I'm told that moving to a Shared Service provider will "improve business resilience" and therefore that's the road that we will go down. Now I'm confident that the staff who will be left "in house" will be able to pick up the pieces if and when when the SHTF but it's a bit demoralising when pretty much everybody in the team knows this is being done as a "vanity project" for the benefit of somebody's CV.

A few moons ago I was the Financial Controller for the UK banking division of a large multi-national Corp. A year prior to my joining my boss had seen fit to replace the two middle aged women that had been successfully completing the bank account reconciliations for the previous ten years with two outsourced Indians. This rapidly turned into a situation where the accounts became unreconciled for six months needing five Indians to pretend to be making progress on daily telephone calls. Unsurprisingly I wasn't made aware of this, and similar messes, when I took the role. Every time I made a little progress I lost an outsourced head as the good ones were the expensive ones so had to go. I lasted six months in that job.

The role after we "outsourced" internally to Eastern Europe, this was done before I arrived and the local team had gone so all the knowledge was sat in process notes. The team in Eastern Europe had a high churn so the second hand knowledge soon turned to third/fourth hand. That was two years wasted on conference calls and screen sharing trying to sort out compounded mess. I've recently heard that the existing UK roles are now going...lessons never seem to be learned. I've since vowed that if the people doing the work I'm responsible for aren't in the UK, preferably the same building, I'm not interested., life is too short.

The good news is that in my current role I've successfully supported a business case to bring roles back onshore from India for one of the other departments.

[ruggedscotty]

pouring water on the roof would have no effect ? Actually it would, Putting water on the roof condensors would result in the water helping the cooling process.

Remember if your ambient environment temperature is high then pulling heat out takes some doing, the condensing pressures of your chiller plant will be pretty high. Anything that would help that would be welcome, Actually using cold water on your condensing matrix would have that effect, it would provide a huge assistance to the chillers. the average joe wouldnt realise and assume they were spraying water on the roof, but you know any assistance to get those temperatures down would be welcome, and this would do it. ive seen it done and it works a treat.

[Murph7355]

ruggedscotty said:

pouring water on the roof would have no effect ? Actually it would, Putting water on the roof condensors would result in the water helping the cooling process.

Remember if your ambient environment temperature is high then pulling heat out takes some doing, the condensing pressures of your chiller plant will be pretty high. Anything that would help that would be welcome, Actually using cold water on your condensing matrix would have that effect, it would provide a huge assistance to the chillers. the average joe wouldnt realise and assume they were spraying water on the roof, but you know any assistance to get those temperatures down would be welcome, and this would do it. ive seen it done and it works a treat.

A properly designed system in this country should not need that. In fact a properly designed system in any country shouldn't!

[ruggedscotty]

Yes that is so but things go wrong - we get high temperatures and fans fail, when your in the corner you will do what ever you can to keep the show on the road.

[gavsdavs]

ruggedscotty said:

pouring water on the roof would have no effect ? Actually it would, Putting water on the roof condensors would result in the water helping the cooling process.

Remember if your ambient environment temperature is high then pulling heat out takes some doing, the condensing pressures of your chiller plant will be pretty high. Anything that would help that would be welcome, Actually using cold water on your condensing matrix would have that effect, it would provide a huge assistance to the chillers. the average joe wouldnt realise and assume they were spraying water on the roof, but you know any assistance to get those temperatures down would be welcome, and this would do it. ive seen it done and it works a treat.

The quote is " staff were having to hose the top of the building down to keep it cool."

This suggests that the actual roof is hot, not the condensers. Adding cold waters to condensers which are a part of the cooling system makes sense, simply pouring water on a hot roof does not make sense.

My comment about "turning stuff off" is generally what I've seen done first in this kind of scenario. Almost every company will have a mixture of prod, dev, qa kit in a given room (though arguably they shouldn't) and if they're any good they'll know what's critical and what isn't. The first reponse is "shut down dev, qa", etc to stop putting more heat into the room. Shut down non critical prod. If there's any horizonally scaled web stuff, shutdown the instances in the room overheating.

This BA thing doesn't sound like they has chance to move workload out of the room and if the story is true - they lost the room in one event and didn't have suffficient headroom in the power system to bring everything back online again.

[V8 Fettler]

In other sectors, there are occasionally issues with the failure of diesel-generator sets used for secondary power supplies. Generally caused by poor maintenance and inadequate testing. The same issues probably apply to data centres.

[gooner1]

https://www.linkedin.com/pulse/official-statement-...

[Puggit]

http://www.bbc.co.uk/news/business-40117382

"It was not an IT issue, it was an issue with the power"

"The team at British Airways did everything they could in the circumstances to recover the operation as quickly as they could."

So lack of investment in the infrastructure then, but what about the IT issue that failover did not work?

[Puggit]

Some insights in to what should have happened from past BA employees: http://www.bbc.co.uk/news/business-40117381

[dmsims]

"The airline said on Wednesday that a loss of power to a UK data centre was "compounded" by a power surge that took out its IT systems."

This just gets funnier, they cannot make up the "correct" story (jackanory)

[Munter]

dmsims said:

"The airline said on Wednesday that a loss of power to a UK data centre was "compounded" by a power surge that took out its IT systems."

This just gets funnier, they cannot make up the "correct" story (jackanory)

Sounds like this might be a press release in a similar concept to a Ferrari F1 car having an electrical problem. Because a piston broke free and cut through the wiring loom.

https://www.linkedin.com/pulse/ba-what-just-happen...

"Hey Bob, how about we turn it off and back on again..."

[clonmult]

dmsims said:

"The airline said on Wednesday that a loss of power to a UK data centre was "compounded" by a power surge that took out its IT systems."

This just gets funnier, they cannot make up the "correct" story (jackanory)

the alternative is that these are all true, and that some systems were hit by a surge, others were hit by a power loss, whilst all of this was happening they forgot that they had a DR site and didn't have anything fail over to DR.

They're making themselves look even more incompetent each day. Quite amazing.

[Murph7355]

ruggedscotty said:

Yes that is so but things go wrong - we get high temperatures and fans fail, when your in the corner you will do what ever you can to keep the show on the road.

I agree. But I've thankfully never worked anywhere where I've been asked to get the hosepipe out and spray the kit on the roof down. Even in South Africa and Asia

V8 Fettler said:

In other sectors, there are occasionally issues with the failure of diesel-generator sets used for secondary power supplies. Generally caused by poor maintenance and inadequate testing. The same issues probably apply to data centres.

The same issues potentially apply, but IME (which is evidently of good quality set ups) the people involved know the importance of both testing and maintenance.

Doesn't stop people asking if corners can be cut. This is where senior mgmt need to both understand and be able and prepared to deliver the word "no" A rare commodity these days, at least with BAU staff (which I am ever grateful for).

[Otispunkmeyer]

dmsims said:

Puggit said:

dmsims said:

Max temp at Heathrow in 2016 was 24.7 C - hardly high

And not correct either - there were multiple days above 30 degrees at Heathrow in July.

http://www.standard.co.uk/news/uk/london-weather-h...

Blimey! I got that figure from Heathrow weather station data at the Met office - 2016 ranked 21st since 1948

Depends where the station is. If its plonked on the tarmac it'll read much higher than one plonked away from the runway in a nearby field. The former of course suffers from UHI effect and other factors (i.e. lots of hot jet engine air).

[Munter]

Otispunkmeyer said:

Depends where the station is. If its plonked on the tarmac it'll read much higher than one plonked away from the runway in a nearby field. The former of course suffers from UHI effect and other factors (i.e. lots of hot jet engine air).

I believe it's here:
https://goo.gl/maps/T14aJrPcmMw

[Vaud]

Munter said:

Sounds like this might be a press release in a similar concept to a Ferrari F1 car having an electrical problem. Because a piston broke free and cut through the wiring loom.

I thought that was a Renault turbo?