BA systems down globally
Discussion
saaby93 said:
Confirms my post at 17:15 from Radio4 news?
It tells the same story for sure.A power issue a week after the biggest and most intrusive bit of malware for years that has everyone running around trying to patch things, it is possible but that's way too much coincidence for me and the story sounds made up by someone non technical, like the surge wss so big it flowed right through the primary systems and took out the backup too. A power tsunami if you will...
anonymous said:
[redacted]
Alex Cruz has nobody responsible for IT reporting to him. All of BA's IT was moved to its owner IAG, the person in charge is Bill Francis, Head of Group IT. Francis reports to Ignacio de Torres Zabala, Director of Global Services, who in turn reports to Willie Walsh, Chief Executive of IAG.
Walsh has amongst his direct reports Cruz and Zabala, but not Francis. It may be relevant that Francis is not an IT man - he is perhaps best known for being in charge of cabin crew.
If this is true, how can the BA CEO have no one from IT reporting to him? Then how come the Head of IT is not an IT man? It beggars belief. Cruz is looking like a fall guy in this. IMO Walsh should be in the frame along with Cruz.Walsh has amongst his direct reports Cruz and Zabala, but not Francis. It may be relevant that Francis is not an IT man - he is perhaps best known for being in charge of cabin crew.
IT is a group function BA just uses whats provided by group
How would Cruz know or be expected to know whats happening back up the chain to Group and down into IT?
How come only BA affected? Does group have separate data centres for the different operations?
saaby93 said:
Surely thats what youd expect
IT is a group function BA just uses whats provided by group
How would Cruz know or be expected to know whats happening back up the chain to Group and down into IT?
I have the integration structure somewhere in a ppt but I think that is the case.IT is a group function BA just uses whats provided by group
How would Cruz know or be expected to know whats happening back up the chain to Group and down into IT?
I do recall that the proposed IT integration was as one colleague put it... "they think it's great but it's a basket case, I hope we don't win it"
Chim said:
wombleh said:
The update story rings true with me. My first thought was that either management or Infosec teams had demanded systems get patched following the wannacry drama and the patching had broken things.
Just saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
Someone somewhere is feeding the exec some serious crap. Of all the excuses in the IT bible to come up with this has to one of the worst I have heardJust saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
Murph7355 said:
Any data centre worth its salt will be able to withstand a shaky mains supply.
From the locations of the data centres upthread, it sounds like they are home made. Most of my experience is with data centre providers - so Tier 3 or 4, loads of rooms, high levels of professionalism - no shonky installations here. Most "internal" data centres I have seen have not been as good. Single power supplies, haven't changed the UPS batteries for a decade, never tested A/B power since the thing was built, all the comms coming through a single duct. All of these things are really expensive to retrofit, so if no one at board level is listening, then it will never be fixed.
Looking on the positive side, they're probably listening now....
Apparently Cruz said:
“we were unable to restore and use some of those backup systems because they themselves could not trust the messaging that had to take place amongst them.”
Which is about the only thing that makes sense, depending on the DR/replication they use, if neither side can decide which is primary (or worse both think they are), you're in a world of trouble. And even once you sort that, one system restore, even half a dozen is easy, much more than that though and suddenly it's far more difficult, you hit bandwidth issues and all sorts moving the data around not to mention the human intervention of sanity checking them against logical corruption.
I mean, something so critical the DR should have been able to work around this, but still that explanation he gave is definitely plausible for me because I've seen it happen.
“we were unable to restore and use some of those backup systems because they themselves could not trust the messaging that had to take place amongst them.”
Which is about the only thing that makes sense, depending on the DR/replication they use, if neither side can decide which is primary (or worse both think they are), you're in a world of trouble. And even once you sort that, one system restore, even half a dozen is easy, much more than that though and suddenly it's far more difficult, you hit bandwidth issues and all sorts moving the data around not to mention the human intervention of sanity checking them against logical corruption.
I mean, something so critical the DR should have been able to work around this, but still that explanation he gave is definitely plausible for me because I've seen it happen.
rxe said:
From the locations of the data centres upthread, it sounds like they are home made. Most of my experience is with data centre providers - so Tier 3 or 4, loads of rooms, high levels of professionalism - no shonky installations here. Most "internal" data centres I have seen have not been as good. Single power supplies, haven't changed the UPS batteries for a decade, never tested A/B power since the thing was built, all the comms coming through a single duct.
All of these things are really expensive to retrofit, so if no one at board level is listening, then it will never be fixed.
Looking on the positive side, they're probably listening now....
My experience has been with blue chip firms over the last 20yrs. They haven't all started out great, and there are still some horror stories out there which only haven't made the news through good fortune rather than planning. But I find it hard to believe an organisation like BA would be running such fundamental risks. All of these things are really expensive to retrofit, so if no one at board level is listening, then it will never be fixed.
Looking on the positive side, they're probably listening now....
Then again...
If they are, the CEO wants to be shouldering 100% of the blame. An entirely avoidable circumstance.
I think we need to join up the dots:
(1) what started this whole process off - was it the wannacry malware?
(2) what was being done just before the problem started (patching followed by reboots).
(3) what was actually the problem - power surge is just BS - I would be looking to SSE for a statement then. Cloud providers like AWS and Azure have millions of machines, storage devices and network switches in their data centres and I am sure that they regularly patch and re-cycle them without "power issues".
(4) what is the actual state of the data centre(s) involved. The timing for this is very embarrassing since on Thursday last week the head of Cloud at Microsoft (Scott Guthrie) was playing a BA video to 400 Techies showing how Microsoft Cloud had transformed the IT at BA. So I am not understanding why if Microsoft is involved in BA IT how this issue affected virtually every bit of BA's IT systems Has BA been de-investing in their on premises data centres for sometime now so that they acceptted the risks? You will be actually surprised how many blue chip companies do take huge risks with their IT in the name of cost especially when they are in the throws of moving to a public cloud provider where more money can be saved.
John
(1) what started this whole process off - was it the wannacry malware?
(2) what was being done just before the problem started (patching followed by reboots).
(3) what was actually the problem - power surge is just BS - I would be looking to SSE for a statement then. Cloud providers like AWS and Azure have millions of machines, storage devices and network switches in their data centres and I am sure that they regularly patch and re-cycle them without "power issues".
(4) what is the actual state of the data centre(s) involved. The timing for this is very embarrassing since on Thursday last week the head of Cloud at Microsoft (Scott Guthrie) was playing a BA video to 400 Techies showing how Microsoft Cloud had transformed the IT at BA. So I am not understanding why if Microsoft is involved in BA IT how this issue affected virtually every bit of BA's IT systems Has BA been de-investing in their on premises data centres for sometime now so that they acceptted the risks? You will be actually surprised how many blue chip companies do take huge risks with their IT in the name of cost especially when they are in the throws of moving to a public cloud provider where more money can be saved.
John
ruggedscotty said:
Vaud said:
Outsourcing is fine when done properly. How far do you vertically integrate?
But it is never ever done properly - this is what happens - once you hand that control over to a 3rd party it is out of your hands. You think you have control but you dont. its over to them and they do mess up. especially if you outsource to a different country. different totally different. What's the betting that BA will end up making more from suing whichever of the "big four" advised them on their outsourcing and IT than they pay out in compensation to passengers? Could end up being a nice little learner. Big bonuses for the CEO and his friends in due course, no doubt!
Gassing Station | News, Politics & Economics | Top of Page | What's New | My Stuff