BA systems down globally
Discussion
I've been called to go and fix machines in data centre's which have been out sourced to India.
First question I ask on arrival over the phone is where is the machine, lots of times nobody knows or I get the reply "Can you go and look for it". err, NO it's not my job to find machines in football pitch size data centres when I don't work for your company, you need to tell me where it is located.
This is basic stuff, that's how bad it is sometimes.
First question I ask on arrival over the phone is where is the machine, lots of times nobody knows or I get the reply "Can you go and look for it". err, NO it's not my job to find machines in football pitch size data centres when I don't work for your company, you need to tell me where it is located.
This is basic stuff, that's how bad it is sometimes.
38911 said:
anonymous said:
[redacted]
I'm just curious, why you think India is any worse than remote staff in any other location in the world? I've worked extensively all over India (as well as in the US, Asia and throughout Europe) - and I have found IT colleagues in India to be just as skilled and capable IT professionals as anywhere else in the world. The problem is companies screw outsourcing suppliers down to the lowest possible price, which forces suppliers into heavily depending on automation and tooling and minimal staffing required to meet SLAs (and nothing more) in order to meet their cost-base.
I for one don't consider India any worse that remote staff elsewhere. However I will throw one cultural thing into the mix, but I don't blame this on the Indian IT sub-contractors themselves, it is still a management failure, a failure to adjust properly to a different set of cultural norms.
We Brits like to complain and grumble, especially about work, we are used to it, we expect it. This trait is not prevalent amongst the Indian staff I have worked with. They are very restrained even when they would have solid grounds to be complaining loudly. The word 'No' is almost taboo for them. This is especially the case when they perceive a power imbalance, you are older, more senior or experienced. They have a lot of respect for experience and elders. I've found British management typically fail to appreciate and adjust to this difference.
We need to learn to take expressions like "Let me see", "I will try", "I will do my best" as usually meaning no, or you've missed something important.
If they get to "I have doubts/concerns" it really should be translated as meaning no f*** way is that feasible or reasonable.
Anything other than an empathic yes should be considered to mean no.
Edited by 4x4Tyke on Wednesday 31st May 22:22
TheGroover said:
anonymous said:
[redacted]
Any chance of a link to where you read that? I'd be interested to read it.However, they are taking control of their own new platforms and rebuilding in-house IT capability (while still outsourcing some of the industrialised boring stuff like testing)
I flew BA LHR - YYZ yesterday. We were only delayed by 90 minutes which, having cancelled my flight from DOH - LHR on Friday 19th May with no alternative offered and then delayed my flight from LHR - MAD by 5 hours on Sunday 21st May, is the least disruption BA have caused to my travel plans over the past 3 weeks.
They really are a complete shambles of a business.
They really are a complete shambles of a business.
4x4Tyke said:
I don't read his words that way.
I for one don't consider India any worse that remote staff elsewhere. However I will throw one cultural thing into the mix, but I don't blame this on Indian IT sub-contractor, it is still a management failure, a failure to adjust properly.
We Brits like to complain and grumble, we are used to it we expect it. This trait is not prevalent amongst the Indians I have worked with in IT. They are very restrained when they would have solid grounds to be complaining loudly. The word 'No' is almost taboo for them. This is especially the case when they perceive a power imbalance, you are older, more senior or experienced. I've found British management typically failure to appreciate this difference.
We need to learn to take expressions like "Let me see", "I will try", "I will do my best" as usually mean no, or you've missed something important.
If they get to "I have concerns" it really should be translated as meaning no f*** way is that feasible or reasonable.
Anything other than an empathic yes should be considered to mean no.
A friend of mine has been out to Bangalore to help with her company's outsourcing project and she was surprised that the Indians say 'yes, they can do such and such' when they don't know how to do it. She found it very frustrating.I for one don't consider India any worse that remote staff elsewhere. However I will throw one cultural thing into the mix, but I don't blame this on Indian IT sub-contractor, it is still a management failure, a failure to adjust properly.
We Brits like to complain and grumble, we are used to it we expect it. This trait is not prevalent amongst the Indians I have worked with in IT. They are very restrained when they would have solid grounds to be complaining loudly. The word 'No' is almost taboo for them. This is especially the case when they perceive a power imbalance, you are older, more senior or experienced. I've found British management typically failure to appreciate this difference.
We need to learn to take expressions like "Let me see", "I will try", "I will do my best" as usually mean no, or you've missed something important.
If they get to "I have concerns" it really should be translated as meaning no f*** way is that feasible or reasonable.
Anything other than an empathic yes should be considered to mean no.
Edited by 4x4Tyke on Monday 29th May 14:12
schmalex said:
I flew BA LHR - YYZ yesterday. We were only delayed by 90 minutes which, having cancelled my flight from DOH - LHR on Friday 19th May with no alternative offered and then delayed my flight from LHR - MAD by 5 hours on Sunday 21st May, is the least disruption BA have caused to my travel plans over the past 3 weeks.
They really are a complete shambles of a business.
There's no excuse for any of those. Qatar airways is oneworld so should be an easy swap and Madrid is serviced by the whole of bloody IAG!They really are a complete shambles of a business.
schmalex said:
I flew BA LHR - YYZ yesterday. We were only delayed by 90 minutes which, having cancelled my flight from DOH - LHR on Friday 19th May with no alternative offered and then delayed my flight from LHR - MAD by 5 hours on Sunday 21st May, is the least disruption BA have caused to my travel plans over the past 3 weeks.
They really are a complete shambles of a business.
I am guessing you are kind of a big deal? They really are a complete shambles of a business.
Two aspects of this which I don't understand :
1) How can the business infrastructure have been designed to be so vulnerable to a single point of failure ? I would have expected such mission-critical systems to have hot standbys running in geographically remote locations. Such a single point of failure could be susceptible to all sorts of failures, including terrorist attacks, fires, floods, ....
2) I have only seen mention of disruption at LHR and LGW - what was the situation at Manchester, Glasgow, ... ?
1) How can the business infrastructure have been designed to be so vulnerable to a single point of failure ? I would have expected such mission-critical systems to have hot standbys running in geographically remote locations. Such a single point of failure could be susceptible to all sorts of failures, including terrorist attacks, fires, floods, ....
2) I have only seen mention of disruption at LHR and LGW - what was the situation at Manchester, Glasgow, ... ?
jonny996 said:
No one has mentioned this happened on one of the hottest days, I will wager the DC provider got gready and has signed up to load shedding, this is were they step off grid power on to generator in times when the grid can't meet demand& the demand would have been high with Air con on max. Thing is with most shedding schemes you have to give control of your generators over to the grid & they don't tell you when they are switching you, if you line that up with poor change management you then have potential for a switch to happen at same time as generator maintenance.
Just a guess but a very well educated one
No chance - IT would have pretty much grounded the idea of using generators to peak lope on a critical site like this - I worked in a bank and if we took a power hit and went to generators during the processing time then we stayed on generators until the process time was complete for that day. Just a guess but a very well educated one
Those that look after IT and plan these systems dont take chances. There would be no way that they would have done this, Ill bet there was an issue with a generator maintenance or the likes.
limpsfield said:
schmalex said:
I flew BA LHR - YYZ yesterday. We were only delayed by 90 minutes which, having cancelled my flight from DOH - LHR on Friday 19th May with no alternative offered and then delayed my flight from LHR - MAD by 5 hours on Sunday 21st May, is the least disruption BA have caused to my travel plans over the past 3 weeks.
They really are a complete shambles of a business.
I am guessing you are kind of a big deal? They really are a complete shambles of a business.
Vaud said:
Outsourcing is fine when done properly. How far do you vertically integrate?
But it is never ever done properly - this is what happens - once you hand that control over to a 3rd party it is out of your hands. You think you have control but you dont. its over to them and they do mess up. especially if you outsource to a different country. different totally different. anonymous said:
[redacted]
Fuel supplies are measured in days - you will have the main bulk tanks rated for 7 days or more at full running load and day tanks sited near to the diesels that are filled on demand from the site bulk store tanks. Fuel supply to site normally on call-standby with a fuel supplier and would be there on site with tankers within the hour. With emissions control and all that these days power back up is designed to pull from the grid when it can, your data centres having dual string supplies where each and every piece of IT equipment is supplied from two independent supplies. one supply fails it takes power from the other supply seemlessly. This is design practice. The incoming supplies to site can be configured to automatically switch to source from the available supply and ensure that the site remains on utility. should utility fail then it will start and run the diesels. The UPS units providing power during switchover operations automatically.
pretty robust systems and takes a lot to make them fall over, although it can happen.
DC power systems are good at down/up but not if there was a sustained sag, you can't predict every failure state - even the best DC operators in the world suffer failure, it's how you make sure it doesn't happen again that's important. It's hilarious to read the accounts on here of someone who wrote a DR plan taking the high ground. No one outside BA knows what happened here.
Flew GLA-LHR-BUD today. All on time, all calm. Told every BA person I met that we knew it wasn't their fault and to keep smiling - they appreciated it and thanked me, saying not everyone was seeing it that way
Flew GLA-LHR-BUD today. All on time, all calm. Told every BA person I met that we knew it wasn't their fault and to keep smiling - they appreciated it and thanked me, saying not everyone was seeing it that way
4x4Tyke said:
I don't read his words that way.
I for one don't consider India any worse that remote staff elsewhere. However I will throw one cultural thing into the mix, but I don't blame this on the Indian IT sub-contractors, it is still a management failure, a failure to adjust properly.
We Brits like to complain and grumble, we are used to it, we expect it. This trait is not prevalent amongst the Indians I have worked with in IT. They are very restrained when they would have solid grounds to be complaining loudly. The word 'No' is almost taboo for them. This is especially the case when they perceive a power imbalance, you are older, more senior or experienced. I've found British management typically fail to appreciate and adjust to this difference.
We need to learn to take expressions like "Let me see", "I will try", "I will do my best" as usually meaning no, or you've missed something important.
If they get to "I have concerns" it really should be translated as meaning no f*** way is that feasible or reasonable.
Anything other than an empathic yes should be considered to mean no.
Having worked with a number of Indians, your description is accurate. Furthermore, I have learned that nodding of their heads does not indicate agreement, this means; "I hear what you are saying, but think you're an idiot". Instead, agreement is signalled by a side-to-side head "waggle".I for one don't consider India any worse that remote staff elsewhere. However I will throw one cultural thing into the mix, but I don't blame this on the Indian IT sub-contractors, it is still a management failure, a failure to adjust properly.
We Brits like to complain and grumble, we are used to it, we expect it. This trait is not prevalent amongst the Indians I have worked with in IT. They are very restrained when they would have solid grounds to be complaining loudly. The word 'No' is almost taboo for them. This is especially the case when they perceive a power imbalance, you are older, more senior or experienced. I've found British management typically fail to appreciate and adjust to this difference.
We need to learn to take expressions like "Let me see", "I will try", "I will do my best" as usually meaning no, or you've missed something important.
If they get to "I have concerns" it really should be translated as meaning no f*** way is that feasible or reasonable.
Anything other than an empathic yes should be considered to mean no.
Edited by 4x4Tyke on Monday 29th May 14:41
This reticence to be honest really is a pest.
I was just reminded that I had acquired a test evaluation of an expensive software module, if it worked it was going to solve a troublesome process, save a considerable amount of time and provide a lot of data in a very useful format. The evaluation was time limited, I installed it and initiated the trial, at the request of an Indian colleague, I went on holiday a few days after the install. A couple of weeks later, on my return, I enquired to the colleague if he had used it and found it to be useful "Yes", on loading the logs I could see he hadn't used it. Thankfully the software company were accommodating, and provided me an extension to really evaluate it.
Edited by zb on Monday 29th May 21:29
The update story rings true with me. My first thought was that either management or Infosec teams had demanded systems get patched following the wannacry drama and the patching had broken things.
Just saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
Just saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
wombleh said:
The update story rings true with me. My first thought was that either management or Infosec teams had demanded systems get patched following the wannacry drama and the patching had broken things.
Just saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
Confirms my post at 17:15 from Radio4 news?Just saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
wombleh said:
The update story rings true with me. My first thought was that either management or Infosec teams had demanded systems get patched following the wannacry drama and the patching had broken things.
Just saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
A power surge that affected two data centres? This is real life we're talking about, not some video game.Just saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
I could cope with the idea of a power issue taking out a data centre - it's very rare but it can happen. But two? That's some strange wiring...
wombleh said:
The update story rings true with me. My first thought was that either management or Infosec teams had demanded systems get patched following the wannacry drama and the patching had broken things.
Just saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
Someone somewhere is feeding the exec some serious crap. Of all the excuses in the IT bible to come up with this has to one of the worst I have heardJust saw this on reuters which has my bsometer off the scale:
BA Chief Executive Alex Cruz said the root of the problem, which also affected passengers trying to fly into Britain, had been a power surge on Saturday morning which hit BA's flight, baggage and communication systems. It was so strong it also rendered the back-up systems ineffective, he said.
Gassing Station | News, Politics & Economics | Top of Page | What's New | My Stuff