BA systems down globally
Discussion
anonymous said:
[redacted]
In a nutshell... no.I've worked for about 10 different clients (all well known brands, mainly global companies) over the last 20 years in a variety of business sectors, and have seen the rise and rise of IT offshoring to india since it started in earnest about 15 years ago. I have yet to see it achieve any of the promised quality, quantity or cost benefits that it was based on.
It's a scam, an accountant's / bungee manager's scam, and I can only look on with unsurprised amusement whenever I see the inevitable results come home to roost.
Purpleturtle summed it up pretty much spot on above and it's something I've been banging on about for at least 5-10 years as I've seen skilled jobs being removed from the UK market in the basis of lies and false economies. But I was accused of being racist and non-globalist the last time so don't worry about it anymore.
bga said:
Plenty of my clients have had similar positive experiences.
The common factor is that they all have been very selective about what is done offshore and/or outsourced.
Exactly. And generally a relationship built up over time, scaling services over time as trust builds up... rather than a big bang driven by procurement and the CFO.The common factor is that they all have been very selective about what is done offshore and/or outsourced.
The latter ones go wrong more in my experience.
In my experience offshore IT support has led to initial cost savings which look good then everything is lost and more in poor communication and service from the remote support location. This dealing with fairly specialist users. Probably works out well for target driven managers who only care about this years results.
Piersman2 said:
In a nutshell... no.
I've worked for about 10 different clients (all well known brands, mainly global companies) over the last 20 years in a variety of business sectors, and have seen the rise and rise of IT offshoring to india since it started in earnest about 15 years ago. I have yet to see it achieve any of the promised quality, quantity or cost benefits that it was based on.
It's a scam, an accountant's / bungee manager's scam, and I can only look on with unsurprised amusement whenever I see the inevitable results come home to roost.
Purpleturtle summed it up pretty much spot on above and it's something I've been banging on about for at least 5-10 years as I've seen skilled jobs being removed from the UK market in the basis of lies and false economies. But I was accused of being racist and non-globalist the last time so don't worry about it anymore.
Interesting reading this thread as there's a lot of echos of my own industry (building, I'm a sparky) in that promised cheapness, willingness and subservience are no substitute for skills and experience but so long as the man paying the money is being told what he wants to hear he doesn't question it even when it seems blatantly really not right.I've worked for about 10 different clients (all well known brands, mainly global companies) over the last 20 years in a variety of business sectors, and have seen the rise and rise of IT offshoring to india since it started in earnest about 15 years ago. I have yet to see it achieve any of the promised quality, quantity or cost benefits that it was based on.
It's a scam, an accountant's / bungee manager's scam, and I can only look on with unsurprised amusement whenever I see the inevitable results come home to roost.
Purpleturtle summed it up pretty much spot on above and it's something I've been banging on about for at least 5-10 years as I've seen skilled jobs being removed from the UK market in the basis of lies and false economies. But I was accused of being racist and non-globalist the last time so don't worry about it anymore.
The worst is when they come back to the actually skilled workers but think they need to treat you like you're an idiot/2nd guess everything you do as thats what they're used to now!
I'll stop before I derail this too much...
Vaud said:
bga said:
Plenty of my clients have had similar positive experiences.
The common factor is that they all have been very selective about what is done offshore and/or outsourced.
Exactly. And generally a relationship built up over time, scaling services over time as trust builds up... rather than a big bang driven by procurement and the CFO.The common factor is that they all have been very selective about what is done offshore and/or outsourced.
The latter ones go wrong more in my experience.
Too often organisations want to outsource a problem. Often because the cost to fix the problem has been presented and the great and the good think outsourcing is the answer
If your environment is in good condition, well understood and documented then outsourcing it should be relatively low risk (though I would still be reticent to outsource mission critical, life/death stuff). But in that scenario the cost savings are unlikely to stack up any more. Especially for organisations of any scale. Other than via bookkeeping jiggery-pokery...
robinessex said:
Accountants, they know the cost of everything, the value of nothing !
It's also a leadership issue.Often the CEO/COO/CIO will have wanted to buy a fully modernised service - the gold standard... but once the business case/cost model gets to the CFO there is downward pressure to simplify/delay transformation projects, etc... so it becomes "your mess for less" based on labour arbitrage.
Yet they still sell internally that they have bought the gold standard. Where in reality they (knowingly) bought the bronze level standard with in some areas, no SLA... but egos don't allow the truth...
clonmult said:
A DC could be potentially rather large - thousands of servers. Use of a UPS across all of them would possibly not be required, it would be applied on a system-by-system basis. So systems deemed critical would be behind a UPS, and even if they werent, the failover to the secondary DC would typically be handled fairly quickly (few hours?).
Which is why the claims of it being a power problem are increasingly unbelievable.
Now you're into the realms of DR planning. Which is far less likely to have been done well IME Which is why the claims of it being a power problem are increasingly unbelievable.
In times when costs are important (when aren't they?) I'd argue strongly that anything not mission critical shouldn't exist anyway. And it certainly shouldn't be taking up expensive data centre real estate.
clonmult said:
Murph7355 said:
MitchT said:
Overlooking sarcasm, is there a reason why every electronic device that was rated for use in such critical applications couldn't have a battery to provide a buffer of emergency power?
In a decent data centre in essence they do, it's called a UPS. It's not intended to provide power on its own for very long, just enough to regain a stable supply either via the grid or generators (or to shut down gracefully). It's centrally provided within the DC.
Of course there's no law that says you must provide one. But I cannot imagine a company like BA not having this functionality in its DCs. If it doesn't, I refer back to my earlier root cause hypothesis
Which is why the claims of it being a power problem are increasingly unbelievable.
So BA have a vacancy for a "Global PR Manager" on their careers website
https://jobs.ba.com/jobs/vacancy/global-pr-manager...
I'm sure it's been there since before the weekend, but might explain why their handling of the crisis over the weekend with Captain Hi Viz was so poor, if the previous PR Manager has already left, or is currently working their notice and said "sod off I'm not handling this mess"
Bet a few of the applicants are reconsidering their salary expectations now!
https://jobs.ba.com/jobs/vacancy/global-pr-manager...
I'm sure it's been there since before the weekend, but might explain why their handling of the crisis over the weekend with Captain Hi Viz was so poor, if the previous PR Manager has already left, or is currently working their notice and said "sod off I'm not handling this mess"
Bet a few of the applicants are reconsidering their salary expectations now!
PurpleTurtle said:
At the start they throw their best people at it. A handful are good - we have had one or two over the years who we would have hired, had outsourcing not occurred. In the main however the Indian staff are fresh out of college, have been chucked on a course in whatever technology it is they are to support, then thrown at the Production application to support it. They have very little experience in business in general, and zero in the business they are supposed to be supporting. If a problem can be scripted down to the Nth degree such that following steps 1-10 to the letter always results in the same outcome then they are good, hard workers. Those kinds of thing are the exception, rather than the rule. I digress, but I should add that when I started in IT nearly 25 years ago the accepted wisdom was that you added no value for 2yrs, you were a trainee until then, and only really became trusted and productive after 2yrs. In my background nobody inexperienced was let anywhere near Production (live) systems. You cut your teeth in Development.
The problem comes when things get any more complicated than basic scripted tasks. Anything requiring some analytical thinking, some liason with users or some deep understanding of the business is beyond the Outsource staff, they just don't have the experience. To compound that risk, there is the Indian cultural norm of not wanting to 'lose face' and not wanting to admit that they do no know something.
Every week I have this kind of conversation:
Me: "Program XYZ has failed with return code ABC123. Do you know what that means and can you fix it?"
Offshore Techie: "Yes"
Me: "OK, what does it mean?"
OT: (long silence, bit of scratching about, more silence): " err, I don't know"
It is mind-numbingly tedious that they cannot just grasp the simple concept of Western business that people we would much rather they said "I'm not sure, but I can find out". I have repeated this until I am blue in the face, nothing changes.
Add in to the mix that all of these bodies in a Bangalore office ALL want to be the manager not the techie (it's a status thing) then none of them help each other out - they are competing with each other for promotion. My man who doesn't know what ABC123 means will never go to his colleague to find out - they would rather bluster around on their own completely in the dark before eventually admitting defeat than seek help. As long as they do something (even it it's not the right thing) to in some way justify closure of their incident ticket within SLA they move on to the next one.
I have had five years of dealing with this, and know many friends in different industries who will tell you the exact same tale.
Excellent summary, exactly what I have experienced in companies who've outsourced to India. The problem comes when things get any more complicated than basic scripted tasks. Anything requiring some analytical thinking, some liason with users or some deep understanding of the business is beyond the Outsource staff, they just don't have the experience. To compound that risk, there is the Indian cultural norm of not wanting to 'lose face' and not wanting to admit that they do no know something.
Every week I have this kind of conversation:
Me: "Program XYZ has failed with return code ABC123. Do you know what that means and can you fix it?"
Offshore Techie: "Yes"
Me: "OK, what does it mean?"
OT: (long silence, bit of scratching about, more silence): " err, I don't know"
It is mind-numbingly tedious that they cannot just grasp the simple concept of Western business that people we would much rather they said "I'm not sure, but I can find out". I have repeated this until I am blue in the face, nothing changes.
Add in to the mix that all of these bodies in a Bangalore office ALL want to be the manager not the techie (it's a status thing) then none of them help each other out - they are competing with each other for promotion. My man who doesn't know what ABC123 means will never go to his colleague to find out - they would rather bluster around on their own completely in the dark before eventually admitting defeat than seek help. As long as they do something (even it it's not the right thing) to in some way justify closure of their incident ticket within SLA they move on to the next one.
I have had five years of dealing with this, and know many friends in different industries who will tell you the exact same tale.
gavsdavs said:
PurpleTurtle said:
At the start they throw their best people at it. A handful are good - we have had one or two over the years who we would have hired, had outsourcing not occurred. In the main however the Indian staff are fresh out of college, have been chucked on a course in whatever technology it is they are to support, then thrown at the Production application to support it. They have very little experience in business in general, and zero in the business they are supposed to be supporting. If a problem can be scripted down to the Nth degree such that following steps 1-10 to the letter always results in the same outcome then they are good, hard workers. Those kinds of thing are the exception, rather than the rule. I digress, but I should add that when I started in IT nearly 25 years ago the accepted wisdom was that you added no value for 2yrs, you were a trainee until then, and only really became trusted and productive after 2yrs. In my background nobody inexperienced was let anywhere near Production (live) systems. You cut your teeth in Development.
The problem comes when things get any more complicated than basic scripted tasks. Anything requiring some analytical thinking, some liason with users or some deep understanding of the business is beyond the Outsource staff, they just don't have the experience. To compound that risk, there is the Indian cultural norm of not wanting to 'lose face' and not wanting to admit that they do no know something.
Every week I have this kind of conversation:
Me: "Program XYZ has failed with return code ABC123. Do you know what that means and can you fix it?"
Offshore Techie: "Yes"
Me: "OK, what does it mean?"
OT: (long silence, bit of scratching about, more silence): " err, I don't know"
It is mind-numbingly tedious that they cannot just grasp the simple concept of Western business that people we would much rather they said "I'm not sure, but I can find out". I have repeated this until I am blue in the face, nothing changes.
Add in to the mix that all of these bodies in a Bangalore office ALL want to be the manager not the techie (it's a status thing) then none of them help each other out - they are competing with each other for promotion. My man who doesn't know what ABC123 means will never go to his colleague to find out - they would rather bluster around on their own completely in the dark before eventually admitting defeat than seek help. As long as they do something (even it it's not the right thing) to in some way justify closure of their incident ticket within SLA they move on to the next one.
I have had five years of dealing with this, and know many friends in different industries who will tell you the exact same tale.
Excellent summary, exactly what I have experienced in companies who've outsourced to India. The problem comes when things get any more complicated than basic scripted tasks. Anything requiring some analytical thinking, some liason with users or some deep understanding of the business is beyond the Outsource staff, they just don't have the experience. To compound that risk, there is the Indian cultural norm of not wanting to 'lose face' and not wanting to admit that they do no know something.
Every week I have this kind of conversation:
Me: "Program XYZ has failed with return code ABC123. Do you know what that means and can you fix it?"
Offshore Techie: "Yes"
Me: "OK, what does it mean?"
OT: (long silence, bit of scratching about, more silence): " err, I don't know"
It is mind-numbingly tedious that they cannot just grasp the simple concept of Western business that people we would much rather they said "I'm not sure, but I can find out". I have repeated this until I am blue in the face, nothing changes.
Add in to the mix that all of these bodies in a Bangalore office ALL want to be the manager not the techie (it's a status thing) then none of them help each other out - they are competing with each other for promotion. My man who doesn't know what ABC123 means will never go to his colleague to find out - they would rather bluster around on their own completely in the dark before eventually admitting defeat than seek help. As long as they do something (even it it's not the right thing) to in some way justify closure of their incident ticket within SLA they move on to the next one.
I have had five years of dealing with this, and know many friends in different industries who will tell you the exact same tale.
clonmult said:
57 Chevy said:
anonymous said:
[redacted]
Comment from a Times article.
From the IT rumour mill
Allegedly, the staff at the Indian data centre were told to apply some security fixes to the computers in the data centre. The BA IT systems have two, parallel systems to cope with updates. What was supposed to happen was that they apply the fixes to the computers of the secondary system, and when all is working, apply to the computers of the primary system. In this way, the programs all keep running without any interruption.
What they actually did was apply the patches to _all_ the computers.
From the IT rumour mill
Allegedly, the staff at the Indian data centre were told to apply some security fixes to the computers in the data centre. The BA IT systems have two, parallel systems to cope with updates. What was supposed to happen was that they apply the fixes to the computers of the secondary system, and when all is working, apply to the computers of the primary system. In this way, the programs all keep running without any interruption.
What they actually did was apply the patches to _all_ the computers.
Edited to add link... https://www.theregister.co.uk/2012/06/25/rbs_natwe...
https://www.theregister.co.uk/2012/06/26/rbs_natwe...
https://www.theregister.co.uk/2012/06/28/rbs_job_c...
Edited by 57 Chevy on Tuesday 30th May 14:06
No way could you get approval for an outage to services across both DCs and all systems. And I find it highly unlikely that anyone in an outsourced support role would actually go ahead and start making such changes without being asked to do so as a result of change request .... but again, it depends on the processes in place at BA.
And the chances of the real truth coming out on this are really improbable.
Vaud said:
It's also a leadership issue.
Often the CEO/COO/CIO will have wanted to buy a fully modernised service - the gold standard... but once the business case/cost model gets to the CFO there is downward pressure to simplify/delay transformation projects, etc... so it becomes "your mess for less" based on labour arbitrage.
Yet they still sell internally that they have bought the gold standard. Where in reality they (knowingly) bought the bronze level standard with in some areas, no SLA... but egos don't allow the truth...
Or alternatively you have the conversation where they specified at length what they want for a critical business system, you have given them a break down of how that can be delivered, and they start by saying:Often the CEO/COO/CIO will have wanted to buy a fully modernised service - the gold standard... but once the business case/cost model gets to the CFO there is downward pressure to simplify/delay transformation projects, etc... so it becomes "your mess for less" based on labour arbitrage.
Yet they still sell internally that they have bought the gold standard. Where in reality they (knowingly) bought the bronze level standard with in some areas, no SLA... but egos don't allow the truth...
"Couldn't you just..."
From the telegraph report here's how I see it. It's bank holiday weekend so maintenance/testing is planned. Wrong mains breaker is shutdown and unplanned things start to happen because ups fails, so someone says oh s
t, quick, switch mains breaker back on. Now there is an inrush of current because all circuit breakers to all the server cabinets are on. I still don't get why the dr site didn't kick in though. But it does explain that the "SURGE" was not a electrical supply company issue.
Or it could just be a smoke screen for something else of course.
t, quick, switch mains breaker back on. Now there is an inrush of current because all circuit breakers to all the server cabinets are on. I still don't get why the dr site didn't kick in though. But it does explain that the "SURGE" was not a electrical supply company issue.Or it could just be a smoke screen for something else of course.
Gassing Station | News, Politics & Economics | Top of Page | What's New | My Stuff