BA systems down globally

Author
Discussion

saaby93

32,038 posts

179 months

Wednesday 31st May 2017
quotequote all
anonymous said:
[redacted]
We'll just test the failure systems to see what happens at the most critical time?

Chernobyl

Du1point8

21,612 posts

193 months

Wednesday 31st May 2017
quotequote all
Nardiola said:
gavsdavs said:
PurpleTurtle said:
At the start they throw their best people at it. A handful are good - we have had one or two over the years who we would have hired, had outsourcing not occurred. In the main however the Indian staff are fresh out of college, have been chucked on a course in whatever technology it is they are to support, then thrown at the Production application to support it. They have very little experience in business in general, and zero in the business they are supposed to be supporting. If a problem can be scripted down to the Nth degree such that following steps 1-10 to the letter always results in the same outcome then they are good, hard workers. Those kinds of thing are the exception, rather than the rule. I digress, but I should add that when I started in IT nearly 25 years ago the accepted wisdom was that you added no value for 2yrs, you were a trainee until then, and only really became trusted and productive after 2yrs. In my background nobody inexperienced was let anywhere near Production (live) systems. You cut your teeth in Development.

The problem comes when things get any more complicated than basic scripted tasks. Anything requiring some analytical thinking, some liason with users or some deep understanding of the business is beyond the Outsource staff, they just don't have the experience. To compound that risk, there is the Indian cultural norm of not wanting to 'lose face' and not wanting to admit that they do no know something.

Every week I have this kind of conversation:

Me: "Program XYZ has failed with return code ABC123. Do you know what that means and can you fix it?"
Offshore Techie: "Yes"
Me: "OK, what does it mean?"
OT: (long silence, bit of scratching about, more silence): " err, I don't know"

It is mind-numbingly tedious that they cannot just grasp the simple concept of Western business that people we would much rather they said "I'm not sure, but I can find out". I have repeated this until I am blue in the face, nothing changes.

Add in to the mix that all of these bodies in a Bangalore office ALL want to be the manager not the techie (it's a status thing) then none of them help each other out - they are competing with each other for promotion. My man who doesn't know what ABC123 means will never go to his colleague to find out - they would rather bluster around on their own completely in the dark before eventually admitting defeat than seek help. As long as they do something (even it it's not the right thing) to in some way justify closure of their incident ticket within SLA they move on to the next one.

I have had five years of dealing with this, and know many friends in different industries who will tell you the exact same tale.
Excellent summary, exactly what I have experienced in companies who've outsourced to India.
Exactly what I'm working with right now!
You guys missed out the most important bit.

Once the outsource team have been in X position for Y time, they all sod off as they now have experience and want to better themselves and the process of a new team comes in, there is no handover as the old set basically:

a) Dont document crap as it could mean someone else knows the knowledge.
b) Refuse to impart their knowledge properly as they dont want their replacement to know everything.
c) Culuturally believe that if they know something you dont, then they are actually better at the job than you.

So despite assurances that the team we trained up have documented everything and are teaching their peers, they dont and the account manager of the outsource company (usually someone in the UK onshore, that is more sales) gets it in the neck and has to deal with the fallout kicking his own staff in his own company to stop doing it.

Did anyone have the following genius scam?

Staff A works on project B for me, I can only use communicator to get hold of him and he sets it to away constantly (probably learned that in the past not to set to active) and he ignores emails/communicator or is very slow on the pickup.

My colleague (sat not more than 10ft away from me) working on project C is swearing out aloud that his outsource bod is MIA and he cant get hold of him and cant figure out why he is not at his desk and in meetings as there are none planned.

We go out for a beer after work one day and the conversation gets onto how bad outsourcing is...

I mention that I can never get one of my "experts" in the afternoon and even when I do the work is so slow I might as well never bother, he mentions that he has the opposite issue, there is a eureka moment and we double check the guys name/email/communicator... Same fking guy!!

Get some evidence together by asking for timesheets and billables for said outsource bod, all signed off by his manager, its transpires that we are both being charged for 10 hours work a day by the same guy for 2 different projects signed off by the same manager.

A conference call was organised, with just me on the invite, outsourced person, and manager.... not on the invite was the account manager (told to get his ass in the office ASAP) and my colleague (due to them realising we know what's going on) who were both in the room too.

I ask them to both verify the timesheet/billable hours for my project, they confirm.... tell them to hold for a second... then colleague comes on and asks them to verify his timesheet/billable hours on his project... They realise the gig is up and start making excuses, account manager has twigged what is happening and is fuming...

The outsource manager then states that it is probably a miss-communication on our side and we are wrong, despite emails/communicator/name are all identical on timesheets etc (can't answer that other than state a lot of people with that name exist in the company)... They had been doing that only for 2/3 weeks and thought it was acceptable.

Manager and half the team in the outsource company (16 staff) were fired from our project (probably still kept on and told to cover tracks better next time), we then subsequently removed the vendor from our records and went with new outsource team until onshore staff could be hired.

Laplace

1,090 posts

183 months

Wednesday 31st May 2017
quotequote all
Human error is a possibility here. A perfectly feasible scenario is that the UPS was undergoing maintenance and the attending engineer made a switching error, it does happen.

When taking a UPS offline for essential/routine maintenance there is a switching process which transfers the load onto raw mains, or "bypass", allowing you to carry out planned maintenance on the now safely isolated UPS. During this process the load is at risk from mains fluctuations, transients and brownouts etc. Ideally you would have a dual system where you can transfer the load onto the A side while you work on the B side, and vice versa.

Typically you first transfer the UPS load to what we call "static bypass" by either shutting down the inverter or commanding "static bypass" (depending on the UPS). This is a make before break process handled by the UPS in milliseconds, assuming the bypass is within tolerance ( the inverter will "track" the bypass supply so that if it needs to transfer to bypass it can make a synchronous transfer). It's from this point that the manual switching by the user takes place. In order to finally transfer the load from "static bypass" to full "bypass" a few breakers need to be thrown in the correct sequence; if not, lights out. If this is what happened then the panicked user will likely have corrected his error post haste and I believe this is the so called "surge" being reported; reading between the lines that is.

I've seen guys successfully transfer a UPS to bypass with no interruption to the load but accidentally open the bypass supply breaker thinking it was the UPS mains input breaker. There are numerous ways human error could have dropped the load during switching/maintenance.

For those that are interested, a typical UPS block diagram is below. Once the load has been transferred to the static switch and the inverter is off Q3 would be closed followed by opening Q4. You would then open Q1, Q6 and Q5 to make the UPS volt free - don't touch Q2 or Q3!

Someone asked previously about overloading a UPS. It varies, but as an example a 2005 era 90 NET from Chloride, a system I've seen in many DCs of this type, will give you about 125% of rated load for about 10 mins before transferring to bypass or 1 min at 150% rated load before transferring to bypass, assuming the bypass is within tolerance. I don't think it's relevant in this instance but there you go.




I wonder who has the UPS maintenance contract down there, I may have to do some digging biggrin

GCH

3,993 posts

203 months

Wednesday 31st May 2017
quotequote all
saaby93 said:
anonymous said:
[redacted]
We'll just test the failure systems to see what happens at the most critical time?

Chernobyl
B*llocks - beaten to it.

Exactly what I was going to say hehe

Nardiola

1,173 posts

220 months

Wednesday 31st May 2017
quotequote all
Du1point8 said:
Nardiola said:
gavsdavs said:
PurpleTurtle said:
At the start they throw their best people at it. A handful are good - we have had one or two over the years who we would have hired, had outsourcing not occurred. In the main however the Indian staff are fresh out of college, have been chucked on a course in whatever technology it is they are to support, then thrown at the Production application to support it. They have very little experience in business in general, and zero in the business they are supposed to be supporting. If a problem can be scripted down to the Nth degree such that following steps 1-10 to the letter always results in the same outcome then they are good, hard workers. Those kinds of thing are the exception, rather than the rule. I digress, but I should add that when I started in IT nearly 25 years ago the accepted wisdom was that you added no value for 2yrs, you were a trainee until then, and only really became trusted and productive after 2yrs. In my background nobody inexperienced was let anywhere near Production (live) systems. You cut your teeth in Development.

The problem comes when things get any more complicated than basic scripted tasks. Anything requiring some analytical thinking, some liason with users or some deep understanding of the business is beyond the Outsource staff, they just don't have the experience. To compound that risk, there is the Indian cultural norm of not wanting to 'lose face' and not wanting to admit that they do no know something.

Every week I have this kind of conversation:

Me: "Program XYZ has failed with return code ABC123. Do you know what that means and can you fix it?"
Offshore Techie: "Yes"
Me: "OK, what does it mean?"
OT: (long silence, bit of scratching about, more silence): " err, I don't know"

It is mind-numbingly tedious that they cannot just grasp the simple concept of Western business that people we would much rather they said "I'm not sure, but I can find out". I have repeated this until I am blue in the face, nothing changes.

Add in to the mix that all of these bodies in a Bangalore office ALL want to be the manager not the techie (it's a status thing) then none of them help each other out - they are competing with each other for promotion. My man who doesn't know what ABC123 means will never go to his colleague to find out - they would rather bluster around on their own completely in the dark before eventually admitting defeat than seek help. As long as they do something (even it it's not the right thing) to in some way justify closure of their incident ticket within SLA they move on to the next one.

I have had five years of dealing with this, and know many friends in different industries who will tell you the exact same tale.
Excellent summary, exactly what I have experienced in companies who've outsourced to India.
Exactly what I'm working with right now!
You guys missed out the most important bit.

Once the outsource team have been in X position for Y time, they all sod off as they now have experience and want to better themselves and the process of a new team comes in, there is no handover as the old set basically:

a) Dont document crap as it could mean someone else knows the knowledge.
b) Refuse to impart their knowledge properly as they dont want their replacement to know everything.
c) Culuturally believe that if they know something you dont, then they are actually better at the job than you.

So despite assurances that the team we trained up have documented everything and are teaching their peers, they dont and the account manager of the outsource company (usually someone in the UK onshore, that is more sales) gets it in the neck and has to deal with the fallout kicking his own staff in his own company to stop doing it.

Did anyone have the following genius scam?

Staff A works on project B for me, I can only use communicator to get hold of him and he sets it to away constantly (probably learned that in the past not to set to active) and he ignores emails/communicator or is very slow on the pickup.

My colleague (sat not more than 10ft away from me) working on project C is swearing out aloud that his outsource bod is MIA and he cant get hold of him and cant figure out why he is not at his desk and in meetings as there are none planned.

We go out for a beer after work one day and the conversation gets onto how bad outsourcing is...

I mention that I can never get one of my "experts" in the afternoon and even when I do the work is so slow I might as well never bother, he mentions that he has the opposite issue, there is a eureka moment and we double check the guys name/email/communicator... Same fking guy!!

Get some evidence together by asking for timesheets and billables for said outsource bod, all signed off by his manager, its transpires that we are both being charged for 10 hours work a day by the same guy for 2 different projects signed off by the same manager.

A conference call was organised, with just me on the invite, outsourced person, and manager.... not on the invite was the account manager (told to get his ass in the office ASAP) and my colleague (due to them realising we know what's going on) who were both in the room too.

I ask them to both verify the timesheet/billable hours for my project, they confirm.... tell them to hold for a second... then colleague comes on and asks them to verify his timesheet/billable hours on his project... They realise the gig is up and start making excuses, account manager has twigged what is happening and is fuming...

The outsource manager then states that it is probably a miss-communication on our side and we are wrong, despite emails/communicator/name are all identical on timesheets etc (can't answer that other than state a lot of people with that name exist in the company)... They had been doing that only for 2/3 weeks and thought it was acceptable.

Manager and half the team in the outsource company (16 staff) were fired from our project (probably still kept on and told to cover tracks better next time), we then subsequently removed the vendor from our records and went with new outsource team until onshore staff could be hired.
Ah-ha! The outsourcer I'm currently working with has a way around this though, they have the same email address with a series of numbers after them. Sometimes I'll get emails from Outsource.Guy@Offshore.com, other days Outsource.Guy2@Offshore.com and so on. His Skype account is generally 'Away' as you mention, but sometimes they're different statuses, leading me to believe he's probably got some VMs for each of these accounts.

The Account Manager thing is so true, in my case a guy from the US, he's employed because he's white and speaks English (and because of his experience obviously) but they don't listen to him, he's purely the face, but the Indian chaps who run the actual business about only interested in invoicing the client, delivering as little as possible, with as many excuses as possible, they don't care about their relationship with the client once the deal is done.

ruggedscotty

5,629 posts

210 months

Wednesday 31st May 2017
quotequote all
Laplace said:
Human error is a possibility here. A perfectly feasible scenario is that the UPS was undergoing maintenance and the attending engineer made a switching error, it does happen.

When taking a UPS offline for essential/routine maintenance there is a switching process which transfers the load onto raw mains, or "bypass", allowing you to carry out planned maintenance on the now safely isolated UPS. During this process the load is at risk from mains fluctuations, transients and brownouts etc. Ideally you would have a dual system where you can transfer the load onto the A side while you work on the B side, and vice versa.

Typically you first transfer the UPS load to what we call "static bypass" by either shutting down the inverter or commanding "static bypass" (depending on the UPS). This is a make before break process handled by the UPS in milliseconds, assuming the bypass is within tolerance ( the inverter will "track" the bypass supply so that if it needs to transfer to bypass it can make a synchronous transfer). It's from this point that the manual switching by the user takes place. In order to finally transfer the load from "static bypass" to full "bypass" a few breakers need to be thrown in the correct sequence; if not, lights out. If this is what happened then the panicked user will likely have corrected his error post haste and I believe this is the so called "surge" being reported; reading between the lines that is.

I've seen guys successfully transfer a UPS to bypass with no interruption to the load but accidentally open the bypass supply breaker thinking it was the UPS mains input breaker. There are numerous ways human error could have dropped the load during switching/maintenance.

For those that are interested, a typical UPS block diagram is below. Once the load has been transferred to the static switch and the inverter is off Q3 would be closed followed by opening Q4. You would then open Q1, Q6 and Q5 to make the UPS volt free - don't touch Q2 or Q3!

Someone asked previously about overloading a UPS. It varies, but as an example a 2005 era 90 NET from Chloride, a system I've seen in many DCs of this type, will give you about 125% of rated load for about 10 mins before transferring to bypass or 1 min at 150% rated load before transferring to bypass, assuming the bypass is within tolerance. I don't think it's relevant in this instance but there you go.




I wonder who has the UPS maintenance contract down there, I may have to do some digging biggrin
Very good - but......

There would probably be more than one UPS being used and very likey dual string at least. You would never have to put a UPS into bypass, that is last resort and even then if you had then you would have the other string available. This smacks of something more major, like why would you resort to putting your super critical IT infrastructure into bypass ? The system is usually configured that you can one UPS out for maintenance without any effect on the other units or the IT load. Even then you have it set up that you can have another UPS unit fail and still run with no issue.

A good few years ago I was involved in a UPS program of works that involved installing a new ups unit, The issue that we had was that they didn't talk the new unit with the old units and we had to upgrade the old ones to the new software. Trouble was that the old units couldn't be upgraded live so they had to be taken out of service. In a live working environment.

That was a nightmare to organise, getting permission from the business and sorting it out so that we could do it with minimum impact to the business. So much was involved behind the scenes, we decided to rely on the other string, so we had to survey the whole site and ensure that there was dual string capability on every item, and that it was true dual string and that both supplies had not been fed in error from the one source. We then had to prove that under all scenarios we were able to cope, plan was to place one string onto bypass and carry out the work, During this process we were running the affected string off raw mains so we decided that we would have that on generator. When we made that decision we had switching programs thought out and in place so that we ran as best as we could. we even did pre checks to ensure that if we had our IT on generators and we had power cut the other string going to generator would not affect the load characteristics or disrupt the power quality and knockon the IT.

Absolute nightmare.

All I can say is something terrible must have happened with BA.

eliot

11,444 posts

255 months

Wednesday 31st May 2017
quotequote all
I think we are applying comercial datacentre principles - but i'm guessing (by location) these are privately owned DCs. In my experience the quality, understanding and maintenance of them is far lower.
A commercial provider lives or dies by the availability of the facilities and customers can vote with thier feet.
The temptation of private dc owners to cut costs and avoid preventative maintenance is very high.

dmsims

6,541 posts

268 months

Wednesday 31st May 2017
quotequote all
Report in the press (sepculative) that heat may have played a part

http://www.dailymail.co.uk/news/article-4556640/Di...

from that article "Because of the high temperatures last year, staff were having to hose the top of the building down to keep it cool."

seriously ?

and "The Daily Mail has learned that a small team of experienced engineers responsible for maintaining BA’s data centres and back-up systems were made redundant last year."

Du1point8

21,612 posts

193 months

Wednesday 31st May 2017
quotequote all
anonymous said:
[redacted]
Thats why the outsource team is run via a manager in the UK with the power to rip their balls off, or worse case scenario, we go across there and make their lives hell at their expense.

I have sent home the outsource team that arrived in the UK and revoked their visas when I saw both experts had a collective total of 12 weeks coding experience and no business experience.

They stayed in London 2 nights and flew home... the outsource company were warned not to take the piss.

Countdown

39,977 posts

197 months

Wednesday 31st May 2017
quotequote all
robinessex said:
Accountants, they know the cost of everything, the value of nothing !
biggrin

In some cases Accountants are told by the FD/CEO that "We want to do Option X, please make X the most viable/profitable option".

Quite recently I've provided costings that show outsourcing Payroll will be 25% more expensive for a more basic service. However I'm told that moving to a Shared Service provider will "improve business resilience" and therefore that's the road that we will go down. Now I'm confident that the staff who will be left "in house" will be able to pick up the pieces if and when when the SHTF but it's a bit demoralising when pretty much everybody in the team knows this is being done as a "vanity project" for the benefit of somebody's CV.



Equilibrium25

653 posts

135 months

Wednesday 31st May 2017
quotequote all
Piersman2 said:
In a nutshell... no.

I've worked for about 10 different clients (all well known brands, mainly global companies) over the last 20 years in a variety of business sectors, and have seen the rise and rise of IT offshoring to india since it started in earnest about 15 years ago. I have yet to see it achieve any of the promised quality, quantity or cost benefits that it was based on.

It's a scam, an accountant's / bungee manager's scam, and I can only look on with unsurprised amusement whenever I see the inevitable results come home to roost.

Purpleturtle summed it up pretty much spot on above and it's something I've been banging on about for at least 5-10 years as I've seen skilled jobs being removed from the UK market in the basis of lies and false economies. But I was accused of being racist and non-globalist the last time so don't worry about it anymore.
+1, a million times over

I engage with a range of outsourced IT providers on a daily basis, all providing services to blue chip clients. The service that they provide, the lack of skills that they bring and the massive avoidance of taking responsibility is disastrous. It's a scam at the senior levels and an exercise in avoidance and obfuscation at the junior levels. It's getting worse as well as the blind are now being led by the blind.

55palfers

5,914 posts

165 months

Wednesday 31st May 2017
quotequote all
I am not an IT bod, just a consumer of corporate "IT solutions"

So much of this rings true.

A brilliant thread and thanks to all those who have taken the time to contribute.

lemmingjames

7,462 posts

205 months

Wednesday 31st May 2017
quotequote all
Laplace said:
I wonder who has the UPS maintenance contract down there, I may have to do some digging biggrin
Probably ISS who havent done any maintenance on the UPS's since they went active (it wouldnt be the first time)


gavsdavs

1,203 posts

127 months

Wednesday 31st May 2017
quotequote all
speedyman said:
........ I still don't get why the dr site didn't kick in though.........
Y'know - If only it were that simple.

Proper DR is complex and some key decisions have to be taken (especially around replication, you need to start copying stuff back the other way - which /can/ mean thing like your DR storage arrays start overwriting your prod arrays (if you automate things).

Generally quite a few manual levers need to get pulled to get things up and running, never mind running full DR (where the DR site becomes the primary site for storage replication)

SunsetZed

2,257 posts

171 months

Wednesday 31st May 2017
quotequote all
gavsdavs said:
speedyman said:
........ I still don't get why the dr site didn't kick in though.........
Y'know - If only it were that simple.

Proper DR is complex and some key decisions have to be taken (especially around replication, you need to start copying stuff back the other way - which /can/ mean thing like your DR storage arrays start overwriting your prod arrays (if you automate things).

Generally quite a few manual levers need to get pulled to get things up and running, never mind running full DR (where the DR site becomes the primary site for storage replication)
This, it's why you need good people working in key positions 24/7 for companies such as BA, and most large organisations.

babatunde

736 posts

191 months

Wednesday 31st May 2017
quotequote all
couple of things,
I wrote my Masters Thesis (1997) on how outsourcing a company's core competency is a shortcut to failure and how for many companies IT is their core competency. Outsource your cleaners if you must but outsourcing IT in a logistics company is as stupid as outsourcing R&D


Spent many a year designing, building and working in Server rooms and NO I repeat NO proper server room has a single UPS.

When building Racks the redundant power supplies on each individual server will be plugged into a different power source, which are attached to separate UPS's, anything less than this is unacceptable. Redundency is designed into Servers, and the whole Server Room environment is designed around redundancy

SO either it was true amateur hour or they are lying.

Puggit

48,486 posts

249 months

Wednesday 31st May 2017
quotequote all
Telegraph article said:
A spokesman for Mr Walsh told The Telegraph: “Alex has Willie’s absolute support. As far as Willie is concerned, this has nothing to do with the decisions Alex has made.”
This would backup the statements on this thread recently that it's WW/IAG's fault and not AC's.

Equilibrium25

653 posts

135 months

Wednesday 31st May 2017
quotequote all
SunsetZed said:
This, it's why you need good people working in key positions 24/7 for companies such as BA, and most large organisations.
Absolutely agreed, it's the decision making, problem solving skills and holistic knowledge that are needed when things go wrong. That's when properly run companies get value returned from all of their ongoing IT investments.

The days when everything runs well and to the script are the ones that any fool can run things. Outsourcing is based on those days and gradually ratchets up the risk of the bad day occurring - as ongoing mismanagement (including both people and systems management) builds up a stack of problems waiting to bite. Once things don't go to the script, no-one has the capability to find a solution.

jonny996

2,618 posts

218 months

Wednesday 31st May 2017
quotequote all
The sad fact is this will not stop any of the big companies outsourcing & with the emergence of the cloud the CTO's will be rushing in blind to be there first.

oyster

12,609 posts

249 months

Wednesday 31st May 2017
quotequote all
Puggit said:
anonymous said:
[redacted]
Many organisations I've spoken to have moved over to India and come back.

Now they're all starting the journey to the public cloud instead, and the IT circle of life continues...
Noting that at each phase of a cycle, a senior IT and/or CFO/CEO will have been deemed a success and received a hefty bonus.


For me the real test of whether the ever-shifting onshore/offshore model is correct is what privately owned companies are doing. The companies whose planning vision is more than just next financial quarter reporting and next year's bonus scheme.