Natwest debarcle - ramifications for offshoring?
Discussion
Everyone is aware of the amount of work that has been offshored over recent years, not just to India, but other parts of the world also.
In light of the current issues with Natwest, which can at least partially be attributed to offshoring, what are thoughts on whether this could be some form of turning point?
Banks are making massive profits, which are in part down to cost saving in this way, but has the penny finally dropped, and they realised that price is not the ONLY consideration?
In which case, this could actually be GOOD for the UK IT industry...
or will it just be forgotten in 6 months and everything stays the same?
Thoughts anyone?
In light of the current issues with Natwest, which can at least partially be attributed to offshoring, what are thoughts on whether this could be some form of turning point?
Banks are making massive profits, which are in part down to cost saving in this way, but has the penny finally dropped, and they realised that price is not the ONLY consideration?
In which case, this could actually be GOOD for the UK IT industry...
or will it just be forgotten in 6 months and everything stays the same?
Thoughts anyone?
thetapeworm said:
We still make mistakes in the UK and chances are any overseas issues will be blamed on resource here not managing things correctly... or maybe that's just how it works with my lot.
The thing is , as shown below, we have many many years of experience in the UK of resolving issues such as this. India does not.bit of info here:
Subject: Cheap and Cheerful does it > How NatWest's IT meltdown developed
NatWest has admitted that it could not say exactly how much money should be
in individual accounts as the crisis caused by a failed software update last
week spiralled out of control for days.
The bank was quick to deny claims by the Unite union that the "offshoring"
of IT jobs to locations in India had led to the the problems which appeared
on Tuesday night and which paralysed its systems through to Friday, and
which have not yet been fixed.
However a number of programmers and experts who have worked on or with
NatWest systems told the Guardian that they could not imagine the problem
happening in the period before the redundancies of experienced staff since
2010.
"[NatWest owner] Royal Bank of Scotland has 40 years' experience running
these systems and banks as a rule don't drop the ball like this," one said.
"Somebody somewhere made a decision that has led to this."
The Guardian's investigations suggest that NatWest's problems began on
Tuesday night when it updated a key piece of software ? CA-7, which controls
the batch processing systems that deal with retail banking transactions ?
ahead of the regular nightly run.
RBS/NatWest has not said what went wrong, though one programmer who has
worked on RBS/NatWest's systems told the Guardian: "CA-7 is a very common
and reliable product used to automate large sequences of batch mainframe
work [which are usually referred to as 'jobs']. It will start jobs, wait for
them to run, then start other jobs dependent on the first ones completing,
and so on. RBS processes accounts overnight via thousands of jobs."
The jobs take transactions from various places, such as ATM withdrawals,
bank-to-bank salary payments, and so on, and finish by updating the master
copy of the account ? in a system known as Caustic ? with the definitive
balance.
"It seems whoever made the update to CA-7 managed to delete or corrupt the
files which hold the schedule for the overnight jobs, so they did not run,
or ran incorrectly," the programmer told the Guardian. "They have backed out
from this change, but now are trying to play catch-up, and have been doing
so for a few days."
The batch processing system, which reconciles the movement of money in and
out of more than 10m NatWest and Ulster Bank accounts, did not run correctly
for three nights ? meaning that millions of transactions were not processed
until it did begin running correctly on Friday. Even when it had been fixed,
the batches of transactions have had to be re-run in order, beginning with
Tuesday, so that nobody's account goes wrongly into overdraft.
A NatWest spokesperson, asked whether it knew how much money people had at
any time, said: "All the money is safe in the bank. It's being applied to
people's accounts. We can show people statements on screens if they come
into branches." The bank is offering special opening hours, extending to
6pm, this week. NatWest also ran extra batches to catch up with the
transaction backlog over the weekend.
NatWest, like many other banks, uses the CA-7 software and attendant files
to fit its own custom needs. The problem only surfaced once the batch run
was underway in the early hours of Wednesday. RBS appears to have advertised
for specialists in CA-7 in February in India
[http://hyderabad.quikr.com/Batch-Admin-with-CA7-tool-W0QQAdIdZ81783921"
title="] ? to which a number of its IT jobs were moved after 2010. "Looking
for candidates having 4-7 years of experience in Batch Administration using
CA7 tool," the advert read. "Urgent Requirement by RBS."
The Unite union has criticised RBS management
[http://www.guardian.co.uk/business/2012/jun/25/unite-attacks-rbs-slashing-j
obs" title="] for cutting jobs in the UK and shifting a number of them
offshore. Since 2010, hundreds of IT jobs have been cut from RBS's Edinburgh
headquarters and shifted abroad. RBS/NatWest has denied that it made any
difference.
But some observers strongly disagree. "This was not inevitable ? you can
always avoid problems like this if you test sufficiently," said David
Silverstone, delivery and solutions manager for NMQA, which provides
automated testing software to a number of banks, though not RBS/NatWest.
"But unless you keep an army of people who know exactly how the system
works, there may be problems maintaining it."
One programmer who worked on the RBS/NatWest systems during the takeover in
2001-02 said that the latest problems suggested a paucity of staff on the
spot with experience of what to do. "The people in India will have done
their darndest to do a good job, but without the knowledge of the overall
system that you get from years of experience on the ground, it's easier to
see how you get a big operational failure."
Banks have for decades used huge mainframe systems to process payments such
as cheques and to update customers' accounts; the transactions for each day
are collected and are then run in a single gigantic batch overnight, so that
accounts have been credited and debited with the correct amounts by the
morning. That is why internet banking transactions are not processed if you
carry them out after certain times: the banks' systems simply don't add them
into the queue for that night's batch.
Sources familiar with NatWest's systems, and who have also spoken to staff
there, explained that the problems with the update surfaced during the batch
run. NatWest confirmed on Monday that the problem first surfaced on Tuesday,
and that "we confirmed the fix on Friday".
The problems with the upgrade were spotted during the overnight run ahead of
Wednesday morning. "We have guardian systems which spot when things go
wrong," a NatWest spokesperson said.
But by Friday, when the fix was implemented, three sets of batch runs had
failed. If a batch fails badly ? as here ? then all of the transactions,
including the payments in and out of accounts, are "rolled back" to the
starting point, as if it had never run. The set of transactions from
Wednesday was then added to the pending list on Wednesday, and attempted to
run in the early hours of Thursday; that too failed. By the time the fix had
been done, there were three days' worth of unimplemented transactions queued
up.
Richard Price, a Norwich-based systems developer who has worked on banking
systems that linked into NatWest's, explains: "Banking systems are like a
huge game of Jenga [the tower game played with interlaced blocks of wood].
Two unrelated transactions might not look related now, but 500,000
transactions from now they might have a huge relation. So everything needs
to be processed in order." Thus Tuesday's batch must run before Wednesday's
or Thursday's to avoid, for example, penalising someone who has a large sum
of money leave their account on Thursday that might put them in debt but
which would be covered by money arriving on Wednesday.
Price said that any software update would first have been subjected to
quality assurance and user acceptance testing before being implemented.
CA-7 is familiar to many in the banking industry: it was originally released
in 1980 by Uccel ? which was then taken over by Computer Associates, which
provides key software for scores of banks. Computer Associates told the
Guardian: "RBS is a valued CA Technologies customer, we are offering all
assistance possible to help them resolve their technical issues which are
highly unique to their environment. We do not comment on customer
confidential issues." However it declined to say whether CA-7 lay at the
heart of the problems.
It'll make no difference at all:
1) Other companies will think it won't happen to them because they used someone better or their T&Cs + SLAs were better
2) RBS would have known the risks and service compromises as do all the other companies that go down this route
3) The very people who will complain most vocally (shareholders and customers) will be at the route of the strategy to offshore; they wanted better dividends, more services, cheaper services - that has to be paid somehow and cutting headcount + operational expense is the preferred way of doing it with many companies
4) You need to consider how many companies have offshored vs home many headlines you've seen similar to the RBS issue. I work in IT so I'm not a fan of offshoring but I'm only too well aware that it can and does save money all the time the dice roll the right way - there are many, many companies taking that gamble and getting lucky - the longer they get lucky, the more money they save
1) Other companies will think it won't happen to them because they used someone better or their T&Cs + SLAs were better
2) RBS would have known the risks and service compromises as do all the other companies that go down this route
3) The very people who will complain most vocally (shareholders and customers) will be at the route of the strategy to offshore; they wanted better dividends, more services, cheaper services - that has to be paid somehow and cutting headcount + operational expense is the preferred way of doing it with many companies
4) You need to consider how many companies have offshored vs home many headlines you've seen similar to the RBS issue. I work in IT so I'm not a fan of offshoring but I'm only too well aware that it can and does save money all the time the dice roll the right way - there are many, many companies taking that gamble and getting lucky - the longer they get lucky, the more money they save
GnuBee said:
It'll make no difference at all:
1) Other companies will think it won't happen to them because they used someone better or their T&Cs + SLAs were better
2) RBS would have known the risks and service compromises as do all the other companies that go down this route
3) The very people who will complain most vocally (shareholders and customers) will be at the route of the strategy to offshore; they wanted better dividends, more services, cheaper services - that has to be paid somehow and cutting headcount + operational expense is the preferred way of doing it with many companies
4) You need to consider how many companies have offshored vs home many headlines you've seen similar to the RBS issue. I work in IT so I'm not a fan of offshoring but I'm only too well aware that it can and does save money all the time the dice roll the right way - there are many, many companies taking that gamble and getting lucky - the longer they get lucky, the more money they save
I agree, I am guessing that Natwest will have pretty much wiped out any saving they made and the some!1) Other companies will think it won't happen to them because they used someone better or their T&Cs + SLAs were better
2) RBS would have known the risks and service compromises as do all the other companies that go down this route
3) The very people who will complain most vocally (shareholders and customers) will be at the route of the strategy to offshore; they wanted better dividends, more services, cheaper services - that has to be paid somehow and cutting headcount + operational expense is the preferred way of doing it with many companies
4) You need to consider how many companies have offshored vs home many headlines you've seen similar to the RBS issue. I work in IT so I'm not a fan of offshoring but I'm only too well aware that it can and does save money all the time the dice roll the right way - there are many, many companies taking that gamble and getting lucky - the longer they get lucky, the more money they save
I think I'm more shocked that they managed to get this through a change management process without having a robust (and quickly implemented) plan to reverse the upgrade or restore the config. I guess it's the kind of thing you learn a valuable lesson from but when I've worked on the systems in banking environments we've usually gone through the process a few times in test environments before going live.
I would imagine the chain of blame is quite substantial on this one but I almost feel sorry for the guy right at the top who seems to be getting all the grief aimed at him - yes he's ultimately responsible I suppose but he wasn't there mashing a keyboard.
Anyway if it helps bring some jobs back to the UK then it's a good thing, given the sort of day rates I see for offshore vs UK resource I can't see it happening though (usually 1/6th cheaper even if it does take 3 of them and 27 supporting people in the UK to push anything through using a detailed script that has been written by someone who could have just done the work in less time).
Out of interest does anyone know which companies are are involved in the IT side of RBS? I presume they have internal resource backed up by an assortment of service providers?
I would imagine the chain of blame is quite substantial on this one but I almost feel sorry for the guy right at the top who seems to be getting all the grief aimed at him - yes he's ultimately responsible I suppose but he wasn't there mashing a keyboard.
Anyway if it helps bring some jobs back to the UK then it's a good thing, given the sort of day rates I see for offshore vs UK resource I can't see it happening though (usually 1/6th cheaper even if it does take 3 of them and 27 supporting people in the UK to push anything through using a detailed script that has been written by someone who could have just done the work in less time).
Out of interest does anyone know which companies are are involved in the IT side of RBS? I presume they have internal resource backed up by an assortment of service providers?
thetapeworm said:
I think I'm more shocked that they managed to get this through a change management process without having a robust (and quickly implemented) plan to reverse the upgrade or restore the config. I guess it's the kind of thing you learn a valuable lesson from but when I've worked on the systems in banking environments we've usually gone through the process a few times in test environments before going live.
It sounds like the back out plan worked but the volume of data awaiting processing was way to large to quickly recover. It's like they didn't correctly identify the problem for a few days. And that's become the main issue rather then backing out the change.I doubt they included a "what if we don't realise it's buggered for 2 or 3 days and have to process all that data" test. I don't know how common it would be to include that kind of test either?
thetapeworm said:
...
Anyway if it helps bring some jobs back to the UK then it's a good thing, given the sort of day rates I see for offshore vs UK resource I can't see it happening though (usually 1/6th cheaper even if it does take 3 of them and 27 supporting people in the UK to push anything through using a detailed script that has been written by someone who could have just done the work in less time)
...
Several internet sleuths have dug up the Hyderabad RBS job ads which state a pay of ~11k pa. So at a guess that's only 1/3rd of what a UK support analyst might earn, not 1/6th. Then you have to factor in wage inflation running at 11%pa (versus at best 2-3% in the UK) and additional costs due to the time difference/inexperienced staff/language barrier/cultural differences. Finally, when you add the expected £100m+ loss due to this debacle I'd estimate the whole offshoring operation is running at a large loss.Anyway if it helps bring some jobs back to the UK then it's a good thing, given the sort of day rates I see for offshore vs UK resource I can't see it happening though (usually 1/6th cheaper even if it does take 3 of them and 27 supporting people in the UK to push anything through using a detailed script that has been written by someone who could have just done the work in less time)
...
Just read a few articles (on some sites that I am taking with a very large pinch of salt) saying that it's rumoured that there is more to this than meets they eye.
At the same time as this, BT's national broadband network went down for 4 hours, and MI5 have been banging on about th danger of cyber attacks all week....
At the same time as this, BT's national broadband network went down for 4 hours, and MI5 have been banging on about th danger of cyber attacks all week....
I would imagine (since I provide backup support worldwide for major companies) that once spotted, the issue was addressed fairly quickly - but with systems this big, that simply takes time, then it has to catch up.
We lost part of the billing system at my previous firm, the hardware fault that caused it took x hours to find (in fairness, it was unusual), then the new bits had to arrive, then fitted. Then it was found it had corrupted the DB, so that had to be restored, it all added up to 39 hours - and a massive loss, which due to the nature of the business was unrecoverable.
This was quite a small part of a big overall system - now scale that up to banking size, and you'll have days of recovery.
"In which case, this could actually be GOOD for the UK IT industry..."
I don't know - I suspect a lot of firms will think it can never happen to them.
My current firm have (some of) our own IT support in India. When ever I get a call from them on our own backup systems, because they cannot fix it - well, I don't mind that, but I'm rarely impressed by their ability - they have little grasp of what I call the basics, and I have no idea who designed the last backup system I looked at, but I think all my office very quickly got my view of it when I put the phone down. It's laughable, we can't even get 'real' experts running our own systems. Many big firms are at it it's no different anywhere.
In fairness, part of the issue is the culture - they change jobs very quickly/ often, so never gain experience.
India is getting more expensive, so things will move somewhere cheaper, and the issues will repeat.
It's funny, just this afternoon, I could hear my colleague on the phone being requested for 'webex' - having sent the exact 'copy/ paste' instructions to get some logs sent in. I wandered over to see who the customer / country was (oh, no surprise there then ...) - turns out the customer, who was running a backup environment for a major company, was unable to even create a directory.
Unfortunately, this is a daily event - it is frequent that I or my colleagues answer the phone and the first word we hear is 'webex' - the companies involved have staff with so little skill, that then expect us to run their environments for them, which unfortunately, is not what we do. These companies include major banks and credit card companies.
It is utterly shocking what we see - perhaps they have procedures/ DR systems in place etc... but their biggest risk, in many cases, are their staff.
Martin
We lost part of the billing system at my previous firm, the hardware fault that caused it took x hours to find (in fairness, it was unusual), then the new bits had to arrive, then fitted. Then it was found it had corrupted the DB, so that had to be restored, it all added up to 39 hours - and a massive loss, which due to the nature of the business was unrecoverable.
This was quite a small part of a big overall system - now scale that up to banking size, and you'll have days of recovery.
"In which case, this could actually be GOOD for the UK IT industry..."
I don't know - I suspect a lot of firms will think it can never happen to them.
My current firm have (some of) our own IT support in India. When ever I get a call from them on our own backup systems, because they cannot fix it - well, I don't mind that, but I'm rarely impressed by their ability - they have little grasp of what I call the basics, and I have no idea who designed the last backup system I looked at, but I think all my office very quickly got my view of it when I put the phone down. It's laughable, we can't even get 'real' experts running our own systems. Many big firms are at it it's no different anywhere.
In fairness, part of the issue is the culture - they change jobs very quickly/ often, so never gain experience.
India is getting more expensive, so things will move somewhere cheaper, and the issues will repeat.
It's funny, just this afternoon, I could hear my colleague on the phone being requested for 'webex' - having sent the exact 'copy/ paste' instructions to get some logs sent in. I wandered over to see who the customer / country was (oh, no surprise there then ...) - turns out the customer, who was running a backup environment for a major company, was unable to even create a directory.
Unfortunately, this is a daily event - it is frequent that I or my colleagues answer the phone and the first word we hear is 'webex' - the companies involved have staff with so little skill, that then expect us to run their environments for them, which unfortunately, is not what we do. These companies include major banks and credit card companies.
It is utterly shocking what we see - perhaps they have procedures/ DR systems in place etc... but their biggest risk, in many cases, are their staff.
Martin
Edited by mph999 on Tuesday 26th June 23:45
Olivera said:
Several internet sleuths have dug up the Hyderabad RBS job ads which state a pay of ~11k pa. So at a guess that's only 1/3rd of what a UK support analyst might earn, not 1/6th. Then you have to factor in wage inflation running at 11%pa (versus at best 2-3% in the UK) and additional costs due to the time difference/inexperienced staff/language barrier/cultural differences. Finally, when you add the expected £100m+ loss due to this debacle I'd estimate the whole offshoring operation is running at a large loss.
What you're not factoring in is that the wage paid to the local is the cost to the offshore partner company.They will be charging him to RBS at multiples of that rate. I think the basic offshore resources we use are charged out to us at £150 a day, that's about £33k. Also add in offshore partner management charges and it's going to be somewhere in the region of £40K that RBS will have paid for that indian resource.
Bring them over to the UK and the cost rate jumps to £375 a day. You can get decent UK contractor resources for that money.
Having the IT staff in the UK wouldnt necessarily make any difference, the same could happen over here its far from unheard of for this kind of thing to happen over here, people make stupid mistakes, the main difference was they didnt have a decent Disaster recovery system in place to pick up the pieces and sort things out meaning the problems have run on so long. You cant stop a system going down but you can minimise the recovery time
Many companies especially in the financial sector have reduced DR spending as its like insurance if you dont use it its money wasted but if you need it .........
Many companies especially in the financial sector have reduced DR spending as its like insurance if you dont use it its money wasted but if you need it .........
liner33 said:
Having the IT staff in the UK wouldnt necessarily make any difference, the same could happen over here its far from unheard of for this kind of thing to happen over here, people make stupid mistakes, the main difference was they didnt have a decent Disaster recovery system in place to pick up the pieces and sort things out meaning the problems have run on so long. You cant stop a system going down but you can minimise the recovery time
Many companies especially in the financial sector have reduced DR spending as its like insurance if you dont use it its money wasted but if you need it .........
I don't think its necessarily the lack of DR facilities, most systems of this size will have a fully tested DR capability to meet regulatory approval.Many companies especially in the financial sector have reduced DR spending as its like insurance if you dont use it its money wasted but if you need it .........
The issue is that a software corruption is not something that most DR systems are designed to protect against, e.g. if you are replicating your data in real time / near real time, any corruption is present in the DR system before you can switch over. It looks like it took them a day or so to detect the problem, and so the DR system is already infected.
DR is for physical disasters such as data centre failures.
Regarding offshoring, I think there are two issues -
Getting rid of experienced people who know exactly how a very complex process should work without looking for documentation (or know exactly where to look in any documentation)
Detachment of the offshore resources from the system / task - onshore resources are fully aware of any critical systems, their business impact etc. Offshore staff generally see it as working on just another system without necessarily appreciating the business criticality. Also as serveral people have observered, there is very little continuity in the staff assigned to accounts, you find that the people who were onboarded with the KT etc may well be moved to the next "hot" account after a relatively short time and so KT becomes a game of Chinese Whispers.
Edited by Burrow01 on Wednesday 27th June 12:23
How is off-shoring any different then to any off-site IT support??
My point is that fact the support was/is in India is no different and carries no more risk than if it was in Birmingham.
The insinuation that the personnel are somehow less qualified doesnt carry any weight imo. There is no shortage of IT numpties over here , anyone who has spent any time in the industry and can give a few horror stories
My point is that fact the support was/is in India is no different and carries no more risk than if it was in Birmingham.
The insinuation that the personnel are somehow less qualified doesnt carry any weight imo. There is no shortage of IT numpties over here , anyone who has spent any time in the industry and can give a few horror stories
liner33 said:
The insinuation that the personnel are somehow less qualified doesnt carry any weight imo.
They dumped experienced UK staff and replaced them with paper-qualified Indian staff. They thought that experience and accumulated expertise was worth nothing. Now they have a better estimate.If they had dumped all of their experienced staff and replaced them with inexperienced Brummies that would also have dropped them in the s
t, but Brummies aren't as cheap as Indians so that isn't what happened.liner33 said:
How is off-shoring any different then to any off-site IT support??
My point is that fact the support was/is in India is no different and carries no more risk than if it was in Birmingham.
The insinuation that the personnel are somehow less qualified doesnt carry any weight imo. There is no shortage of IT numpties over here , anyone who has spent any time in the industry and can give a few horror stories
We outsourced some work to India once, once we had paid them for the work, tried to fix their code and then dumped the lot and started again it came in at twice the budget it should have, we now don't outsourceMy point is that fact the support was/is in India is no different and carries no more risk than if it was in Birmingham.
The insinuation that the personnel are somehow less qualified doesnt carry any weight imo. There is no shortage of IT numpties over here , anyone who has spent any time in the industry and can give a few horror stories
Also we recently advertised for a testing role, 4 candidates from India great CV's but weren't expecting a test, they got the lowest marks anyone has ever achieved
otolith said:
They dumped experienced UK staff and replaced them with paper-qualified Indian staff. They thought that experience and accumulated expertise was worth nothing. Now they have a better estimate.
If they had dumped all of their experienced staff and replaced them with inexperienced Brummies that would also have dropped them in the st, but Brummies aren't as cheap as Indians so that isn't what happened.
Experience doesnt equal competence. If they had dumped all of their experienced staff and replaced them with inexperienced Brummies that would also have dropped them in the s
t, but Brummies aren't as cheap as Indians so that isn't what happened.There is a skills shortage in some sections of IT at the moment in the UK
liner33 said:
Experience doesnt equal competence.
No, it doesn't. But lack of it may equal incompetence, which appears to have been the problem in this case.liner33 said:
There is a skills shortage in some sections of IT at the moment in the UK
An unwillingness to pay the going UK rate does not a shortage make - but then with the offshoring mentality, your shortage is a self-fulfilling prophecy. I certainly wouldn't advise a bright young person to study for a career in IT now.Gassing Station | Jobs & Employment Matters | Top of Page | What's New | My Stuff