Natwest/RBS glitch
Discussion
hornetrider said:
roachcoach said:
Yup, though I think more likely the issue sits firmly on the desk of the head of tech (or whatever flavour of name they have over there).
CEO has a responsibility to be sure, but there's got to be a degree of trust in the staff you delegate to, otherwise he'd be manning the phone in the call centres
Problem is Mr IT Dept Manager will have a budget and targets to hit regarding CapEx and OpEx. They come from oop above.CEO has a responsibility to be sure, but there's got to be a degree of trust in the staff you delegate to, otherwise he'd be manning the phone in the call centres
Support is easy to get rid of and to live without, until you need them. Then you're properly f
ked 
hornetrider said:
Problem is Mr IT Dept Manager will have a budget and targets to hit regarding CapEx and OpEx. They come from oop above.
Indeed.And if Mr CEO is being driven by, say, shareholders for a particular level of return, cutting operating costs when times are hard is a good way of achieving that end. Or if he's being driven by other factors that might lead to the same behaviour.
There's always a reason/driver for these things happening. And there's no shortage of shiny presentation packs that the big consultancies can wheel out with their MBA laden staff to prove how shifting a few hundred/thousand roles East will be the answer you're looking for. It's just a shame people don't ask deeper questions of them or do their due diligence better.
Oh wait, what was one of the reasons the organisation in question got into bother...
(btw, off-shoring programmes are often done very quickly. A trend may have been started pre-2008, but what's happening as of right now will have been later than that).
Murph7355 said:
Along with all the other things....why were they trying to do this in the middle of an incident?
Because if the scheduler had been deleted by "inexperienced staff" and a backup screws up a bit more stuff your pretty much lost. Though you have to ask yourself how did it feel for the "inexperinced employee" to bring a bank to its knees in one foul swoop. Bet they dropped their arse there and then.
Murph7355 said:
The idea is that you have all of this stuff properly documented and constantly managed and tested before an incident as it is one of the tools that helps you get out of one.
I suspect the lack of solid system documentation cannot be blamed on any outsourcing/offshoring agreements. Equally, I would guess it's a contributory factor in making the system difficult to support by people who are most adept at rote type scenarios (and here I very much agree that Indian outsourced/offshored outfits tend to fall into this camp).
Been contracting for years now and not once have I reached for documentation during an incident. I suspect the lack of solid system documentation cannot be blamed on any outsourcing/offshoring agreements. Equally, I would guess it's a contributory factor in making the system difficult to support by people who are most adept at rote type scenarios (and here I very much agree that Indian outsourced/offshored outfits tend to fall into this camp).
Murph7355 said:
The other amusing thing with offshoring is mapping the costs. It started out in Asia and has moved West over time, with the most recent trends to plonk people in Eastern Europe. Map the salaries of these areas from when offshoring began to now and you'll see some pretty obvious trends, to the point where the next big "offshoring" move will likely be placing people in Manchester or Newcastle 
So over the last 10-15yrs we'll have gone full circle. Spent loads, created significant risk and we'll end up right back where we started.
The technology circle is a lovely beast, and keeps contractors in plenty of work. Often sorting the fq ups that the big boys (Accenture, E&Y, PWC...) leave behind.
And long may it continue! So over the last 10-15yrs we'll have gone full circle. Spent loads, created significant risk and we'll end up right back where we started.The technology circle is a lovely beast, and keeps contractors in plenty of work. Often sorting the fq ups that the big boys (Accenture, E&Y, PWC...) leave behind.
Edited by Sexual Chocolate on Tuesday 26th June 14:07
Sexual Chocolate said:
Murph7355 said:
Along with all the other things....why were they trying to do this in the middle of an incident?
Because if the scheduler had been deleted by "inexperienced staff" and a backup screws up a bit more stuff your pretty much lost. Though you have to ask yourself how did it feel for the "inexperinced employee" to bring a bank to its knees in one foul swoop. Bet they dropped their arse there and then.
Murph7355 said:
The idea is that you have all of this stuff properly documented and constantly managed and tested before an incident as it is one of the tools that helps you get out of one.
I suspect the lack of solid system documentation cannot be blamed on any outsourcing/offshoring agreements. Equally, I would guess it's a contributory factor in making the system difficult to support by people who are most adept at rote type scenarios (and here I very much agree that Indian outsourced/offshored outfits tend to fall into this camp).
[/qoute]
Been contracting for years now and not once have I reached for documentation during an incident. I suspect the lack of solid system documentation cannot be blamed on any outsourcing/offshoring agreements. Equally, I would guess it's a contributory factor in making the system difficult to support by people who are most adept at rote type scenarios (and here I very much agree that Indian outsourced/offshored outfits tend to fall into this camp).
[/qoute]
Murph7355 said:
The other amusing thing with offshoring is mapping the costs. It started out in Asia and has moved West over time, with the most recent trends to plonk people in Eastern Europe. Map the salaries of these areas from when offshoring began to now and you'll see some pretty obvious trends, to the point where the next big "offshoring" move will likely be placing people in Manchester or Newcastle So over the last 10-15yrs we'll have gone full circle. Spent loads, created significant risk and we'll end up right back where we started.
The technology circle is a lovely beast, and keeps contractors in plenty of work. Often sorting the fq ups that the big boys (Accenture, E&Y, PWC...) leave behind.
And long my it continue! So over the last 10-15yrs we'll have gone full circle. Spent loads, created significant risk and we'll end up right back where we started.The technology circle is a lovely beast, and keeps contractors in plenty of work. Often sorting the fq ups that the big boys (Accenture, E&Y, PWC...) leave behind.
Edited by Murph7355 on Tuesday 26th June 14:07
Murph7355 said:
There's something amusing and ironic in the above
I was suffering a technical glitch. For a second there I outsourced that part of my brain that deals in written communication to India however the system remains UK based.Edited by Murph7355 on Tuesday 26th June 14:07
Murph7355 said:
Along with all the other things....why were they trying to do this in the middle of an incident?
The idea is that you have all of this stuff properly documented and constantly managed and tested before an incident as it is one of the tools that helps you get out of one.
I suspect the lack of solid system documentation cannot be blamed on any outsourcing/offshoring agreements.
Probably because some managers kept presenting them with JFDI deadlines and didn't estimate enough project time for documentation, especially if they went 'agile'. The idea is that you have all of this stuff properly documented and constantly managed and tested before an incident as it is one of the tools that helps you get out of one.
I suspect the lack of solid system documentation cannot be blamed on any outsourcing/offshoring agreements.
As for offshored documentation, have you ever read what they write?
Murph7355 said:
Sexual Chocolate said:
Murph7355 said:
Along with all the other things....why were they trying to do this in the middle of an incident?
Because if the scheduler had been deleted by "inexperienced staff" and a backup screws up a bit more stuff your pretty much lost. Though you have to ask yourself how did it feel for the "inexperinced employee" to bring a bank to its knees in one foul swoop. Bet they dropped their arse there and then.
Murph7355 said:
The idea is that you have all of this stuff properly documented and constantly managed and tested before an incident as it is one of the tools that helps you get out of one.
I suspect the lack of solid system documentation cannot be blamed on any outsourcing/offshoring agreements. Equally, I would guess it's a contributory factor in making the system difficult to support by people who are most adept at rote type scenarios (and here I very much agree that Indian outsourced/offshored outfits tend to fall into this camp).
[/qoute]
Been contracting for years now and not once have I reached for documentation during an incident. I suspect the lack of solid system documentation cannot be blamed on any outsourcing/offshoring agreements. Equally, I would guess it's a contributory factor in making the system difficult to support by people who are most adept at rote type scenarios (and here I very much agree that Indian outsourced/offshored outfits tend to fall into this camp).
[/qoute]
Murph7355 said:
The other amusing thing with offshoring is mapping the costs. It started out in Asia and has moved West over time, with the most recent trends to plonk people in Eastern Europe. Map the salaries of these areas from when offshoring began to now and you'll see some pretty obvious trends, to the point where the next big "offshoring" move will likely be placing people in Manchester or Newcastle So over the last 10-15yrs we'll have gone full circle. Spent loads, created significant risk and we'll end up right back where we started.
The technology circle is a lovely beast, and keeps contractors in plenty of work. Often sorting the fq ups that the big boys (Accenture, E&Y, PWC...) leave behind.
And long my it continue! So over the last 10-15yrs we'll have gone full circle. Spent loads, created significant risk and we'll end up right back where we started.The technology circle is a lovely beast, and keeps contractors in plenty of work. Often sorting the fq ups that the big boys (Accenture, E&Y, PWC...) leave behind.
Edited by Murph7355 on Tuesday 26th June 14:07
Not sure if it was intended but I can imagine a large and scruffy seagull dive bombing and depositing large dollops of birdie poop on Hester's desk.
Sexual Chocolate said:
I walked past the recovery room and they where trying to map it all out. It was crazy with jobs listed here and there with arrows going all over the place.
Now that is scary, we run maestro here as our scheduling tool, but the schedules/jobs themselves are quite simple so could be run manually in the event of a major FU. The last company i worked at had (large gas supplier) had massive schedules, with large bits of kit just to run maestro. However, they did have a team of abotu 4 UK people supporting it, with proper redundacy, dev enviroment and a complete offsite DR setup. Do you problem/incident managers? How are they coping?
Murph7355 said:
hornetrider said:
Problem is Mr IT Dept Manager will have a budget and targets to hit regarding CapEx and OpEx. They come from oop above.
Indeed.And if Mr CEO is being driven by, say, shareholders for a particular level of return, cutting operating costs when times are hard is a good way of achieving that end. Or if he's being driven by other factors that might lead to the same behaviour.
There's always a reason/driver for these things happening. And there's no shortage of shiny presentation packs that the big consultancies can wheel out with their MBA laden staff to prove how shifting a few hundred/thousand roles East will be the answer you're looking for. It's just a shame people don't ask deeper questions of them or do their due diligence better.
Oh wait, what was one of the reasons the organisation in question got into bother...
(btw, off-shoring programmes are often done very quickly. A trend may have been started pre-2008, but what's happening as of right now will have been later than that).
Everyone at the sharp end of things knows the weak links of their operation, and if they are diligent deal with the ones they can deal with and, if a suitable culture exists, point out the rest of the weak spots, within reason, in a constructive way one hopes.
The CEO may or may not also know about these weak spots, but of all these improvements the only cost saving that they can 100% guarantee is reducing the headcount, which has a short term cost in redundancy etc, but a long term saving. The problem that time and again happens is that the cost savings never materialise, and after a few months you get staff back that you have made redundant / retired early but now as contractors / consultants, and on higher costs. I can quote one case where a retiree is now back working 50% time for 125% of his previous salary. This is quite a moderate case too.
It's rife through both the private and public sectors how much of this goes on.
ExFiF said:
Thing is though, Mr CEO might be under pressure to improve results, there are a number of ways of doing this, increasing market share, increasing operational efficiency, improving yields, all generally aimed at reducing costs.
Everyone at the sharp end of things knows the weak links of their operation, and if they are diligent deal with the ones they can deal with and, if a suitable culture exists, point out the rest of the weak spots, within reason, in a constructive way one hopes.
The CEO may or may not also know about these weak spots, but of all these improvements the only cost saving that they can 100% guarantee is reducing the headcount, which has a short term cost in redundancy etc, but a long term saving. The problem that time and again happens is that the cost savings never materialise, and after a few months you get staff back that you have made redundant / retired early but now as contractors / consultants, and on higher costs. I can quote one case where a retiree is now back working 50% time for 125% of his previous salary. This is quite a moderate case too.
It's rife through both the private and public sectors how much of this goes on.
You can thank amazing process efficiency suits-with-no-fecking-clue like Accenture for that one Everyone at the sharp end of things knows the weak links of their operation, and if they are diligent deal with the ones they can deal with and, if a suitable culture exists, point out the rest of the weak spots, within reason, in a constructive way one hopes.
The CEO may or may not also know about these weak spots, but of all these improvements the only cost saving that they can 100% guarantee is reducing the headcount, which has a short term cost in redundancy etc, but a long term saving. The problem that time and again happens is that the cost savings never materialise, and after a few months you get staff back that you have made redundant / retired early but now as contractors / consultants, and on higher costs. I can quote one case where a retiree is now back working 50% time for 125% of his previous salary. This is quite a moderate case too.
It's rife through both the private and public sectors how much of this goes on.
roachcoach said:
Add to that DR is only any use if the error is caught outright and you have excellent roll-forward capabilities. If you had a problem that ran on a day or so, then transacted after it, then noticed it was borked you're in deep water.
You'd be replicating the the DR pretty aggressively I'd imagine (since its primary function is hardware failover, not crap code failover) so the issue may have propagated there before it was caught: See your point about experience, experienced folks would have known of all the downstream nuances, dependencies and critically, their impact. FNG following a wiki...unlikely.
There's a whole lot of things go on in places that are all but 'un-backoutable' once the business gets its paws on it.
The DR should be fit for recovering from a disaster. I would have planned for it to be ready for a failed code implementation event. Easy of course to say after the event. DR infrastructure is also coupled with Business processes that go hand in hand. Don't tell me RBS etc don't do that either.You'd be replicating the the DR pretty aggressively I'd imagine (since its primary function is hardware failover, not crap code failover) so the issue may have propagated there before it was caught: See your point about experience, experienced folks would have known of all the downstream nuances, dependencies and critically, their impact. FNG following a wiki...unlikely.
There's a whole lot of things go on in places that are all but 'un-backoutable' once the business gets its paws on it.
Well, I should have had my expenses paid last week in to my NaffWest account. It left my employer last Wednesday, and would normally hit the account on Friday.
It still hasn't arrived.
I had to go in to my local branch yesterday to get some money out, the tellers all said they "hadn't known anything like it before". They also said it would be Wednesday before it was all back to normal.
I've seen direct debits go out of the account, but my credit still hasn't hit. Its showing on the system though.
Amazingly, in the branch yesterday, the cashier had to use a calculator to work out what my balance should have been.
If I wasn't already intending to move after Santander taking my account in the near future, this has only served to want to hasten that switch.
It still hasn't arrived.
I had to go in to my local branch yesterday to get some money out, the tellers all said they "hadn't known anything like it before". They also said it would be Wednesday before it was all back to normal.
I've seen direct debits go out of the account, but my credit still hasn't hit. Its showing on the system though.
Amazingly, in the branch yesterday, the cashier had to use a calculator to work out what my balance should have been.
If I wasn't already intending to move after Santander taking my account in the near future, this has only served to want to hasten that switch.
From: http://www.theregister.co.uk/2012/06/26/rbs_natwes...
"Following our revelation yesterday that a bungled update to CA-7 batch processing software used by RBS lay behind the collapse, further details have emerged. According to a Register source who worked at RBS for several years, an inexperienced operative made a major error while performing the relatively routine task of backing out of an upgrade to the CA-7 tool. It is normal to find that a software update has caused a problem; IT staff expect to back out in such cases.
But in the process of backing out a major blunder was committed, according to our source. It was this error which made the task of restoring services so prolonged:
When they did the back-out, a major error was made. An inexperienced person cleared the whole queue ... they erased all the scheduling.
That created a large backlog as all the wiped information had to be re-inputted to the system and reprocessed. A complicated legacy mainframe system at RBS and a team inexperienced in its quirks made the problem harder to fix, our source adds.
CA Technologies - the makers of the CA-7 software at the heart of the snarl-up - are helping RBS to fix the disaster that has affected 16.9 million UK bank accounts
The CEO of RBS Group, Stephen Hester, has said that there is no evidence that the problem is connected to lack of investment in technology at RBS and the outsourcing of IT jobs to India. Asked directly by Sky news yesterday, he said:
Well I have no evidence of that. The IT centre - our main centre, we’re standing outside here in Edinburgh, [is] nothing to do with overseas. Our UK backbone has seen substantial investment.
We put our sources' revelations regarding poor levels of knowledge in the handling of CA-7 to RBS spokespeople - and reminded the company's representatives of the job ad showing that at least some of the firm's CA-7 handlers were recruited in India this year, as we reported yesterday. We offered the company an opportunity to confirm that the critical blunder was committed by a UK-based rather than an India-based operator.
However the bank's spokesmen refused to offer any further comment."
"Following our revelation yesterday that a bungled update to CA-7 batch processing software used by RBS lay behind the collapse, further details have emerged. According to a Register source who worked at RBS for several years, an inexperienced operative made a major error while performing the relatively routine task of backing out of an upgrade to the CA-7 tool. It is normal to find that a software update has caused a problem; IT staff expect to back out in such cases.
But in the process of backing out a major blunder was committed, according to our source. It was this error which made the task of restoring services so prolonged:
When they did the back-out, a major error was made. An inexperienced person cleared the whole queue ... they erased all the scheduling.
That created a large backlog as all the wiped information had to be re-inputted to the system and reprocessed. A complicated legacy mainframe system at RBS and a team inexperienced in its quirks made the problem harder to fix, our source adds.
CA Technologies - the makers of the CA-7 software at the heart of the snarl-up - are helping RBS to fix the disaster that has affected 16.9 million UK bank accounts
The CEO of RBS Group, Stephen Hester, has said that there is no evidence that the problem is connected to lack of investment in technology at RBS and the outsourcing of IT jobs to India. Asked directly by Sky news yesterday, he said:
Well I have no evidence of that. The IT centre - our main centre, we’re standing outside here in Edinburgh, [is] nothing to do with overseas. Our UK backbone has seen substantial investment.
We put our sources' revelations regarding poor levels of knowledge in the handling of CA-7 to RBS spokespeople - and reminded the company's representatives of the job ad showing that at least some of the firm's CA-7 handlers were recruited in India this year, as we reported yesterday. We offered the company an opportunity to confirm that the critical blunder was committed by a UK-based rather than an India-based operator.
However the bank's spokesmen refused to offer any further comment."
hornetrider said:
You can thank amazing process efficiency suits-with-no-fecking-clue like Accenture for that one
Looks in mirror, sees 4 days a month for almost bang on full time original monthly salary. Why not, if the stupid buggers will pay it?Private sector, so no taxpayer money involved, btw.
rich1231 said:
The DR should be fit for recovering from a disaster. I would have planned for it to be ready for a failed code implementation event. Easy of course to say after the event. DR infrastructure is also coupled with Business processes that go hand in hand. Don't tell me RBS etc don't do that either.
Yes, but if the failure was caught late, you may be in a bit of bother depending on your retention policy/point in time recovery capability.I know the likes of Oracle can do brilliant point in time recovery, but that's one of its massive selling points so I imagine that capability is quite rare plus this sounded like a far more fundamental level problem than a given database or set thereof having a 'moment'.
Eric Mc said:
I like the way theyw1 skirt around the anser "I have no evidence for this" means absolutely nothing.
All the Hester is saying is "I don't want to admit my cost-cutting and offshoring plans for IT and Operations were a mistake. By the time everyone on the Internet seems to know more about what happened than the CEO I'll have found someone else to blame for not keeping me informed etc. Meanwhile RBS will use the BBC as it's spokesperson for not investigating the cause."Gassing Station | News, Politics & Economics | Top of Page | What's New | My Stuff