Halifax IT systems currently down - HOW????
Discussion
Halifax online banking and call centre systems have been down all day, apparently they've had a 'power cut'.
Speaking as an IT person HOW is this possible???
Surely, given they're one of the biggest banks in the UK, there would be enough resilience that they would fail over to another datacentre within minutes at most???
I used to severely get my ass kicked by the CEO of the company i used to work for if the systems were down for minutes, let alone for the most of a day?
Speaking as an IT person HOW is this possible???
Surely, given they're one of the biggest banks in the UK, there would be enough resilience that they would fail over to another datacentre within minutes at most???
I used to severely get my ass kicked by the CEO of the company i used to work for if the systems were down for minutes, let alone for the most of a day?
Mainly because they have cut back on IT drastically since the bust, all projects cancelled and half the staff paid off. The chickens are now coming home to roost. You will find a few banks in this position soon, at this moment in time all hell is breaking loose on the executive floors and every body and there Aunty Bessy are being called in.
They will now get absolutely hammered by regulatory for the systems going down and it will cost them millions, I think it works out at something like 10 million every hour the systems are down (depending on which ones of course). If trading is effected it is a monumental disaster.
Cue inquiry and large increase in IT spend closely followed by the other banks.
They will now get absolutely hammered by regulatory for the systems going down and it will cost them millions, I think it works out at something like 10 million every hour the systems are down (depending on which ones of course). If trading is effected it is a monumental disaster.
Cue inquiry and large increase in IT spend closely followed by the other banks.
elster said:
More than likely is that the maintenance staff haven't fuelled up the backup gennies.
This is the state I find most UPS systems as, no matter how large financial consequences they could create.
???? UPS, out of fuel, a UPS is powered by batteries. Think you will find this is smoke and mirrors statement from the Bank, the Data Centre will have two completely independent phases for power, each Cab will have a UPS that will provide Approx 15-30 Minutes of power to the cab. The backup Generator is always fueled.This is the state I find most UPS systems as, no matter how large financial consequences they could create.
Sounds to me more like the Core switch has went down and the backup switch has failed, this is not an easy fix if this has happened. It should also have failed over to there DR site as soon as this happened or power was lost, sounds again like this has failed to happen as the fail over would be Core switch dependent. I can bet that they are trying to bring the DR site up right now as this will be faster than trying to find out what has caused the network failure.
It is standard policy to blame the grid if things go down, if you say your IT procedures failed you are left wide open.
chimera40 said:
elster said:
More than likely is that the maintenance staff haven't fuelled up the backup gennies.
This is the state I find most UPS systems as, no matter how large financial consequences they could create.
???? UPS, out of fuel, a UPS is powered by batteries. Think you will find this is smoke and mirrors statement from the Bank, the Data Centre will have two completely independent phases for power, each Cab will have a UPS that will provide Approx 15-30 Minutes of power to the cab. The backup Generator is always fueled.This is the state I find most UPS systems as, no matter how large financial consequences they could create.
Sounds to me more like the Core switch has went down and the backup switch has failed, this is not an easy fix if this has happened. It should also have failed over to there DR site as soon as this happened or power was lost, sounds again like this has failed to happen as the fail over would be Core switch dependent. I can bet that they are trying to bring the DR site up right now as this will be faster than trying to find out what has caused the network failure.
It is standard policy to blame the grid if things go down, if you say your IT procedures failed you are left wide open.
Not a little piddly one, but the man size ones. They are diesel powered.
I have worked in a datacentre that had just been constructed, and they had a powercut a month in.
The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
TheCarpetCleaner said:
I have worked in a datacentre that had just been constructed, and they had a powercut a month in.
The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Hope they didn't, if the data centre hadn't tested them.....The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
eldar said:
TheCarpetCleaner said:
I have worked in a datacentre that had just been constructed, and they had a powercut a month in.
The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Hope they didn't, if the data centre hadn't tested them.....The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
elster said:
eldar said:
TheCarpetCleaner said:
I have worked in a datacentre that had just been constructed, and they had a powercut a month in.
The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Hope they didn't, if the data centre hadn't tested them.....The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Obviously some numbnut didnt do their contracted job, therefore missing the fact it was fuelled and in full operational status, alongside a plum in the monitoring centre not doing their job either - or a comms failure between the two. did not care as was not my problem at the time as I worked for a different company who handled the servers and robotic units that failed once this state of the art (?!) genny went down! My problem was reconfiguring stuff afterwards.
The agency that I was working on at the time did like their outsourcing... This was the result it would seem.
TheCarpetCleaner said:
elster said:
eldar said:
TheCarpetCleaner said:
I have worked in a datacentre that had just been constructed, and they had a powercut a month in.
The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Hope they didn't, if the data centre hadn't tested them.....The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Obviously some numbnut didnt do their contracted job, therefore missing the fact it was fuelled and in full operational status, alongside a plum in the monitoring centre not doing their job either - or a comms failure between the two. did not care as was not my problem at the time as I worked for a different company who handled the servers and robotic units that failed once this state of the art (?!) genny went down! My problem was reconfiguring stuff afterwards.
The agency that I was working on at the time did like their outsourcing... This was the result it would seem.
elster said:
TheCarpetCleaner said:
elster said:
eldar said:
TheCarpetCleaner said:
I have worked in a datacentre that had just been constructed, and they had a powercut a month in.
The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Hope they didn't, if the data centre hadn't tested them.....The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Obviously some numbnut didnt do their contracted job, therefore missing the fact it was fuelled and in full operational status, alongside a plum in the monitoring centre not doing their job either - or a comms failure between the two. did not care as was not my problem at the time as I worked for a different company who handled the servers and robotic units that failed once this state of the art (?!) genny went down! My problem was reconfiguring stuff afterwards.
The agency that I was working on at the time did like their outsourcing... This was the result it would seem.
I would have loved to be a fly on the wall in the meeting room where the person responsible was getting b
king 
elster said:
TheCarpetCleaner said:
elster said:
eldar said:
TheCarpetCleaner said:
I have worked in a datacentre that had just been constructed, and they had a powercut a month in.
The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Hope they didn't, if the data centre hadn't tested them.....The brand new generators did not kick in, because they were out of fuel. The contractors that were brought in to set up this thing (about the size of two large portacabins), failed to fuel it.
I would not have believed this, however I was the on-call engineer that night, and I had to co-ordinate a lot of the work.
Apparently the company that supplied it got sued for a very, very large amount of money...
Obviously some numbnut didnt do their contracted job, therefore missing the fact it was fuelled and in full operational status, alongside a plum in the monitoring centre not doing their job either - or a comms failure between the two. did not care as was not my problem at the time as I worked for a different company who handled the servers and robotic units that failed once this state of the art (?!) genny went down! My problem was reconfiguring stuff afterwards.
The agency that I was working on at the time did like their outsourcing... This was the result it would seem.
Gassing Station | The Lounge | Top of Page | What's New | My Stuff