BA systems down globally

Author
Discussion

The Selfish Gene

5,517 posts

211 months

Tuesday 11th September 2018
quotequote all
they must do some internal as I was approached recently to go and work there via a headhunt agency.

The job would have been in this area.

I declined although was interested.

anonymous-user

Original Poster:

55 months

Tuesday 11th September 2018
quotequote all
Didn't BA recently fire hundreds of IT staff and outsource their jobs to India? scratchchin

The Selfish Gene

5,517 posts

211 months

Tuesday 11th September 2018
quotequote all
fblm said:
Didn't BA recently fire hundreds of IT staff and outsource their jobs to India? scratchchin
they won't have fired them, that's difficult. My have had redundancies.

That's normal in the industry, but generally the Head of levels will all be UK.



The Selfish Gene

5,517 posts

211 months

Tuesday 11th September 2018
quotequote all
fairly possible it was someone that had been hired to work there - and not the old version of a hack too.


captain_cynic

12,084 posts

96 months

Tuesday 11th September 2018
quotequote all
The Selfish Gene said:
fblm said:
Didn't BA recently fire hundreds of IT staff and outsource their jobs to India? scratchchin
they won't have fired them, that's difficult. My have had redundancies.

That's normal in the industry, but generally the Head of levels will all be UK.
The head of levels will be blameless in this though.

At worst, one will have a golden parachute.

The Selfish Gene

5,517 posts

211 months

Tuesday 11th September 2018
quotequote all
captain_cynic said:
The Selfish Gene said:
fblm said:
Didn't BA recently fire hundreds of IT staff and outsource their jobs to India? scratchchin
they won't have fired them, that's difficult. My have had redundancies.

That's normal in the industry, but generally the Head of levels will all be UK.
The head of levels will be blameless in this though.

At worst, one will have a golden parachute.
well I'm a Head of for a similar company - and I would have definitely been fired for this...........

Jobs in the past (not this one) I could also technically go to prison in certain circumstances.

Whoever was running this must have been fired though - as the head-hunters are calling out to the industry..

Unless they have a vacuum in the area and are looking to bolster it, but I'm amazed.

dmsims

6,541 posts

268 months

Tuesday 11th September 2018
quotequote all
anonymous said:
[redacted]
Fair enough on the CSP

The file integrity check is purely for IT, nothing to do with end users, values can be stored on a separate server in a separate db

It is not expensive and very simple

CzechItOut

2,154 posts

192 months

Tuesday 11th September 2018
quotequote all
anonymous said:
[redacted]
It will be interesting to see how BA found out about this.

Gareth79

7,698 posts

247 months

Tuesday 11th September 2018
quotequote all
dmsims said:
Fair enough on the CSP

The file integrity check is purely for IT, nothing to do with end users, values can be stored on a separate server in a separate db

It is not expensive and very simple
The integrity value on the script tag is for this sort of situation, but only makes complete sense when the script is hosted by a third party. Eg. for the Browsealoud hack a few months ago that hit a lot of governement sites - the file would have been modified since the integrity hash was created and the script would have failed to validate and run in the user's browser.

It might have helped here if the hacker was able to modify the javascript file, but not where the script tags were being inserted into the page content (the CMS or templating engine), but I suspect they had full access :/



s2art

18,937 posts

254 months

Tuesday 11th September 2018
quotequote all
Not sure I understand all this. Dont BA have a test rig where they can test stuff and monitor the outputs? Easy enough to spot spurious messages being sent.

CzechItOut

2,154 posts

192 months

Tuesday 11th September 2018
quotequote all
s2art said:
Not sure I understand all this. Dont BA have a test rig where they can test stuff and monitor the outputs? Easy enough to spot spurious messages being sent.
Presumably only the production version of the file was modified with the additional code.

s2art

18,937 posts

254 months

Tuesday 11th September 2018
quotequote all
CzechItOut said:
s2art said:
Not sure I understand all this. Dont BA have a test rig where they can test stuff and monitor the outputs? Easy enough to spot spurious messages being sent.
Presumably only the production version of the file was modified with the additional code.
You would still think that the production version would be copied over to a sandboxed test rig.

The Selfish Gene

5,517 posts

211 months

Tuesday 11th September 2018
quotequote all
s2art said:
Not sure I understand all this. Dont BA have a test rig where they can test stuff and monitor the outputs? Easy enough to spot spurious messages being sent.
s2art said:
Not sure I understand all this. Dont BA have a test rig where the can test stuff and monitor the outputs? Easy enough to spot spurious messages being sent.
not that simple

my similar company (although bigger) have approximately 9 test environments for about a 1/5th of the business.

Each of those environments have stacks, OSS, with maybe 100 systems in each - all integrated together.

These things are hugely complex - I'm not entirely sure what went wrong at BA (other than a hack) as it's a bit of busman's holiday reading about it and I have enough professional issues of my own dealing with here.

Also, to note - the failover issue they had either earlier in the year, or maybe last year that they blamed on not testing properly in my opinion was almost certainly a hack for bitcoin.

same as the NHS definitely was.

This is a hugely complicated set up

The easiest way to hack and inject code is to get a job there, trust me, it is very easy to do if you're a skilled developer and want to make money with hacking.

This is not mission impossible movies.

Although we do protect from rogue cleaners etc, there isn't much we can do about a skilled employee.

s2art

18,937 posts

254 months

Tuesday 11th September 2018
quotequote all
The Selfish Gene said:
s2art said:
Not sure I understand all this. Dont BA have a test rig where they can test stuff and monitor the outputs? Easy enough to spot spurious messages being sent.
s2art said:
Not sure I understand all this. Dont BA have a test rig where the can test stuff and monitor the outputs? Easy enough to spot spurious messages being sent.
not that simple

my similar company (although bigger) have approximately 9 test environments for about a 1/5th of the business.

Each of those environments have stacks, OSS, with maybe 100 systems in each - all integrated together.

These things are hugely complex - I'm not entirely sure what went wrong at BA (other than a hack) as it's a bit of busman's holiday reading about it and I have enough professional issues of my own dealing with here.

Also, to note - the failover issue they had either earlier in the year, or maybe last year that they blamed on not testing properly in my opinion was almost certainly a hack for bitcoin.

same as the NHS definitely was.

This is a hugely complicated set up

The easiest way to hack and inject code is to get a job there, trust me, it is very easy to do if you're a skilled developer and want to make money with hacking.

This is not mission impossible movies.

Although we do protect from rogue cleaners etc, there isn't much we can do about a skilled employee.
I have worked on and implemented highly secure systems. The clients insisted we implemented a test rig that mirrored the production system. It wasnt cheap.

dmsims

6,541 posts

268 months

Tuesday 11th September 2018
quotequote all
anonymous said:
[redacted]
All the things you have mentioned can be done easily, it's a website at the end of the day, if they are too stupid to do it themselves buy a commercial solution

No need to make a distinction between file types, js, html, php - whatever

I am utterly amazed there was no integrity monitoring

Code was added to a file (that's the favourite theory)

The Selfish Gene

5,517 posts

211 months

Tuesday 11th September 2018
quotequote all
s2art said:
I have worked on and implemented highly secure systems. The clients insisted we implemented a test rig that mirrored the production system. It wasnt cheap.
then you will know, that it makes no difference.

Other than to test baselines of code, load, performance and failover, even on a staged rig that is a % the size of production.

If someone injects code into your baseline, at the appropriate time it (the baseline) will be functionally tested and flow through the lifecycle. Thereby making it from the 'sandpit' , to functional and integration testing.

The business will sign it off with it in there (as it will be dormant), and then it will be operationally tested, deployed and post verified............all dormant.

The only way to catch it would be with penetration and/or security - and in my experience, the hackers are usually more skilled than the security test teams/software.

This is also exactly why the government have told people to stop using Kaspersky for AV/AS..........Russian owned etc.

It's the modern version of putting a bug somewhere in the cold war............the problem is when it comes to software thus far we don't have the best 'bug' detectors as yet.

The best way is to keep them out of your systems, again though, if they're part of the team developing thousands of lines of code - unless you're doing the equivalent of a FAGAN inspection (which we used to do on military applications) you're unlikely to spot 20 lines of rogue code, written by an employee at the correct time in the process.

Still, all part of the rich tapestry of life isn't it.



essayer

9,085 posts

195 months

Tuesday 11th September 2018
quotequote all
I think a correctly implemented CSP would have mitigated it here, if the hackers couldn't control the webserver to turn the headers off ? Obviously if they can change any file all bets are off, but if all they can do is drop in new web code, it could have prevented a large set of c/card details being stolen

CSP connect-src, which would apply to all scripts on that page, blocking the request to baways.com, right?

Also if they'd added report-uri, someone might have seen it in a log. (hahaha)

Food for thought, anyway.

s2art

18,937 posts

254 months

Tuesday 11th September 2018
quotequote all
anonymous said:
[redacted]
I just phrased it badly. I meant that the production version would be implemented on a test rig. (from the same repository). This is why its not clear to me.

Harpoon

1,872 posts

215 months

Tuesday 11th September 2018
quotequote all
The Selfish Gene said:
The easiest way to hack and inject code is to get a job there, trust me, it is very easy to do if you're a skilled developer and want to make money with hacking.

This is not mission impossible movies.

Although we do protect from rogue cleaners etc, there isn't much we can do about a skilled employee.
RiskIQ are attributing the attack to the same group (Magecart) that hit Ticketmaster and others

https://www.riskiq.com/blog/labs/magecart-ticketma...

The Selfish Gene

5,517 posts

211 months

Tuesday 11th September 2018
quotequote all
dmsims said:
All the things you have mentioned can be done easily, it's a website at the end of the day, if they are too stupid to do it themselves buy a commercial solution

No need to make a distinction between file types, js, html, php - whatever

I am utterly amazed there was no integrity monitoring

Code was added to a file (that's the favourite theory)
that works on small implementations - I'm not sure how integrity monitoring something the size of a BA release would work. Remember all these things are risk based against cost.

All of my answers above though are based on me not working there.

Chances are, BAs system are developed and tested on the cheap by an outsource company and it was always going to happen at some point. All of the outsources I've worked with over 20 years have been utter ste - cheap though.

I'd take 20 guys sitting in front of me, every day. Throats to choke are more important than cost, as many organisations are starting to find out now that security of platforms is costing them real money and reputation.