Well?

[0a]

I'd echo the feedback and say thanks for answering our questions, very interesting indeed. It's a credit to the site that feedback is dealt with in such an open way.

I have also had sites go down on me before. That call when you are in the pub on a Friday (why do my sites go down on a Friday?) is not one you want to take. The last one was on brand new mirrored infrastructure ("it can't go down" - titanic?).

[Famous Graham]

0a said:

The last one was on brand new mirrored infrastructure ("it can't go down" - titanic?).

You're not alone. Amazon's EC2 setup in Ireland got clobbered by a lightning strike at the weekend. Took all of our production servers down with it :/

[DJRC]

Dennis99 said:

I'm so old I remember when that was BS 5750 and Def Stan 05-21 before that.

Hey I still refer to Mil Stans and Def Stans

[0a]

Famous Graham said:

0a said:

The last one was on brand new mirrored infrastructure ("it can't go down" - titanic?).

You're not alone. Amazon's EC2 setup in Ireland got clobbered by a lightning strike at the weekend. Took all of our production servers down with it :/

This one was a multi million user site, hundreds of thousands spent to prevent the issues we'd had in the past, several secure and pci compliant data centres with non pci backup if need be to ensure the site could actually work.

Went down on day 3

[Sonic]

0a said:

Famous Graham said:

0a said:

The last one was on brand new mirrored infrastructure ("it can't go down" - titanic?).

You're not alone. Amazon's EC2 setup in Ireland got clobbered by a lightning strike at the weekend. Took all of our production servers down with it :/

This one was a multi million user site, hundreds of thousands spent to prevent the issues we'd had in the past, several secure and pci compliant data centres with non pci backup if need be to ensure the site could actually work.

Went down on day 3

Out of interest, what took it offline?

[KaraK]

Sonic said:

Stuart said:

On the encryption front, when we acquired the site passwords were not encrypted and were stored in text. We added encryption to all passwords of member accounts early in 2008, and we use AES to encrypt passwords.

The inherent problem with encryption when used in this application is that it's designed to allow decryption, using the same key in AES, which you'll probably be storing in a server-side script for easy processing. Nice and insecure, and completely flawed if the server or code is compromised.

I didn't see an IV field in the user table scheme on that hacker site, so i hope you're not using the same IV and key for each password?

It's quite common in a lot of AES implementations to simply store the IV at start of the ciphertext so the lack of a seperate column doesn't mean that a standardised IV is being used.

Sonic said:

Now, about storing account passwords in a decryptable state...

I'm with you there - AES is only as good as your key storage and storing that securely on an web server while still being able to actually use it is a serious headache. AFAIK the forgot password functionality on PH just generates you a random password so if it were me I'd just be using an SHA implementation with a per-user salt (just to make life harder for all the Rainbow Table enthusiasts out there)

NB: I'm not criticising PH here! Just a topic I'm interested in (although woefully out of touch lately)

[Steffan]

Clearly PistonHeads have a problem with their administration or technology.

Some business offer lots of excuses some businesses offer the truth.

Up to PistonHeads. Clearly there is and has been for some days a problem

[0a]

Sonic said:

0a said:

Famous Graham said:

0a said:

The last one was on brand new mirrored infrastructure ("it can't go down" - titanic?).

You're not alone. Amazon's EC2 setup in Ireland got clobbered by a lightning strike at the weekend. Took all of our production servers down with it :/

This one was a multi million user site, hundreds of thousands spent to prevent the issues we'd had in the past, several secure and pci compliant data centres with non pci backup if need be to ensure the site could actually work.

Went down on day 3

Out of interest, what took it offline?

It turned out there were some major errors in how the main site dealt with the mirror, both were corrupted. I'm not a tekkie! As I understand it code was changed on the main site which meant the database was being accessed very inefficiently. An incident occurred which led to 10x traffic and db requests, should not have been a problem as we expect this. The code had been copied to the mirror and emergency sites - so no point in having them!

Eta: of couse when I rang various people they were already working on it. Sods law that I was rang in the pub by our American owners.

Edited by 0a on Thursday 11th August 15:47

[Stuart]

Steffan said:

Clearly PistonHeads have a problem with their administration or technology.

Some business offer lots of excuses some businesses offer the truth.

Up to PistonHeads. Clearly there is and has been for some days a problem

What? Are you implying that anything I've written here is untrue?

[0a]

Steffan said:

Clearly PistonHeads have a problem with their administration or technology.

Some business offer lots of excuses some businesses offer the truth.

Up to PistonHeads. Clearly there is and has been for some days a problem

All I'll say is I've not come across a business that allows their staff members to post regarding an outage in the way Stuart has here before. If you have a question I'm sure he will answer, you don't have to use this website.

[Snoggledog]

Steffan said:

Clearly PistonHeads have a problem with their administration or technology.

Some business offer lots of excuses some businesses offer the truth.

Up to PistonHeads. Clearly there is and has been for some days a problem

Bit rough when you consider that this site started with a bloke doing it in his spare time and it's grown considerably since then. I'd lay a wager that some of Ted's early coding still resides within PH. I've worked on systems which have outgrown their original platform and design brief and can honestly say that PH has more uptime than a few systems I've been involved with. Hacking back old code and putting in place nice shiny new code is a nightmare for developers no matter how well it's been documented.

ETA. That's not aimed as a dig at Ted. I doubt that when he started, he ever foresaw the size and scale of PH as it is now.

[Steffan]

Stuart said:

Steffan said:

Clearly PistonHeads have a problem with their administration or technology.

Some business offer lots of excuses some businesses offer the truth.

Up to PistonHeads. Clearly there is and has been for some days a problem

What? Are you implying that anything I've written here is untrue?

Not al al;l. No such suggestion was implied or intended. I never thought that at all

I simply applaud the efforts to tell the users what is happening.

Every business suffers IT problems.

[jammy_basturd]

[redacted]

[Steffan]

Stuart said:

What? Are you implying that anything I've written here is untrue?

I do not wish to suggest for a moment that any of the Pistonheads staff are not genuinely trying to be helpful. Nor do I intend to impune the honesty of any individual whatsoever.

However I do think it is better when dealing with a problem to admit to it and state that everything is being done within resources to resolve this difficulty.

I learnt in business that the abscence of confirmation can appear to be a denial

The early reports of difficulties on Pistonheads suggested a temporary difficulty lasting a few minutes.

A few days later it seems that all is still not well.

Good communication requires straightforward statements which are regularly and accurately updated.

I do think this problem could have been handled better but no doubt my many of my former clients would say the same of me.

It is not a perfect world as I know to my cost.

I frequently quote the epithet to clients:

'When you are up to your Ar-e in Aligators it is not easy to remember that the object of the excercise was to drain the Pi--ng Swamp'

I hope the generally excellent website and service will resume asap.

[AndrewW-G]

Steffan said:

Stuart said:

What? Are you implying that anything I've written here is untrue?

Blah Blah Blah

I'm sure you'll agree that this site represents excellent value for money

[Sonic]

AndrewW-G said:

Steffan said:

Stuart said:

What? Are you implying that anything I've written here is untrue?

Blah Blah Blah

I'm sure you'll agree that this site represents excellent value for money

I feel like i have invested far too much time into this site

[Stuart]

Righto, hopefully this serves as an update of sorts.

Pete has spent the whole day crawling over our log files, and we're satisfied that there has been no data leak in the attempt to hack the site which took place yesterday. The site outage was caused by a sudden influx of traffic, but the time it took to get back up was simply because we decided to remove vulnerabilities before doing so, rather than because any damage had been done.

The site has had a few bugs today around classified submission, but we believe that these are related to some of the code changes which were deployed yesterday to close off a couple of vulnerabilities. We have now opened up the classified user edit tools, which means that ads can once again be submitted and edited.

We'll keep monitoring this over the coming days, and there are bugs which we'll need to fix ASAP, but the key thing is that our data hasn't been stolen and no action is required by users.

Thanks all for your patience.

Stuart

[TonyRPH]

The site is really slow and unresponsive right now (17:25 - 17:30)

It took nearly a minute just to load the "Reply to topic" page.

ETA: It took 90+ seconds after clicking "Submit" to process this post.

Edited by TonyRPH on Thursday 11th August 17:34

[carmadgaz]

Glad to hear you guys are on top of it

Still a bit glitchy here on Firefox but as many people have said we get great value for money on this forum so can't complain

[monthefish]

TonyRPH said:

The site is really slow and unresponsive right now (17:25 - 17:30)

It took nearly a minute just to load the "Reply to topic" page.

...v...e...r...y...............s...l...o...w.....................