Friday, 6 July 2012

When turning it off and on again doesn't work

It has always been one of my nightmare scenarios: a software update that goes wrong. The average web site now has so many dependencies ... the OS, Apache, PHP, MySQL, JQuery, Drupal ... to name but a few of the pieces of software, many open source, that need occasional updating on your site ... that it is difficult to keep up. Therefore I have every sympathy with the banks whose systems failed to recover from a maintenance update the other week. It could happen to anyone.

I realise that the bank problem was real heavy-lifting IT, and the interactive media world rarely strays into that kind of territory. But occasionally it will.

So what do you do when it happens to you, and especially when you're the code-monkey who pushed the metaphorical or even literal button to kick-start the mayhem?

According to the brief biog at the foot of his pages, Register writer Dominic Connor 'used to boss around gangs of quants, developers and sysadmins before moving to a life of headhunting in the City spiced up with consultancy' and 'Before becoming a City headhunter, ... was a mildly competent head of IT as well as developer for various banks. RBS still run some decade old code of his, he wishes them good luck with it.'

His two pieces on the subject published on the Register last week make for interesting reading, especially if you manage an IT team or are part of it. I recommend them:

Do you work in IT at RBS? Or at the next place to get hit ...? (June 29) How to survive a crisis.

So, that vast IT disaster you may have caused? Come in, sit down (July 3rd) Welcome to the world of forensic interviewing.