When Software Security company Crowdstrike issued an update to one of the world's most popular enterprise-level security programs, a problem caused computers to hang. As the update was rolled out globally, at the same local time in each timezone, the effect was similar to a rolling blackout affecting the whole world, at least insofar as the world uses Microsoft's Windows operating system. Systems that were vital to the operations of businesses stopped working, unless they were running an alternative operating system.

Identifying the source of the problem was not instant: many wrongly blamed Windows, others a distributed denial of service attack and the one thing no one initially alighted on was a bug in the updated security software.

Worse, once it was identified, it proved difficult and time consuming for businesses to undo the update. Crowdstrike, to its credit, once the problem was identified, immediately admitted that it was the source of the problem, reassured users that there was not a problem with any other part of their system, that there was not an external (or even internal) attack and produced a patch that repaired the update and re-ran it. There was no data lost except any that was in transient storage (RAM, swap files, etc) and no actual security risk or breach.

But everyone agrees: It has happened before with many other products, it should not have happened on any of those occasions and it should never happen again.

On 24th October, multiple US Government agencies and offices have issued "Guidance" for "safe software deployment". It's even been more or less replicated in other countries, for example Australia at the same time, but the next day (demonstrating the time-shift that is inherent in global activities).

The USA's Cyber Defense (sic) Agency headlined its announcement "CISA, US, and International Partners Release Joint Guidance to Assist Software Manufacturers with Safe Software Deployment Processes". CISA is the USA's Cyber Ifrastructre and Security Agency, one of very many government bodies in the field.

The guidance is here: https://www.cisa.gov/resources-tools/resources/safe-software-deployment…

CISA says "This guidance aids software manufacturers in implementing a safe software deployment process with robust testing and measurement components. The process laid out by CISA, FBI, and Australian Cyber Security Centre (ACSC) helps both the security and quality of products and deployment environments.

"This guide provides an overview of six key phases in a safe software deployment process. The authoring agencies strongly encourage the use and regular maintenance of playbooks that provide clear guidelines, best practices and contingency plans meant to guide teams through each phase of software deployment. This guidance is of moderate technical complexity and assumes a basic understanding of cybersecurity."

Basically, it's a scarily basic set of polcies that all software companies, be they giants or tiny open source providers, should always have followed:

They can be summarised as "test, test and test again and fix whatever you find, then test some more and, when you are satisfied, roll it out slowly so as not to cause a mass extinction event across critical and commercial systems."

Also, although it doesn't call it such, there should be a hot-button rollback which, again, should have long been standard practice (but let's face it - only organisations with dedicated staff and massive resources do a system-wide backup before applying patches, especially where there are many desktop, laptop and mobile "touch-points").

Laden with silly buzzwords such as "dog food" and "canary testing", the fundamentals of the guidance is sound. Will it be adopted in the vast majority of, particularly small, software houses? No. If it did, the entire RegTech industry would be dead tomorrow. Across the entire spectrum of software the tendency to release for public use "beta" versions (i.e. "we know it's buggy and we'd like you to debug it for us") has been commonplace since Netscape issued frequent releases of the beta versions of the next version of its Navigator browser. The actual release version was a paid product; the betas were free and widely distributed on floppy disks stuck to the front of magazines.

Yesterday, WMLR reported that a financial sector regulator has announced a "beta" chatbot.

So there's an uphill battle in trying to make software companies release only complete and reliable software. The rush to market means that packages are sometimes updated several times per month, some with features, most with fixes.

So, what effect will this Guidance have?

1. Wise companies will want to see compliance with it before implementing software.

2. Where is it not followed, that will be evidence in negligence claims against the company. Proof it was followed will be persuasive but not conclusive evidence that the company did what it could reasonably be expected to do, or that should be the case if courts do their job.

And the courts will decide what "reasonably" means in those circumstances.