In engineering, one of the most important parts of a machine is a regulator. It might be a bi-metallic strip that pauses operation of a machine when its temperature reaches a preset level or it valve that lets off steam until the pressure drops below a preset pressure; it might even be a device that prevents the machine turning ever faster by interrupting the power source when a certain speed is exceeded.

Regulators are the unsung heroes of engines because they prevent the machines exceeding their design limits. They are also simple. They are mechanical. In computing terms they are binary: working only when required and that requirement is defined by an algorithm – for example, is the machine spinning too fast, then turn off the driving force until it falls back within the required parameters.

By preventing explosions and machines breaking apart as a result of vibrating at speed, regulators save lives.

We have had machines for thousands of years. Some historians say that wind power was used to lift water in China in 200 B.C. So we have a long history with machines, including those that we set and forget. And as technology became ever more sophisticated, leaving machines to do their thing became, well, a thing.

So what happens when machines are left on their own? The answer is, at worst, people die.

It is said by author Jeremy Clay that the first person killed by "an automaton" was "Mr. Maybrook" who, in 1876, climbed into the clocktower of a church in Germany to witness at close quarters the machinery which consisted of two metal representations of humans which struck the bell with large hammers. Mr Maybrook managed to get himself between a hammer and the bell and it bashed his head in. 1

Clearly the machine worked as intended but there was a missing fail-safe i.e. a way of ensuring that nothing came between the hammer and the bell. It could have been something as simple as a fence.

In January 1979, Robert Williams a factory worker at Ford's Dearborn plant was killed when the arm of a robot hit him. He had climbed a storage rack to make a stock-check. The machine put stock on shelves and took it off again. The manufacturer of the machine, not Ford, was found liable in a civil trial. There seems to be no information about any action relating to the safe operating of machines in the store. Williams is widely reported as "the first person killed by a robot."

What is a robot?

There are more definitions than are useful. They can be distilled down to this: a robot is a mechanised device that operates in accordance with predefined instructions.

But that definition includes, for example, The Spinning Jenny1 and The Water Frame2 .

If we look at industrial robots, the most impressive recent of example of which is the Xiaomi car plant in which a car can be assembled in 76 seconds. That it has to be noted depends on an internal supply chain where completed components are picked and delivered to the assembly line. Everything is automated. 3

Traditionally, injuries and deaths involving machines have been most common in agriculture. This author's own grandfather was one such casualty almost a century ago.

But as machines in factories became not only more common but bigger and stronger, the need for fail-safes became ever greater.

Even so, when we look at reports of industrial injury, the vast majority are the result of human error or even deliberate failure to follow instructions. Across the world, reports of workers who remove guards so they can work faster, or without impediment, are legion.

However, those machines and robots are designed to do a simple task or series of tasks with immense precision and without fatigue or, even more important, boredom. They are autonomous only in the sense that their predefined function does not require intervention.

So, when it comes to Rogue Robots, we can safely ignore all of that.

What makes robots go rogue is software – and that means human – failure.

Robots as the term is understood today do more than repetitive tasks. But they still have one thing in common with industrial age machinery: they respond to stimuli. The governor with its action based on a binary condition (too much steam, open valve) is the direct ancestor of the function of robots. They respond to stimuli generated by sensors.

So, when the sensors don't work properly, for example, B737-800 MAX planes kill hundreds of people.

But are those who are responsible for identifying failure looking in the right places?

The USA's National Transportation Safety Board has, it has been reported, said that it wants only "serious" incidents involving self-driving cars to be reported to it. So the world is dependent on media and social media coverage of incidents or on self-reporting by manufacturers on, for example, their own marketing or corporate websites.

Two examples of fatal car crashes indicate that there may be a factor at play that the NTSB did not consider.

In 2016, "a car operating with a automated vehicle control systems" was involved in a fatal crash with a large articulated lorry. The car ran under the trailer, tearing off the roof of the car. The NTSB found that the "probable" proximate cause of the crash was that the lorry driver pulled out of a side-road in front of the car which had right of way. However, the this was "combined with the car driver's inattention due to over reliance on vehicle automation, which resulted in the car driver’s lack of reaction" when the lorry pulled out. The NTSB said "Contributing to the car driver’s over reliance on the vehicle automation was its operational design, which permitted his prolonged disengagement from the driving task and his use of the automation in ways inconsistent with guidance and warnings from the manufacturer."4

The report does not make a finding that the car should have identified the threat and avoided it.

In 2019, the NTSB had to look at a crash that had similarities with the 2016 crash. An articulated lorry driving on a dual carriageway crossed, legally, the opposite carriageway to make a turn. The car – the same model as the previous incident - went under the trailer and "coasted" to a stop almost 1700 feet further on. Again, the roof was torn off and the driver killed.

Again, the proximate cause was taken as the lorry driver having failed to give way to oncoming traffic.

But this time it went further in relation to the automation saying "combined with the car driver’s inattention due to over reliance on automation, which resulted in his failure to react to the presence of the truck. Contributing to the crash was the operational design of Tesla’s partial automation system, which permitted disengagement by the driver, and the company’s failure to limit the use of the system to the conditions for which it was designed."5

This change of attitude demonstrates that there must be effective fail-safes. In short, the principle that required the use of regulators in the 19th Century. The NTSB went further : the failure of the National Highway Traffic Safety Administration to develop a method of verifying manufacturers’ incorporation of acceptable system safeguards for vehicles with Level 2 automation capabilities that limit the use of automated vehicle control systems to the conditions for which they were designed.​" It's a different type of regulator and in this case one that the NTSB said was not doing a good enough job.

But the NTSB appears to have failed to connect the two crashes and to look at a common factor. The car went under the trailer, into a gap under in front of the rear wheels. That gap is taller than the bonnet of the cars.

Did the car in fact "see" the trailer, identify the space under the trailer between the wheels and go for the gap, not recognising, amongst other things, the lack of height which was sufficient to clear the car's bonnet but so low that the entire windscreen frame was removed.

The NTSB, basically, decided it was, fundamentally, operator error.

But is it? Or is it simply that the sensors provided information that the algorithms read as "there's a gap. Go for it." Nothing in the reports indicate whether the car recorded that it sped up as it aimed for what was a moving target. Did the NTSB even consider that the car's programming was making the kind of value judgment that no one talks about any longer; does the car crash itself and risk injury to the driver or does it crash into bystanders in order to save the driver? Did the car's algorithms indicate that to go for the space (insofar as it assessed the size of the space) was safer than to jam on the brakes and risk sliding into the rear axle of the trailers?

When the car, having been slowed by an impact that was so severe it ripped off the entire structure above bonnet level and still travelled 1,700 feet, who made the decision to allow the program to propel the car at such a speed? According to Omnicalculator 6 a speed of 150mph and a one second reaction time results in a stopping distance of 1,300 feet. That's clearly faster than the Teslas were travelling in both of these instances yet at least one travelled much further. It was described as "coasting" i.e. freewheeling, without braking, to a stop which raises another question: where is the failsafe that brings what has, by that point become a projectile, to a safe halt to protect third parties?

In 2018 the NTSB investigated a crash where a car drove into a stationary fire engine. Again, the NTSB said that blame lay not with the car but with an inattentive driver. "We determined that the probable cause of the rear-end crash was the Tesla driver’s lack of response to the stationary fire truck in his travel lane, due to inattention and over reliance on the vehicle’s advanced driver assistance system; the Tesla’s Autopilot design, which permitted the driver to disengage from the driving task; and the driver’s use of the system in ways inconsistent with guidance and warnings from the manufacturer.​"7

Worryingly, the NTSB stopped publishing its series of reports into investigations of crashes involving autonomous vehicles in 2018 but a short media search reveals many incidents. There may be a clue in the blurb on an article in The Verge which says "Of 20 incidents, only two met the federal government’s reporting criteria, and no one was injured."8 In both cases, it reports, a car that was stationary at traffic lights was run into by a car driven by someone who was looking at a phone at the time.

But, Waymo's own data shows that the other 16 crashes had some interesting consistencies, not the least of which were that several were low speed collisions with other, stationary, Waymo's cars.

"Waymo says 55 percent of these minor contact events involved another driver colliding with a stationary Waymo vehicle, and 10 percent occurred at night. None of the events took place at intersections, where most vehicle crashes occur, nor did any involve pedestrians, cyclists, or other vulnerable road users. "

In the UK, there is no equivalent of the NTSB.

A report by the Royal Automobile Club in 2023 said "No transport is perfectly safe. Equipment can fail, people can make mistakes or parts of a system may be incompatible when they come together."9 It goes on "transport services should provide the level of safety that society demands and for which it is prepared to pay" and that accidents under review "occurred because of failings by management, institutional culture, regulatory [supervision] or international coordination. The regulatory framework has to address these as well
as the behaviour of front-line individuals."

In 2002 two aircraft collided near the German village of Überlingen. It involved a Tupolev passenger plane and a Boeing cargo plane. All passengers and crew were killed. The automated systems on both aircraft warned the crews that there was potentially conflicting traffic. Both aircraft, with permission, were flying at 36,000 feet. The accident report found that, amongst the causes, was that there was insufficient integration of systems and therefore there were gaps in the data and delays in responding10.

This shows that even when it is present, automation is subject to human error in the way it is connected.

The fundamental questions are these:

1. do Robots go Rogue or are they as much a victim of inadequate instructions as the people they injure, maim or kill?

2. If it is long established that if a regulator "pops off" and fails to perform its function liability rests with the company (not the individual) operating it albeit with the potential for a third party claim against the manufacturer, why do we assume that a failure to turn off a robot in anticipation of a bad decision is the driver's fault?

3. Why are governments so reluctant to put the blame on the designers of computerised systems that have any autonomous function?

--------------------------------

1. https://www.bbc.com/news/blogs-magazine-monitor-27527035
2. https://economic-historian.com/2022/07/spinning-jenny/
3. https://interestingengineering.com/innovation/richard-arkwright-and-his…
4. https://www.youtube.com/watch?v=gi11NPQciao – it would be best to find the original of this and link to it. This is one of many reposts on YouTube.
5. https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1702.pdf
6. https://www.ntsb.gov/investigations/AccidentReports/Reports/HAB2001.pdf
7. https://www.omnicalculator.com/physics/stopping-distance
8. https://www.ntsb.gov/investigations/AccidentReports/Reports/HAB1907.pdf
9. https://www.theverge.com/2023/2/28/23617278/waymo-self-driving-driverle…
10. See, for example, https://financialcrimeriskandcompliance.com/elan/web/20170712_mundane
11. https://aviation-safety.net/asndb/323026