//php echo do_shortcode(‘[responsivevoice_button voice=”US English Male” buttontext=”Listen to Post”]’) ?>
At first glance, the combination of artificial intelligence (AI) and automotive functional safety requirements do not appear to add up. Unlike traditional software, AI learns from patterns in data that may not be visible to humans, producing an algorithm with variables that may not appear to directly correlate with anything in the input data.
“Put simply, AI is right for the wrong reasons,” Retrospect CEO Michael Woon told EE Times. “It’s been trained to come up with the right answers under certain input conditions, but why is it doing that? We don’t know.”
Woon, who runs a functional safety consultancy, points out that AI is not incompatible with safety in general; AI-powered systems do protect humans from unsafe working conditions. But, he cautioned, AI frequently appears to be a “black box” solution, which is totally at odds with the way today’s safety-oriented systems are developed.
“The notion that [AI and functional safety] seem not to fit together is a good one,” Woon said. “It will take the right kind of explainability in the market to explain why AI alone should not be used in safety critical applications.”
The vast majority of software in functional safety applications is not safety certified, he said. In general, only specific parts of software in specific parts of an application are truly safety-critical—not the entire software system.
Often, Woon said, it’s electronic hardware or mechanical safeties that really bear the safety burden in a lot of cases. In today’s autonomous vehicles (AVs), the safety burden is often carried by a person—a safety driver who must take over in unsafe situations. Whether it’s a human in charge, or another piece of software (that isn’t AI-based), the point is that an AI algorithm should not be solely responsible for functional safety.
AI algorithms will fall into a category called “quality-managed software.” Quality-managed software is developed with some form of quality assurance or quality control and is peer-reviewed and thoroughly tested to prove there are no bugs with respect to the intended functionality.
“‘Quality managed,’ by and large, means ensuring some amount of repeatability, some demonstration of hitting the targets, and doing so to a level that, when deployed at commercial scale, it isn’t going to cause a lot of issues—so you’re not going to have damage to your credibility and reputation,” Woon said.
Quality-managed software
Woon draws parallels between today’s autonomous driving AIs and the automotive safety standard E-GAS, initiated by the German automotive industry in the 1990s. E-GAS was a response to new engine combustion strategies, which are extremely proprietary (they cannot even be shared between OEMs and tier 1s). The difficulty in the 1990s was how to prove the engine could not cause unintended acceleration—the primary failure mode to be prevented.
E-GAS separates software into three levels:
- Level 1 is for top secret, cutting edge, non-safety-certified software that would control the engine with optimal fuel economy and minimal emissions.
- Level 2 is a guardrail for level 1, such that level 1 cannot violate the constraints of level 2; it might be monitoring the overall fuel and air intake to ensure the engine does not exceed a certain amount of torque.
- Level 3 is the software that monitors level 2 to make sure it is working properly. This way of containing quality-managed software using safety-certified software may also be a good solution for AI, Woon said.
“AI always needs to be doing the cutting edge, latest and greatest [functionality] with the latest data and needs to be continuously updated,” he said. “I think that would be the quality-managed software, level 1, but then around that we would have different algorithms that would be deterministic, inspectable and safety-certified—and would contain [AI] to make sure AI isn’t going to go out into some unsafe region.”
It would be difficult to fit AI chip and algorithm systems into the ISO 26262 automotive safety integrity levels (ASILs A to D), Woon pointed out, since AI has a prediction accuracy. That is, it is not 100% reliable, even when working perfectly.
Even with prediction accuracy of 99.9%, if the AI algorithm misclassified an object, how would we know whether that was due to some kind of hardware fault condition, or just the 0.1% of the time the AI is expected to get it wrong? This is part of the reason quality-managed software is a better fit here, he said.
Diversified redundancy
Separately, there are other ways AI can contribute to safety in autonomous driving.
One thing Woon’s company, Resonate, works on is the automation of safety assessments in real time, or close to real time, to ensure risks are being recognised and managed safely. This could mean using algorithms to detect failing light, bad conditions or other drivers driving dangerously.
“As human drivers, [we have two roles]—we’re not just focused on controlling a vehicle to get from A to B,” he said. “We’re also asking ourselves, ‘Are my wipers good enough for me to drive, or do I need to get those fixed? Am I too sleepy, do I need to get coffee?’ We’re monitoring all these different things.”
Diversified redundancy, which fuses data from different sensor modalities (camera, ultrasound, radar, lidar, etc.), is a widely used technique in today’s AVs. The idea is not simply to enable redundancy by running two identical systems, but to make the overall system more robust with sensors and sensor algorithms with different susceptibilities and failure modes.
“Unfortunately, with diversified redundancy… there are just that many more effective ‘waves of water on the windshield’—with lidar, radar and cameras, there’s all sorts of different things that can affect the sensing modes and we can create algorithms to detect when those conditions occur so the autonomous algorithm can still react safely,” he said. A safe reaction might be slowing down, stopping or deemphasizing the sensor modality affected in some way.
Consumer trust
How should AV makers be building consumer trust regarding safety?
While consumers are used to software in all kinds of devices needing updates and bug fixes, publicly stating that the latest updates “improve safety” might not be the way to build consumer trust, since it implies the system was not safe before the fix, Woon said.
Public trust in safety-critical software should instead be encouraged via demonstration.
“The number of hours and the number of miles driven is meaningless,” he said. “That’s a good metric for quality-managed software, but the point of functional safety is that when things go wrong, you need to show the safety system can still maintain a safe state.”
Woon would like to see AV makers clearly displaying to the public all the potential failure modes and the safety mechanisms that keep them from happening.
“The most convincing evidence is, if you demonstrate through fault-injection strategies, that we exercised the safety mechanism and it works, and that we can always monitor the safety mechanism is working, and if it ever isn’t working, we’re going to stop,” he said. “That’s the type of evidence that should be made publicly available and it’s unique to each developer and each platform.”
Asked whether AI can ever be proved to be “safe enough,” Woon said that the question really should be, “How safe is too safe?”
“You can add a third camera, a fourth lidar, a fifth radar, but you’re not really doing that much more to affect the overall safety,” he said. “You can make it safer, but what’s the real, measurable benefit from it? If those things fail, the safety mechanism still needs to stop the car safely.
“I think once we can define the point at which we’re being too safe, then we’ll have standardization across the industry. But if we’re still not up to that level, that’s a problem.”
