How to Build Trust into AI-Driven Software Engineering

01 · Can AI be trusted?

Can we simply trust what the AI produces? In our conversations with customers, there are four questions that come up frequently, and the answer is the same every time.

• Will AI get good enough that we trust the implementation it creates? No. There's no demonstrable path to a model whose output meets the evidence requirements of ISO 26262, IEC 61508 or related standards on its own.
• Can the AI check its own output? No. Same model, same blind spots. Self-evaluation isn't an independent argument.
• Can a second AI check the first? No. Two opinions aren't evidence. If they disagree, nothing decides. If they agree, nothing is proven.
• Can the AI take responsibility for the result? No. A person does. That person needs reasoning they can trace and explain.

What about guardrails, prompt engineering, skills, rule files and coding guidelines in the context window? They help, and they're worth using. But they only steer a probabilistic system. The result is still not deterministic, not reproducible and not explainable. It can't be audited, and it can't be qualified as a tool. Better output on average is not evidence from a safety perspective.

What AI output lacks is not intelligence. It's the properties evidence requires.

None of these answers changes when the models get better. Waiting for better models is not a strategy.

02 · The obvious fix: human review

If the AI's output can't be trusted on its own, a person has to check it. So the natural answer is human review: engineers read the generated code or Simulink models and check the generated tests. Many AI engineering frameworks propose this approach.

But implementations get created faster with AI. Manual review and hand-built test suites don't scale, creating a growing bottleneck and trust gap.

Manual reviews are also error-prone, especially because AI-generated code or models might not look the way the engineer would have built them, which makes them harder to judge.

03 · The good news: nobody ever trusted the author of the code

Here's the good news. Safety engineering never trusted the author in the first place. Hand-written C isn't trusted because a person wrote it. Code generated from a Simulink model was never trusted because a code generator produced it. It earned trust through back-to-back testing and verification. Standards such as ISO 26262, IEC 61508 and ASPICE assume that the verification process, not the source, produces the safety argument.

And the verification side is not a new problem waiting for a solution. The tools already exist, and they're mature:

• Proven in use. Requirements-based testing, back-to-back testing, structural coverage measurement including MC/DC, static analysis and formal verification have been used in series projects for many years, across automotive, aerospace, rail, medical and industrial automation.
• Highly automated. Test generation, test execution, coverage analysis and formal checks run without manual effort, in CI/CD pipelines, overnight and at scale. They keep up with the volume AI produces.
• Qualified for the safety standards. Many of these tools are certified or qualified for use under ISO 26262, IEC 61508 and related standards. Assessors know them and accept the evidence they produce.
• Deterministic. Same input, same result. Every verdict can be explained, reproduced and audited, which are exactly the properties AI output lacks.

So adding AI doesn't mean inventing a new way to build trust. It means connecting a new author to the verification chain the industry already relies on.

The emerging AI standards follow the same logic. ISO/IEC TR 5469 and ISO/PAS 8800 are published today. The upcoming ISO/IEC TS 22440, still under development and not yet a requirement, extends its scope to AI used to design, develop and verify safety-related functions. When AI enters development, the standards don't ask for a better model. They ask for verification which is deterministic.

04 · Why verification was slow, and why that's changing now

If proven, automated and certified verification tools already exist, why hasn't verification always been fast? Why did it still cost so much engineering time? There are two main reasons, and both are now changing.

Reason 1: weak requirements

Verification tools can only check against what has been written down. In practice, requirements were often vague, incomplete or missing altogether. So engineers had to fill the gaps by hand: work out what the system should actually do, write the test cases themselves and decide what the expected result of each test should be. The tools were automated, but feeding them wasn't.

What changes: when AI creates the implementation, the requirement becomes the only input into the process. No engineer is hand-building a model or writing code anymore, silently filling in the missing details straight from their head along the way. Requirements have to become precise anyway, or the AI can't generate the right implementation. And once they're precise, the same requirements also drive test generation and the verdicts automatically and lead to dramatically reduced manual effort.

This also answers the question of where the engineer's review time should really go:

• Not on the code or the model. An AI-generated implementation, whether C code or a Simulink model, is maybe structured differently from what the engineer would have built, so it takes extra effort just to follow it. More importantly, it's almost impossible to judge how it will actually behave just by reading it. Small mistakes, such as a ">" that should be a "<" or a limit that's off by one, slip through easily.
• Not on the tests. The AI produces many tests, so there's simply too much to read. And to judge whether a test expects the right result, you need to know exactly what the system should do.
• On the requirement. The requirement describes what the system must do. It stays the same size no matter how much implementation and how many tests the AI generates. And it captures the knowledge that only the engineer has. That's where human review pays off most.

Requirements are the new gold.

We can also map this to the term shift left. Human effort moves to the start of the V-model, onto the one artefact that affects everything downstream. The engineer reviews and signs the requirement once. From then on, every generated test, every part of the generated code or model and every coverage figure is checked automatically and deterministically against it.

Reason 2: every failure needed a human

The second time sink came at the other end. Every time a tool reported a failed test, a violated property or a coverage gap, an engineer had to step in: analyse the result, find the root cause, debug and fix the implementation, then run everything again. The checks were automated, but reacting to them was not.

What changes: in an AI-driven process, a failed check no longer has to trigger a human. The failure, with its precise reason, is fed straight back into the AI that creates the implementation. The AI fixes the problem, and the checks run again. This agentic loop works like a self-healing system: it repeats until all quality gates are passed. The engineer only steps in when the loop doesn't converge.

Closing the trust gap

With precise requirements at the start and an automated fix loop at the end, the two biggest manual efforts in verification shrink dramatically. Nobody has to read generated code, models and tests line by line, and nobody has to debug every failed check. Review effort drops sharply, and trust can grow at the speed of generation.

05 · How a gold-standard requirement is made

To get from bad to good to great requirements, we propose an AI-driven process running from the initial source intent to a signed baseline step by step, with AI and deterministic analysis working hand in hand. The building blocks of this framework are:

1. Source intent: the need, often a vague or unclear requirement.
2. Requirements: clear and verifiable; the What.
3. Formal Requirement: machine readable, with clear semantics and syntax and clear mapping to the system and/or software architecture.
4. Rule checks: AI-based and also deterministic.
5. Sign-off: the Golden Baseline.
6. Software Detailed Design: the How.

The rules guiding this process don't need to be invented from scratch. They can be derived from established standards such as ISO 26262, the INCOSE Guide to Writing Requirements and ISO/IEC/IEEE 29148.

Inside the AI framework, three AI agents with dedicated roles work together. An Author writes, a Reviewer checks against the rules, and a Guardian keeps the process in scope. The engineer answers the questions the AI may not decide on its own.

Here's what that could look like for one requirement of an adaptive cruise control (ACC):

• Source intent: "When the system gets activated, the current speed of the car is acquired as target speed."
• Requirement (the What): When the ACC system transitions from ACC Inactive to ACC Active, the ACC system shall acquire the current vehicle speed at the moment of activation as the target speed.
• Formal requirement: written as a BTC Universal Pattern (see below).
• Software Detailed Design (the How): On the execution cycle in which accState transitions from INACTIVE to ACTIVE, SE-02 shall set targetSpeed to the value of currentVehicleSpeed sampled in that cycle.

The results serve two purposes:

• The Software Detailed Design drives the implementation. It's precise enough that the AI can create the implementation directly from it, without guessing and without further input, either as production code or as a Simulink model.
• The formal requirement drives the verification. Because it's machine-readable and unambiguous, it's the basis for several different verification methods, such as requirements-based testing, robustness checks, coverage analysis and formal proof. The next chapter describes how they work together.

06 · How the implementation earns trust: generate, check, fix

Even with the Golden Baseline and a precise Software Detailed Design in place, an AI-generated implementation, whether code or model, is untrusted by definition. It earns trust by passing a deterministic quality gate.

AI can generate the implementation in two ways, and both work with the same approach:

• Production code directly. The AI writes C code from the Software Detailed Design. The code is verified against the formal requirement: requirements-based tests, robustness checks, structural coverage and formal proof.
• Simulink models. The AI creates a Simulink model instead, for teams that work model-based. The model is verified in the same way. The production code is then generated from the model with an established code generator, and back-to-back testing shows that model and code behave the same.

Which path fits depends on the team's existing process and toolchain. The principle stays the same: whatever the AI produces, the deterministic gate decides whether it can be trusted.

As described in chapter 04, a failure no longer lands on an engineer's desk. It goes back into the loop with a precise reason until all quality gates pass.

This changes what AI quality means. With a deterministic quality gate, AI quality is an efficiency question, not a safety question. A better model, better prompts or better skills mean fewer loops and a higher chance that the loop finishes at all. They don't decide whether the result can be trusted. The gate decides that. The guardrails and prompts from chapter one aren't wasted. Behind the gate, they become levers for speed and cost.

07 · Summary: past, present, future

AI is changing how safety-critical software is built. But the question of trust is not new, and neither is the answer.

Past: human + deterministic tools. For decades, safety-critical software was written by engineers and verified with proven, automated and certified tools. Trust was high, because every result was deterministic, reproducible and explainable. But effort was high too: engineers built the software by hand, and they also had to fill the gaps left by weak requirements and debug every failed check.

Present: human + AI. AI has changed the building part almost overnight. Code, Simulink models and tests are generated in minutes. But AI output can't be trusted on its own. Better models, AI checking AI, guardrails and prompts don't produce evidence. So the effort has moved to review, and human review doesn't scale with the speed of generation. Effort is now very high and trust is low. That's the trust gap many teams are facing today.

Future: human + AI + deterministic tools. The answer is to combine both worlds. AI does what it's good at: drafting requirements, writing code or Simulink models and generating tests at speed. Deterministic, qualified tools do what they've always done: verify every result against the requirement, with evidence an assessor accepts. Failures go back into an agentic loop until all quality gates pass. The engineer focuses on what only they can do: getting the requirement right and signing it off. Effort concentrates on the specification, and trust is high again.

Trust never came from the author of the code. It came from verification. That stays true when the author is an AI.