Using AI to Create Safety-Critical Engineering Models: What Works and What Doesn't

A common mistake in discussions about AI-assisted engineering is to focus exclusively on the quality of the generated output. Plausible output, however, does not prove correctness.

From a safety perspective, an AI system that creates or modifies engineering artifacts should be treated as a development tool. This means that the risks associated with the tool must be analyzed and reduced to an acceptable level.

This perspective aligns naturally with the rationale used in ISO 26262 for development tools, which require the determination of a Tool Confidence Level and Tool Qualification to reduce the risk of errors introduced by the tool or errors that the tool fails to detect in the product.

But can the same approach be applied to AI-based tools? The answer is no. AI-based tools introduce additional classes of potential errors beyond those found in conventional tools and therefore require extended qualification methods.

ISO 26262 distinguishes between tools whose malfunction cannot affect a safety-related work product and tools whose malfunction might do so.

For tools that can influence safety-relevant artifacts, confidence depends not only on the potential impact of a malfunction but also on the likelihood that such an error would be detected before it reaches the final work product. The resulting combination determines the required level of confidence and corresponding qualification measures.

Although AI tools are not explicitly addressed by current Tool Qualification approaches, the underlying principle remains highly relevant: confidence is established through a structured analysis of potential failures and corresponding detection or mitigation measures.

Modern AI-based workflows rarely consist of a standalone large language model. Instead, they typically involve an orchestration layer that interacts with external tools for activities such as:

• Reading files

• Querying repositories

• Transforming data

• Running analyses

• Creating or modifying models

The Model Context Protocol (MCP) provides a standardized mechanism for connecting these components. From a safety perspective, the protocol itself does not guarantee correctness. Its value lies in making interactions explicit and therefore analyzable.

This architectural decomposition allows us to identify potential error sources separately rather than treating the entire AI system as a single monolithic component.

A practical error analysis begins with a simple observation: an AI agent can only be as reliable as the components involved in producing its output.

Potential failures originate both from the AI itself and from the tools it uses. As a result, the overall error model must consider:

• Errors produced by the AI system

• Errors produced by connected tools

• Errors introduced during data transformations

• Errors occurring at component interfaces

A real-world example illustrates this point. During one experiment, a file-reading MCP server transformed a character sequence incorrectly. A second AI-based tool subsequently processed the corrupted data and produced an incorrect result. At first glance, the failure appeared to be an AI error, but the root cause was actually a software defect in an auxiliary tool.

This distinction matters because qualified conventional tools can significantly reduce the burden placed on later verification activities.

Many AI-generated artifacts can fail in familiar ways. For example, a contract-generation agent deriving formal contracts from requirements may:

• Omit requirements

• Create incorrect contract clauses

• Generate preconditions that are too restrictive or too permissive

• Produce postconditions that are either unimplementable or insufficient

Similarly, an interface-generation agent may introduce incorrect types, missing parameters, inconsistent naming, or redundant interfaces.

These are conventional engineering errors. AI, however, introduces additional categories of risk.

Compared to traditional development tools, AI systems introduce challenges such as:

• Non-deterministic behavior between executions

• Probabilistic and uncertain outputs

• Unnecessary or excessively complex results

• Actions beyond the requested task

• Disclosure of sensitive engineering information to external services

These characteristics require additional safeguards that are typically unnecessary for conventional deterministic tooling.

Many organizations assume that human review alone provides sufficient protection against AI errors. But review effectiveness depends on several factors:

• Reviewer competence

• Workload

• Independence

• Availability of clear review criteria

• Detectability of the underlying error

As AI-generated content volumes increase, reviewers may face hundreds of changes, findings, or generated artifacts. Under such circumstances, relying solely on manual review becomes increasingly problematic.

The most promising results emerge when AI output is combined with established engineering verification techniques. In ISO 26262-compliant development processes, requirements can be transformed into implementation interfaces and formal contracts, for example using ACSL.

Potential generation errors can then be addressed through:

• Traceability analysis

• Syntax checking

• Semantic analysis

• Consistency verification

• Formal verification tools such as Frama-C

These mechanisms provide objective evidence that generated contracts satisfy essential correctness properties.

Another example combines static analysis with AI-generated code corrections. In one case, AI generated a fix for a null-pointer defect identified in an open-source project. The primary question was whether the proposed patch changed any behavior beyond eliminating the reported defect.

Formal verification techniques were then used to analyze the generated solution and prove its correctness.

When the software is not overly complex, formal methods can fully prove its correctness. Restricting human review to cases that cannot be formally proven reduces the number of findings requiring human attention.

A particularly interesting experiment involved the creation of an ISO 26262-oriented software development process model. The model was generated using a domain-specific Process Modeling Tool (PMT) employed by Validas to define safety- and security-compliant qualification processes.

The resulting model contained:

• Hierarchical process structures

• Input/output artifact relationships

• Detailed development and verification activities

• Compliance-relevant documentation structures

Importantly, the model was not generated in a single step. Instead, the process relied on a structured workflow using specialized Excel-based interfaces. These interfaces abstracted away low-level implementation details and presented the information in a format that could be processed effectively by Microsoft Copilot.

The resulting model quality was surprisingly high. Consistency checks showed that artifacts were generally linked to creating and consuming activities throughout the process structure.

The strongest lesson from these experiments is straightforward:

• AI requires safeguards: This may sound obvious, but practical experience repeatedly demonstrated that plausible-looking outputs can still contain substantial errors.

• Structured interfaces matter: Direct generation of domain-specific PMT files delivered poor results. Once the problem was represented through structured Excel tables instead of complex XMI-based model files, model generation quality improved significantly. Abstraction turned out to be a key enabler for reliable AI-assisted modeling.

• Large compliance documents remain challenging: Attempts to formalize ISO 26262 requirements directly encountered practical limitations. In some cases, copyright restrictions prevented processing. In others, context-window limitations reduced quality when large document sections were provided. Manual review remained essential and revealed multiple issues that would otherwise have gone unnoticed.

• Test case generation needs more maturity: Generating test specifications initially appeared successful. The resulting artifacts were often well written and technically plausible. Detailed reviews, however, uncovered omissions, redundancies, and structural weaknesses. Even after prompt refinement, newly generated specifications often differed substantially from previous versions, requiring another full review cycle.

The conclusion was that current AI systems do not yet produce sufficiently reliable test specifications through direct prompting alone. More structured, model-based approaches appear necessary.

To address these challenges, Validas is currently developing a dedicated meta-process for AI-supported compliance model creation. The approach explicitly distinguishes between:

• Conventional engineering activities

• Verification activities

• AI-supported activities

A key principle of this approach is that every AI-generated artifact undergoes a dedicated verification step before being incorporated into the resulting model. This ensures traceability not only of the generated artifacts themselves, but also of the engineering rationale behind their acceptance.

AI can make a meaningful contribution to the creation of domain-specific, safety-critical engineering models, but only when it is integrated into a controlled engineering process.

Confidence should not be derived from persuasive outputs or unspecified human supervision. Instead, AI systems should be analyzed as composite development tools, comprising orchestration logic, connected tools, interfaces, and transformations.

The practical experience presented here suggests that the most reliable results stem from:

• The use of formal methods wherever possible

• Model-based development with structured interfaces

• Incremental generation workflows

• Consistency checks

• A documented meta-process for managing AI usage

Our experience at Validas shows that AI does not eliminate the need for engineering discipline. If anything, it reinforces it. The same principles used to qualify tools, verify results, and demonstrate compliance must also be applied to AI-based workflows, albeit with a higher level of rigor.

• ISO 26262, Road vehicles – Functional safety, applicable project edition, Part 8: Supporting processes.

• Model Context Protocol, protocol specification, see https://modelcontextprotocol.io

• ISO/IEC, ISO/IEC CD TS 22440-1, Artificial intelligence – Functional safety and AI systems – Part 1: Requirements, document under development.

• G. E. P. Box, “Science and Statistics,” Journal of the American Statistical Association, vol. 71, no. 356, pp. 791–799, 1976.

• ACSL, ANSI/ISO C Specification Language, see https://frama-c.com/html/acsl.html

• Frama-C: A platform to make your C code safer and more secure, see https://frama-c.com/

• MCP-Bug report “Markdown Preview render fails for table containing pipe | inside backticks”, see https://github.com/microsoft/vscode/issues/131859

• Static Analysis and AI, “How To Safely Use AI-Based Tools in Embedded Development”, Webinar recording, https://www.perforce.com/resources/events/webinars/how-safely-use-ai-based-tools

• Prof. Yoshua Bengio, speech before the UN on September 2026, see
  https://www.linkedin.com/feed/update/urn:li:activity:7508619166153875456/