Tool Qualification for High Risk AI Applications: A Complete Guide to Building Safe, Compliant AI

Modern AI pipelines shift risk from hand‑written code to data, models, and the tools that transform them. A single tool defect (in data preparation, training, or evaluation) can invisibly corrupt downstream artifacts and surface later in production. Real‑world experience in safety programs has shown that poor process discipline or tool failures can directly cause system failures.

Thus, as organizations shift from traditional software development to AI‑driven services, tools used to generate, test, and train AI models carry increasing influence and potential risk, making tool qualification a first‑order safety activity rather than an afterthought.

High‑risk AI applications must follow strict process rules, including documentation, monitoring, and checks at various stages of development. High‑risk AI is governed by a growing mesh of laws and standards as follows:

• EU AI Act: Classifies AI by risk tier (minimal, limited, high, unacceptable). High‑risk systems require QMS‑backed processes, risk management, documentation, and post‑market monitoring. Unacceptable‑risk use cases are prohibited.

• Functional safety & AI standards referencing ISO 26262 clause 11 (Tool Confidence):

  • ISO/TR 5469 (Functional Safety & AI Systems) – Introduces usage levels and technology classes to reason about AI safety in real systems.

  • ISO 21448 (SOTIF) – Focuses on unknowns and continuous improvement; describes an AI training and verification flow and refers to tool confidence for AI tooling.

  • ISO/PAS 8800 – Highlights data tools, data properties, and mitigations; again aligns assurance with tool confidence.

  • VDE‑AR 2842‑61 – Frames trustworthiness for autonomous/cognitive systems, lists AI methods and metrics, and references ISO 26262 for tool safety.

No matter the vertical, tool safety per ISO 26262 8‑11 is the common thread—making it the practical entry point to de‑risk pipelines for building AI application.

ISO/TR 5469 offers a practical lens to analyze AI safety in physical, real‑world systems where AI decisions can have direct safety impact. It combines Usage Levels with Technology Classes to guide risk reduction measures.

Usage Levels (A–D)

• A1 – AI in the system with automated decision‑making: Highest safety relevance; AI actively commands actions (e.g., a robot deciding to move).

• A2 – AI in the system without automated decision‑making: AI aids perception/analysis; a classical algorithm makes the final decision.

• B1 – AI in development with automated decision‑making: AI influences artifacts like code/tests that enter the product.

• B2 – AI in development without automated decision‑making: AI assists engineers (e.g., diagnostics), with one level less criticality than B1.

• C – Indirect safety impact: AI influences safety indirectly (e.g., testing, risk reduction tools).

• D – No safety impact: Isolated or sandboxed AI with no effect on safety functions.

AI embedded in moving, physical systems is more critical than AI confined to back‑office analysis, and automated decision‑making elevates risk compared to purely assistive AI.

If we want to use AI and AI Agents as “tools” to develop and verify systems, they fall into classes B1 and B2. Even better, existing safety standards like ISO 26262 and IEC 61508 can be applied to acquire the required “tool confidence” into these development tools.

SOTIF emphasizes that AI systems will face unknown scenarios and the systems must be designed to update and improve as those are discovered.

The typical pipeline includes annotated data → received by a training tool→ which creates parametrized outputs → self‑verification and cross‑verfication to validate AI behavior, with explicit reference to using ISO 26262 tool confidence for the training and verification tooling, thus ensuring that the tools used to develop or test AI systems are proven trustworthy and reliable.

SOTIF ensures that unknown risks are identified, tested, and mitigated, especially in complex AI‑driven systems.

ISO/PAS 8800 is a recently released standard specifically targeting the safety of AI in road vehicles.

ISO/PAS 8800 brings a data‑centric lens, highlighting properties such as accuracy, integrity, representativeness, traceability, verifiability, and completeness - all of which can be turned into potential error classes for toolchain risk analysis and qualification. The approach enables preliminary classification and identifies mitigations that reduce tool criticality across the data pipeline.

VDE‑AR 2842‑61 is one of the most comprehensive frameworks available today for evaluating the safety and trustworthiness of AI systems. It is often considered one of the most advanced standards for AI safety.

 This specification starts from the solution layer, the real‑world context and operating domain, and provides in the technical part a catalog of AI methods and metrics while referring back to ISO 26262 for tool confidence. It asks teams to identify uncertainty‑driven AI risks and reduce them with appropriate metrics and methods, aligning system assumptions with field validation.

ISO 26262 (Part 8, Clause 11) operationalizes tool safety via:

i) Tool Impact (TI) - can the tool introduce or fail to detect errors?

ii) and Tool Error Detection (TD) - how likely are you to catch a potential tool error elsewhere?

These map to Tool Confidence Levels (TCL1–TCL3), which then determine whether and how to qualify the tool.

ISO 26262 defines a two‑step logic to arrive at a Tool Confidence Level (TCL):

1. Tool Impact (TI): Can the tool introduce a potential error into the product or fail to detect one?

2. Tool Error Detection (TD): How probable is it that a potential tool error would be caught? (High, Medium, Low/Unknown)

These map to TCL1–TCL3, where:

• TCL1: No qualification needed, but you shouldn’t rely on the tool’s correctness; you must verify everything it produces manually.   

• TCL2/TCL3: Tool must be qualified. TCL3 is typically preferred when you want to trust and rely on the tool (with robust evidence, where manual review is expensive or impossible).

Counter‑intuitive but important: TCL1 is not “better”. It just means “don’t trust the tool; check correctness of outputs.” When detection is costly (e.g., manual code reviews), TCL1 can be more expensive than qualifying the tool.

Delivering safe, reliable, and compliant AI systems requires a structured approach. Our framework is built on three core pillars, each essential for achieving trustworthiness in AI‑driven solutions.

1. Process (and Data) Safety
   Establish a standards‑aligned process that covers data selection, annotation, training, and updates. For SOTIF, plan for unknowns and define update mechanisms to improve coverage over time.

2. Verification & Validation (V&V) Methods
   Use AI‑specific metrics and analyses (e.g., accuracy, precision/recall, mAP, IoU/miou, PSNR, temporal consistency, adversarial tests, saliency maps, neuron coverage) to probe behavior realistically.

3. Tool Confidence
   Classify the toolchain, then qualify the tools that matter. In many AI setups, training tools can be TCL1 (because downstream verification will catch misbehavior), while data and verification tools should be TCL3; qualify the tools and checkers you rely on.

Achieving safe and compliant AI development requires a clear and traceable connection between requirements, process steps, verification activities, and supporting documentation. The Process Modelling Tool (PMT) enables exactly this by providing a structured, model‑based approach to process compliance.

Using Validas’ rigorous model‑based compliance method, PMT formalizes each safety standard by mapping its requirements into a structured requirements tree, linking them to concrete process steps, defining verification and validation acceptance questions, and tying everything to objective evidence such as reports, logs, and qualification records. This detailed and explicit mapping not only streamlines external assessments but has repeatedly revealed defects in open‑source tools, demonstrating how strong process discipline enhances actual safety rather than just producing paperwork. With these capabilities, PMT helps organizations build a transparent, audit‑ready, and highly reliable foundation for safe AI development.

Validas can provide you training and support to model your own processes using PMT.

Ensuring trustworthy AI requires verification and validation methods that go far beyond traditional software testing. AI systems learn from data and perform complex, dynamic tasks, which demand domain‑specific V&V techniques capable of probing behavior from multiple angles. Commonly used approaches include accuracy, precision/recall, confusion matrices, Intersection over Union (IoU) and mean IoU, average precision and mean average precision, model convergence checks, Peak Signal‑to‑Noise Ratio (PSNR), and temporal consistency metrics. More advanced techniques such as adversarial testing, saliency maps, and neuron coverage help reveal how models respond to challenging inputs, highlight decision‑making patterns, and expose vulnerabilities or blind spots. Together, these methods validate that learned behavior aligns with expected, safe performance across intended scenarios, strengthening confidence in AI systems deployed in real‑world environments.

A practical SOTIF example illustrates how AI tools can be classified based on their impact and detectability. Training tools directly influence the product because their output parameters flow into the model, but mistakes in training are usually caught through internal safeguards such as cross‑validation and behavioral verification. For this reason, training tools often fall into TCL1, provided robust, independent checks exist.

In contrast, verification tools determine whether an AI model is “good enough” for deployment. Errors in these tools can mask real defects in the model, making them far more safety‑critical. As a result, verification tools typically require TCL3 classification and full qualification.

When dealing with data tools, ISO/PAS 8800 provides useful properties to identify potential data‑related errors and define appropriate mitigations. This accelerates preliminary toolchain classification and helps organizations understand where qualification delivers the greatest safety value.

ISO/PAS 8800 places data at the center of AI toolchain safety. Building on this, we classify data tools—such as annotation, aggregation, preprocessing, and quality checkers—by first identifying relevant data properties (e.g., accuracy, integrity, representativeness, traceability, verifiability, completeness). Each property translates into potential failure modes when it’s missing or mishandled, which in turn informs mitigations and the preliminary toolchain classification. This structured mapping makes it faster to see where risks concentrate and where tool qualification adds the most value—especially for the verification steps that gate model release. In practice, we model the toolchain at a fine granularity (tools, features, use cases), enumerate the property‑driven error scenarios, and connect them to concrete mitigations, producing a pre‑classified toolchain model that accelerates safety analysis and qualification planning.

To trust AI verification tools, the metrics they compute must be proven correct—especially when those metrics determine whether a model is accepted or rejected. The accuracy metric illustrates this qualification strategy well: by defining accuracy in terms of TP(True Positive), TN(True Negative), FP(False Positive), and FN(False Negative), one can systematically construct equivalence classes and edge‑case scenarios, including empty or zero‑count sets, to validate that the accuracy calculation is always correct. This black‑box testing pattern ensures the metric behaves reliably under all conditions. The same approach generalizes to other performance measures such as precision, recall, IoU, mIoU, mAP, PSNR, and more, providing durable, numerical evidence that your verification tools compute the right values—and therefore can be trusted in safety‑critical decisions.