diff --git a/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-guide.md b/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-guide.md new file mode 100644 index 0000000..fd7e6f3 --- /dev/null +++ b/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-guide.md @@ -0,0 +1,510 @@ +# Guide to LLVM-QUAL-TPL-004 — Tool Qualification Report + +> [!NOTE] +> **Guidance provided by:** [LLVM Qualification Group](https://llvm.org/docs/QualGroup.html) +> **Related template:** `LLVM-QUAL-TPL-004` +> This guide set is informative. It does not replace the applicable safety or +> assurance standard, regulatory or certification guidance, a documented +> interpretation, or an organization's own processes. + +## Purpose and boundary of TPL-004 + +LLVM-QUAL-TPL-004 records the execution and results of qualification activities +defined in one Tool Usage Plan. It integrates the resulting evidence, findings, +limitations, and conditions into a bounded, standard-specific qualification +conclusion. + +TPL-004 is not another qualification plan. It should not repeat: + +- the intended or given tool usage and usage boundary; +- the selected qualification approaches and rationale; +- planned objectives, responsibilities, independence, or work products; +- planned procedures, coverage strategy, or acceptance criteria; +- the classification and malfunction analysis; or +- the planned change-management strategy. + +Those items remain in TPL-002, TPL-003, and any plans or procedures referenced +by them. TPL-004 records what actually happened and whether the defined +objectives and criteria were satisfied. + +The report is not a substitute for the underlying evidence. It summarizes, +references, and integrates controlled requirements, tests, execution records, +process assessments, service-history analyses, certificates, findings, reviews, +and other evidence without copying every artifact into one document. + +The report does not establish universal suitability, certify the tool, or +demonstrate compliance with an entire standard. It supports a conclusion only +for the defined boundary. Any additional review, assessment, approval, or +certification process depends on the applicable standard and the needs of the +organization or team using the template. + +## Guide-set structure + +TPL-004 guidance is divided into three files: + +1. **This central guide** explains the workflow boundary, use of the Tool Usage + Plan, completion of the streamlined template, evidence integration, + findings, conclusions, reuse, maintenance, and review checklist. +2. The [qualification-approaches companion](LLVM-QUAL-TPL-004-qualification-approaches-guide.md) + explains validation, service-history analysis, development-process + evaluation, development under a safety or assurance standard, output + verification and diversity, certification, and evidence reuse. +3. The [standards and test-coverage companion](LLVM-QUAL-TPL-004-standards-guide.md) + explains what qualification or tool confidence means under IEC 61508, + EN 50716, ISO 26262, IEC 62304, and DO-178C/DO-330; compares their approaches; + and explains requirements/behavior and structural-coverage expectations. + +Start with this guide. Consult only the companion sections relevant to the +applicable standard and selected qualification activities. + +## Following the Tool Usage Plan and handling changes + +### Default: follow the plan + +One TPL-004 report is governed by one referenced TPL-002 revision. That revision +defines the tool version and usage boundary addressed by the report. If another +TPL-002 applies to a different tool version or usage boundary, prepare a +separate TPL-004 report. TPL-004 should also identify the corresponding TPL-003 +revision. Use references precise enough to identify the versions on which the +report is based; a changing branch or undated web page may not be sufficient. + +Hence, the streamlined template assumes that the qualification activities conform +to the single referenced TPL-002 revision. It therefore uses references instead of +repeating planned content. + +Before recording the results, check that the following remain consistent with +that plan: + +- tool usage and scope; +- qualification approaches and standard-specific basis; +- objectives and acceptance criteria; +- responsibilities and independence arrangements; +- procedures, coverage strategies, and evaluation criteria; +- assumptions, constraints, and required controls; and +- planned evidence and work products. + +If all remain unchanged, proceed directly to recording the execution and +results. + +### When the plan needs to change + +If the team decides that a planned element needs to change, update TPL-002 +before continuing the affected qualification activity or establishing the +qualification conclusion. Review or approval of that update can follow the +organization's or team's normal process. + +- Update TPL-002 when the usage, approach, scope, objective, responsibility, + procedure, evidence, or acceptance strategy changes. +- Update TPL-003 when the classification boundary or result, malfunction + analysis, confidence in controls, required controls, or resulting + qualification need changes. + +TPL-004 then references the updated revision and, where available, the +associated change record. It does not restate the revised plan. + +### When execution differs from the plan + +An unexpected execution deviation is something that occurred while attempting +to perform the planned activity, for example: + +- a required test could not be executed; +- the actual environment differed from the planned environment; +- an assessment sample or observation period was incomplete; +- an expected independent reviewer was unavailable; +- a procedure step was omitted or performed differently; or +- an evidence source was unavailable or had changed. + +Do not rewrite the plan retroactively to make the execution appear conformant. +Record the deviation and its impact in the activity record. Then either: + +1. correct the condition and re-execute the affected work according to the + plan; or +2. update the plan and determine which work needs to be repeated or extended. + +A qualification conclusion should not rely on an unresolved contradiction +between the plan and the recorded execution. + +## Meaning and limits of qualification + +Software-tool qualification is the structured application of the approaches +and evidence required by the applicable standard to justify confidence or +obtain credit for a defined tool usage and assurance context. + +“Tool X is qualified” is normally too broad. A defensible conclusion is bounded +by the defined usage, tool and evidence baseline, functions and outputs, +environment, integrity or software level, assumptions, restrictions, and +mandatory controls. + +The standards do not all use the same qualification concept. In particular, +IEC 62304 (medical device software) does not define a comparable development-tool +qualification framework, while DO-178C/DO-330 (airborne systems) focuses on +project-specific certification credit and a TQL-dependent objective set. +Consult the [standard-by-standard explanations](LLVM-QUAL-TPL-004-standards-guide.md#meaning-of-tool-qualification-or-confidence-under-each-standard) +before choosing report terminology. + +### Qualification is not certification (and vice-versa) + +Certification is third-party attestation under a defined conformity-assessment +scheme. Its underlying assessment can include process audit, product validation, +lifecycle-compliance review, service history, or a combination. + +The certificate's qualification value depends on what was assessed, what the +attestation claims, and whether its scope matches the current use. +IEC 61508 (E/E/PE systems) gives certified tools and translators an explicit +technique/measure role, but this does not make every certificate sufficient +for every use. + +See the detailed chapter on +[certification and its relationship with qualification](LLVM-QUAL-TPL-004-qualification-approaches-guide.md#certification-and-its-relationship-with-qualification). + +### Qualification is not ordinary product testing + +Regression or conformance testing can contribute important evidence, but a +qualification report also asks whether: + +- expected behaviors relevant to the qualified use were defined; +- test scope is traceable to the defined objectives and TPL-003 concerns; +- expected results and test oracles are trustworthy; +- the actual environment is representative of the planned environment; +- invalid, unusual, boundary, and other error-related conditions are addressed; +- failures, skipped tests, discrepancies, and limitations are retained and + dispositioned; and +- the execution and evidence baseline can be reproduced and reviewed. + +Structural coverage of the tool's code is not a universal tool-validation +requirement. Requirements or behavior coverage also does not by itself +demonstrate adequate validation. See the +[test-coverage chapter](LLVM-QUAL-TPL-004-standards-guide.md#test-coverage-expectations-for-software-tools). + +### Standards terminology in plain language + +Safety and assurance standards sometimes use **anomaly** as a broad term for a +condition, behavior, result, or documentation item that differs from what is +expected. Software developers and maintainers may describe the same thing more +specifically as a bug, regression, crash, incorrect output, missing diagnostic, +test failure, known issue, or limitation. + +An anomaly can be an observation that still needs investigation. It does not +automatically mean that a defect or its root cause has already been confirmed. + +This guide uses the most specific familiar term available. It retains +**anomaly** when explaining a standard or when the nature of an observed issue +has not yet been determined. In particular: + +- a **known anomaly** can usually be read as a known issue, defect, unexpected + behavior, or documented limitation; and +- **anomaly handling** normally means recording, investigating, tracking, and + resolving or otherwise addressing such issues. + +**Anomalous operating conditions** has a different meaning. It refers to +conditions outside the ordinary successful-use path, not necessarily to a bug +in the tool. Examples include malformed or invalid input, boundary values, +unsupported options, missing or corrupted files, unavailable dependencies, +resource exhaustion, interrupted external services, and unexpected +environmental conditions. + +The standards companion may retain _anomaly_ or _anomalous operating conditions_ +where that wording helps preserve the connection to a particular standard. + +## Completing the template + +### Identification and inputs + +Identify the single TPL-002 revision governing the report and the corresponding +TPL-003 revision. Use a revision, commit, release, date, or other reference that +lets a reader find the intended document rather than only naming a changing +branch. + +The tool version is optional in TPL-004 because TPL-002 is the main source for +the tool identity and baseline. Repeating it in TPL-004 can still be convenient +for readers, provided the information remains consistent with TPL-002. + +Use **Qualification perspective** to explain who is preparing the report, such +as an upstream project, tool provider, distributor, integrator, or +project-specific user. This helps readers understand which evidence and +conclusions are within that contributor's knowledge and control. + +The role table is intentionally flexible. List authors, reviewers, approvers, +or other roles only where they are useful. TPL-004 does not require document +status, approval, or sign-off fields; an organization or team can add those +fields according to its own process and the applicable standard. + +### Qualification activity summary + +For each activity: + +- use the activity name from the plan; +- reference the planned activity or applicable TPL-002 section; and +- state its current status or final result. + +If TPL-002 assigns unique identifiers to qualification activities, preserve +them. Otherwise assign a report identifier such as `QMA-01` and reference the +corresponding TPL-002 section unambiguously. + +### Evidence inventory and applicability + +Assign a stable `QE` identifier to every evidence item used by the report. +Identify an immutable version, revision, commit, report date, execution ID, or +equivalent controlled baseline. + +The applicability field is a documented gap analysis. Compare the evidence +context with the current qualification boundary, including: + +- exact tool or component identity; +- covered functions, outputs, objectives, and malfunction concerns; +- host, target, dependencies, configuration, and execution environment; +- use cases, inputs, options, output reliance, and downstream controls; +- applicable integrity or software level and TQL where relevant; +- assumptions, restrictions, known issues, and validity conditions; and +- evidence provenance, review, maintenance, and availability. + +Classify reused evidence as fully applicable, partially applicable with stated +gaps, or not applicable. A partial match can support a bounded portion of the +argument, but it requires additional activity, restriction, or delta +qualification for the uncovered part. + +### Coverage and findings + +Trace the objectives and TPL-003 concerns to the activities and +evidence that address them. Record whether each is covered, partially covered, +not covered, or not applicable, with a rationale and residual action. + +Review cross-activity dependencies and common-cause weaknesses. Several +activities can appear independent while relying on the same incomplete +requirement, incorrect oracle, shared parser, generated harness, infrastructure, +data source, or reviewer assumption. + +Record adverse evidence, including: + +- unexpected tool behavior and incorrect outputs; +- test or expected-result discrepancies; +- process or review findings; +- service-history data gaps; +- certificate or evidence-reuse mismatches; +- execution deviations; and +- unresolved qualification questions. + +Use a `QF` identifier when it is useful to link the same issue or finding from +an activity, the coverage summary, and the qualification conclusion. +`QF` means **Qualification Finding** and is an identifier provided by +this template, not terminology required by the standards. + +For an open finding, record enough information to understand its impact, +rationale, planned follow-up or compensating measure, current status, and +supporting evidence. +Add an owner or decision authority where the team's process needs one. + +### Standard-specific qualification conclusions + +Complete a separate conclusion for each applicable standard. Shared evidence +does not make different standards or qualification levels equivalent. + +Each conclusion should identify or reference: + +- completed activities and principal evidence; +- the requirements or interpretation mapping used for completeness; +- the conclusion and its rationale; +- mandatory conditions, limitations, exclusions, and unresolved items. + +The conclusion remains bounded by the TPL-002 and TPL-003 references and by the +validity statement in the report. If the applicable standard, certification +context, organization, or team requires confirmation, independent assessment, +approval, or authority coordination, record that information in the evidence +inventory or in additional fields chosen by the template user. + +For DO-178C/DO-330, distinguish the certification credit claimed from the +applicable DO-330 qualification objectives. A vendor certificate or existing +qualification package does not by itself grant project certification credit. + +### Detailed qualification activity results + +Copy one complete result chapter for each qualification activity. Reference the +TPL-002 content and record only execution-specific information: + +- actual performers, reviewers, independence, and dates; +- the execution or assessment environment; +- execution deviations and their disposition; +- results, achieved coverage, gaps, discrepancies, and findings; +- applicability of reused evidence or third-party assessment; +- detailed evidence references; +- assessment against the acceptance criteria; and +- limitations, dependent controls, and follow-up. + +The activity result is not automatically the overall qualification conclusion. +An activity can pass while another required activity fails, an objective remains +uncovered, or an assumption is not satisfied. + +The [qualification-approaches companion](LLVM-QUAL-TPL-004-qualification-approaches-guide.md) +provides activity-specific interpretation and review questions. + +### Validity and user information + +Reference the change and re-evaluation conditions defined in the plan. +Record only additional triggers or limitations discovered through the +qualification results. + +Transfer user-relevant conditions into a safety manual such as TPL-005, or into +other suitable user documentation. This can include supported baselines, +mandatory configuration checks, required downstream verification, known +issues, workarounds, unsupported uses, and conditions invalidating the +evidence. + +## Qualification perspectives and evidence layering + +An upstream project can provide reusable source identity, documentation, +behavior descriptions, regression tests, CI definitions and retained results, +contribution and release practices, issue records, and bounded generic analyses. +It often cannot know the final target, options, integrity level, output reliance, +integration environment, or downstream controls. + +A tool provider or distributor can stabilize source and binary baselines, add +controlled builds and documentation, execute validation on defined platforms, +assess processes, and maintain a qualification support package. Patches, +backports, configuration, packaging, and rebuilding can create differences from +upstream evidence that require analysis. + +An integrator or project-specific user normally confirms the actual +installation, use cases, target, dependencies, integrity level, output reliance, +downstream verification, and any organizational review or sign-off needs. + +A qualification service or certification body can perform validation, process +assessment, or certification. Where relevant, record its role, independence, +scheme or criteria, scope, evidence access, result, and limitations. + +A scalable pattern is a generic upstream or provider evidence package plus a +downstream applicability addendum. The addendum preserves every assumption and +restriction of the reusable package and closes the project-specific gaps. + +## Change impact and delta qualification + +Qualification evidence is configuration-dependent. When a tool, usage, +environment, control, standard, or evidence package changes, perform a change +impact analysis before reusing the previous conclusion. + +Classify evidence and activities as: + +- unaffected and reusable with rationale; +- affected but reusable after confirmation; +- requiring partial repetition or extension; +- invalid for the changed boundary; or +- requiring investigation before disposition. + +Delta qualification is not merely rerunning the most convenient tests. +It is a reasoned selection of affected requirements, functions, configurations, +malfunction concerns, evidence sources, activities, and user conditions. + +## Illustrative compiler qualification structure + +For a bounded compiler usage, the plan might select validation and +independent output verification. TPL-004 would then record, by reference rather +than restating the strategy: + +1. the defined tool usage, such as language subset, compiler options, target, + environment, and output reliance; +2. the corresponding TPL-003 code-generation and diagnostic concerns; +3. the exact compiler build, validation suite, harness, target, and execution + campaign actually used; +4. validation results, requirements/behavior and problem-space coverage, + skipped or failed cases, and observed issues; +5. execution and results of the independent output-verification activity; +6. applicability of upstream tests, certificates, prior reports, or other + reused evidence; +7. integrated coverage of the defined objectives and TPL-003 concerns; and +8. a bounded conclusion with required options, checks, restrictions, known + issues, and change triggers. + +## Review and completeness checklist + +Before considering the report complete, check that: + +- [ ] the single TPL-002 revision and corresponding TPL-003 revision are + identified; +- [ ] the recorded activities and results are consistent with TPL-002; +- [ ] plan changes were made before the affected activity continued or the + conclusion was established, and were reviewed or approved if the team's + process requires it; +- [ ] unexpected execution deviations remain visible and are resolved or + dispositioned; +- [ ] every qualification activity has an execution and result record; +- [ ] the evidence and execution context are identifiable and reviewable; +- [ ] reused evidence has a documented applicability assessment; +- [ ] acceptance criteria were defined before interpreting results; +- [ ] achieved coverage, gaps, skipped work, and adverse evidence are visible; +- [ ] findings and common-cause dependencies are assessed; +- [ ] objectives and TPL-003 concerns are traced to activities, + evidence, and conclusions; +- [ ] conclusions are stated separately for each applicable standard; +- [ ] the conclusion does not exceed the boundary defined by TPL-002 and TPL-003; +- [ ] mandatory conditions and known issues are passed to a safety manual or + other suitable user documentation; +- [ ] any review, assessment, approval, sign-off, or authority coordination + required by the applicable context is complete; and +- [ ] report-specific validity limitations and additional re-evaluation + triggers are identified. + +## References to consult + +Use the exact editions and authority material applicable to the project. The +[standards companion](LLVM-QUAL-TPL-004-standards-guide.md#references-to-consult) +provides the complete standards-specific reference list in this order: + +1. IEC 61508; +2. EN 50716; +3. ISO 26262; +4. IEC 62304 and relevant medical-device QMS or regulatory guidance; and +5. DO-178C/DO-330 and applicable certification-authority guidance. + +For certification terminology and conformity assessment, also consult: + +- [ISO — Conformity assessment](https://www.iso.org/conformity-assessment.html); +- [ISO/CASCO — Techniques and schemes](https://casco.iso.org/techniques-and-schemes.html); + and +- [ISO/CASCO — Attestations of conformity](https://casco.iso.org/attestations-of-conformity.html). + +The applicable standards and documented interpretations remain the primary +references for classification, method or objective selection, evidence, +independence, lifecycle data, confirmation measures, assessment, and any +required approval. + +## Acronyms + +| Acronym | Description | +| --- | --- | +| ABI | Application Binary Interface | +| API | Application Programming Interface | +| ASIL | Automotive Safety Integrity Level | +| CI | Continuous Integration | +| CIA | Change Impact Analysis; in the DO-330 context, the applicable tool or operational-environment change-impact analysis | +| COTS | Commercial Off-The-Shelf | +| CUSWT | Confidence in the Usage of Software Tools | +| DO | RTCA document designation, as in DO-178C and DO-330 | +| E/E/PE | Electrical / Electronic / Programmable Electronic | +| EASA | European Union Aviation Safety Agency | +| ED | EUROCAE document designation, as in ED-12C and ED-215 | +| EN | European Standard | +| EUROCAE | European Organisation for Civil Aviation Equipment | +| FAA | United States Federal Aviation Administration | +| FAS | Forum for Aeronautical Software | +| FDA | United States Food and Drug Administration | +| FTP | FAS Topic Paper | +| IEC | International Electrotechnical Commission | +| ISO | International Organization for Standardization | +| MC/DC | Modified Condition/Decision Coverage | +| OSS | Open-Source Software | +| QE | Qualification Evidence identifier used by TPL-004 | +| QF | Qualification Finding identifier used by TPL-004 | +| QMA | Qualification Method or Activity identifier used by TPL-004 | +| QMS | Quality Management System | +| QMSR | FDA Quality Management System Regulation | +| RTCA | Radio Technical Commission for Aeronautics | +| SIL | Safety Integrity Level | +| SQA | Software Quality Assurance | +| SW | Software | +| T1/T2/T3 | Tool classification labels used by IEC 61508 and EN 50716; apply each standard's own definitions | +| TCL | Tool Confidence Level in ISO 26262 | +| TD | Tool Error Detection in ISO 26262 | +| TI | Tool Impact in ISO 26262 | +| TPL | Template identifier in the LLVM Qualification Group workflow | +| TQL | Tool Qualification Level in DO-178C/DO-330 | +| TOR | Tool Operational Requirements in DO-330 | +| UC | Use Case | diff --git a/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-qualification-approaches-guide.md b/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-qualification-approaches-guide.md new file mode 100644 index 0000000..61fc8ae --- /dev/null +++ b/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-qualification-approaches-guide.md @@ -0,0 +1,683 @@ +# Qualification-Approaches Companion to LLVM-QUAL-TPL-004 + +> [!NOTE] +> **Guidance provided by:** [LLVM Qualification Group](https://llvm.org/docs/QualGroup.html) +> **Related template:** `LLVM-QUAL-TPL-004` +> **Central guide:** [`LLVM-QUAL-TPL-004-guide.md`](LLVM-QUAL-TPL-004-guide.md) +> **Standards and test-coverage companion:** [`LLVM-QUAL-TPL-004-standards-guide.md`](LLVM-QUAL-TPL-004-standards-guide.md) +> This companion is informative. Apply the terminology, selection rules, rigor, +> work products, and any review or approval expectations required by the +> applicable standard and organization. + +## Purpose + +This companion provides practical guidance for defining qualification +activities in TPL-002, preparing any supporting procedures, performing and +reviewing the planned activities, and recording their execution and results in +TPL-004. + +It is not a transcription of method-selection tables or normative requirements. +Its purpose is to help contributors turn a selected approach into reviewable +evidence that is useful in an upstream open-source project, a downstream +distribution, a qualification support package, or a project-specific integration. +Selection, scope, objectives, responsibilities, procedures, and acceptance criteria +belong in the single applicable TPL-002. +TPL-004 references that plan and records what was actually executed. + +## Distinguish the kinds of qualification activity + +Standards organize tool confidence differently. Before defining an activity in +TPL-002, identify what kind of activity it is. TPL-004 should preserve the +standard-specific name and basis by reference: + +- A **qualification method** is a method named or selected under the applicable standard, + such as an ISO 26262 software-tool qualification method. +- A **standard-recognized technique or measure** is a technique or measure to which + a standard assigns an assurance role, such as certified tools and translators under IEC 61508. +- A **conformance-evidence route** demonstrates that relevant tool behavior conforms + to its specification or product documentation. +- An **effective control or mitigation route** prevents, detects, corrects, or contains + tool-induced failures when direct qualification or conformance evidence is not used + or is insufficient. +- An **objective-based qualification lifecycle** satisfies a prescribed set of objectives, + activities, independence expectations, and lifecycle data, as in DO-330. +- A **supporting confidence activity** strengthens the argument but does not independently + satisfy a standard-specific route. +- An **evidence source** provides artifacts that can support one or more activities. +- An **applicability assessment** determines whether existing evidence covers the current + tool baseline and usage. +- A **qualification conclusion** integrates the applicable activities, evidence, findings, + restrictions, and any required reviews. + +For example, an upstream regression suite is an evidence source. Selecting relevant tests, +mapping them to qualification requirements, executing them against the controlled tool baseline, +and assessing the results can form part of tool validation. +Linking to the suite is not itself tool validation. + +A certificate is both an attestation and a source of assessed evidence. Under IEC 61508 +it can also support a named technique or measure. Its qualification value still depends +on the scheme, underlying assessment, scope, and applicability. + +## Select and combine approaches + +Approach selection and its rationale should be established in TPL-002 before +the affected work is performed. Use this section when preparing or reviewing +TPL-002. In TPL-004, do not repeat the selection argument; reference the single +applicable TPL-002 revision and assess the execution and results. + +### Begin with the standard-specific need + +Do not begin with “we have these tests” or “the supplier has a certificate”. +Begin with: + +- the bounded qualification claim; +- the TPL-003 classification concern or malfunction; +- the applicable standard, integrity level, tool class, TCL, TQL, or other target; and +- the uncertainty or failure mechanism that the activity must address. + +Then determine whether the proposed activity and evidence address that need. + +### Avoid circular reasoning and double counting + +A control already used to establish the TPL-003 classification should not be +presented again as independent qualification evidence without explaining +the relationship. + +Also check for less visible common cause, for example: + +- the validation harness is generated by the tool being validated; +- the tool and checker share a parser, library, algorithm, model, or requirement error; +- service-history defects are detected using only the tool's own diagnostics; +- a process assessment relies only on self-authored policies; or +- a certificate and qualification kit rely on the same underlying tests, + process records, or assessment and are incorrectly counted as independent + evidence; or +- several activities rely on the same incomplete requirements or incorrect oracle. + +### Combine activities by contribution, not by count + +Two weak activities do not automatically form a strong argument. Explain which +uncertainty each activity addresses and which assumptions it shares with the others. + +A complementary combination might use: + +- process evidence to show systematic prevention and correction of development faults; +- focused validation to check the functions, configurations, and unusual or + error-related conditions relevant to the project; +- service-history evidence to examine realistic workloads and long-term defect experience; and +- independent output verification to detect specified residual malfunction classes in + the downstream workflow. + +The applicable standard determines whether that combination is acceptable. + +## Validation of the software tool + +Validation provides direct evidence that the tool behaves as required within +the qualified boundary. Its standard-specific role is explained in the +[standards companion](LLVM-QUAL-TPL-004-standards-guide.md). + +### Define the validation basis + +Validation requires an expected-behavior baseline. Depending on the tool, +it can include: + +- tool or qualification requirements; +- a product or user manual; +- a language, protocol, or file-format specification; +- command, API, diagnostic, or configuration semantics; +- use-case-specific behavior requirements; +- defined treatment of invalid input and unusual or error-related conditions; +- safety-relevant restrictions or prohibited uses. + +A manual is not automatically an adequate requirements specification. +Assess whether its statements are precise, complete, testable, versioned, +and relevant to the qualified functions. + +For a large general-purpose tool, derive a bounded set of qualification requirements +from the defined use cases while retaining traceability to authoritative specifications +and documentation. +Mark undefined, implementation-defined, unsupported, and deliberately unspecified behavior. + +### Design validation around the qualified use + +Cover the functions, feature interactions, inputs, configurations, targets, +output properties, and failure concerns relevant to the claim. +Useful techniques can include: + +- representative use cases; +- requirements-based positive and negative tests; +- equivalence classes and boundary values; +- invalid, malformed, missing, inconsistent, or extreme inputs; +- option and feature interactions; +- target- and environment-specific behavior; +- stress, resource, interruption, or partial-failure conditions; +- regression tests for relevant known defects; and +- focused tests derived from TPL-003 malfunctions and propagation paths. + +Explain why the selection is adequate. A large test count does not establish coverage. + +### Establish trustworthy test oracles + +An oracle determines whether the output is correct. Possible oracles include: + +- a precise external specification; +- an independently calculated expected result; +- a known reference artifact; +- a sufficiently diverse implementation; +- semantic or property-based checking; +- execution of generated code against independently defined behavior; or +- expert review against explicit criteria. + +Analyze common cause between the tool, requirements, test generator, oracle, +and expected results. Different executable names do not establish independence. + +### Control the validation harness and supporting tools + +Harnesses, scripts, comparators, emulators, target loaders, result parsers, and +report generators can corrupt inputs or misinterpret results. +Establish confidence through measures appropriate to their reliance, such as: + +- simplicity and review; +- independent checks; +- self-tests with deliberately passing and failing cases; +- controlled versions and configuration; +- retained raw inputs and outputs; and +- classification or qualification where required. + +Do not allow a generated summary to hide raw failures, skipped cases, or parser errors. + +### Demonstrate coverage + +Distinguish: + +- requirements or behavior coverage; +- input and problem-space coverage; and +- structural coverage of the tool implementation. + +The [test-coverage chapter](LLVM-QUAL-TPL-004-standards-guide.md#test-coverage-expectations-for-software-tools) +explains the expectations and exceptions across IEC 61508, EN 50716, ISO 26262, IEC 62304, and DO-178C/DO-330. + +Record the denominator, exclusions, and gaps for every reported coverage measure. +Structural coverage does not replace behavior coverage, and a complete +requirements-to-test mapping does not demonstrate that the requirements are complete. + +### Record execution and discrepancies + +Retain enough information to identify: + +- the chronological validation activities; +- tool, requirement, manual, test, and harness revisions; +- functions and operating conditions validated; +- tools, equipment, target, and environment used; +- test cases, expected results, actual results, and oracles; +- passed, failed, blocked, skipped, flaky, and not-run results; +- expected-versus-actual discrepancies; +- tool failures, incorrect outputs, or other unexpected behavior observed; and +- the activity conclusion and limitations. + +Do not remove skipped or flaky cases silently from the denominator. For each +observed failure or incorrect behavior, analyze the effect on the qualified +usages and define corrective action, restriction, or a measure to avoid, +detect, correct, or contain it. + +### Exercise invalid, unusual, and error-related conditions + +Standards may call these **anomalous operating conditions**. They are conditions +outside the ordinary successful-use path and do not necessarily indicate a bug +in the tool. + +Consider expected behavior for: + +- malformed or invalid input; +- incompatible configuration or dependency versions; +- unavailable files, services, or permissions; +- insufficient memory, storage, or time; +- interruption or partial execution; +- corrupt caches or intermediate artifacts; +- unsupported target and feature combinations; and +- internal failure or inconsistent state. + +Expected behavior may be rejection, diagnostic reporting, safe termination, +preservation of inputs, or another defined reaction. +A visible crash and a plausible but incorrect artifact have different consequences. + +### Reuse upstream tests critically + +For an upstream or third-party suite, determine: + +- which qualification requirements and TPL-003 concerns it covers; +- whether its expected results and oracles are suitable; +- whether its configuration matches the qualification boundary; +- whether failing, unsupported, quarantined, and flaky cases are visible; +- whether results are retained for the exact qualified revision; +- whether its harness and support tools are controlled; and +- which gaps require focused qualification tests. + +## Increased confidence from use and service-history evidence + +Use the exact standard-specific method or evidence-route name. +This section uses **service-history evidence** only as a neutral umbrella +for the practical issues shared by several standards. + +### Define the population and observation period + +Record: + +- tool versions or comparable versions; +- dates and duration of use; +- organizations, projects, or deployment population represented; +- number and type of executions, artifacts, users, or user-hours where meaningful; +- environments, targets, configurations, and use cases; +- expected workloads and stress conditions; and +- how the population was selected. + +Elapsed calendar time alone is weak evidence when the volume and opportunity for observation are unknown. + +### Demonstrate similarity + +Compare historical and qualified contexts for: + +- purpose and use case; +- functions and features exercised; +- input domain and complexity; +- configuration and dependencies; +- host and target environment; +- expected outputs and reliance; and +- mechanisms capable of revealing a malfunction. + +Analyze the effect of every material difference. + +### Show that failures could have been observed + +“No reported failures” is meaningful only if the observation and reporting system +could reveal the relevant failure class. Assess: + +- downstream tests, reviews, monitoring, or comparisons; +- user competence and reporting incentives; +- issue-reporting channels and retention; +- triage, duplicate, classification, and closure rules; +- access to private or customer-only reports; +- silent or latent failures; and +- the denominator or exposure against which defects are interpreted. + +### Analyze defects and changes + +Include confirmed and suspected defects, severity, affected features and +configurations, discovery mechanism, detection latency, introduced and +corrected versions, recurrence, regression prevention, unresolved issues, and +changes that affect comparability. + +A mature project with visible and corrected defects can provide stronger evidence +than a project that merely asserts that no defects exist. + +### Build on open-source project evidence effectively + +Open-source projects can provide rich and independently inspectable evidence, +including development history, code reviews, continuous-integration results, +regression tests, release records, issue discussions, known defects, and +corrective changes. This transparency can be a significant strength when +establishing confidence from use. + +Popularity indicators such as downloads, stars, package statistics, project age, +or reputation can provide useful context. On their own, however, they might not +show which tool functions and configurations were exercised, whether the usage +is comparable to the qualified use, or whether the relevant kinds of failure +would have been detected. The same limitation applies to broad adoption claims +made for proprietary tools. + +Use the available evidence according to what it demonstrates. Upstream project +evidence may establish tool history, development activity, testing, defects, +and corrective action. Downstream or integrator evidence may additionally +demonstrate experience with a particular version, configuration, environment, +and use case. These sources are complementary; neither is automatically more +valuable than the other. + +Where the available upstream evidence already establishes sufficient +comparability, exposure, failure detection, and defect history for the intended +use, additional downstream service-history evidence is not required merely +because the tool is open source. Record any remaining gaps or uncertainty and +address them through other qualification activities where necessary. + +### Define acceptance criteria and uncertainty + +Criteria can address minimum comparable exposure, acceptable unresolved defect classes, +required observation mechanisms, maximum unassessed version difference, evidence completeness, +and required supplementation by another activity. + +State the remaining uncertainty, especially for latent malfunction classes. + +## Evaluation of the tool development process + +ISO 26262 identifies evaluation of the tool development process as a +qualification method. Process evidence can support other standards, but its +role needs to be mapped to their own provisions. + +### Evaluate actual practice + +Evaluate both: + +- the defined process; and +- objective evidence that representative releases and changes followed it. + +A policy that is not followed provides weak evidence. Disciplined work that is +not documented may also be difficult to assess or reuse. + +### Select assessment criteria + +Depending on the scope, consider: + +- requirements and intended-behavior management; +- architecture and design control; +- implementation standards and review; +- verification, validation, test design, and regression; +- configuration, build, release, and change management; +- issue and defect management; +- traceability and documentation; +- competence and independence; +- infrastructure and supporting-tool control; +- vulnerability handling where it can affect the qualified baseline; and +- maintenance, support, and end-of-life arrangements. + +Map criteria to the applicable standard or documented procedure. + +### Define sampling and evidence + +If complete review is impractical, sample safety-relevant changes, qualified-function changes, +bug fixes, regression evidence, releases across the claimed period, rejected or reverted changes, +emergency fixes, and work from different contributors or organizations. + +Explain why the sample is representative and which areas remain unassessed. + +### Evaluate open-source development practices and public evidence + +For an open-source tool, evaluation of the development process can use publicly +available and independently inspectable evidence. Relevant evidence may include: + +- contribution and review practices; +- maintainer responsibilities; +- continuous-integration and testing workflows; +- regression-test requirements; +- issue reporting and triage practices; +- change and release policies; +- documentation and communication of known issues; and +- traceability between reported problems, corrective changes, and tests. + +The absence of a conventional supplier process or a single organizational owner +does not mean that the development process cannot be evaluated. Assess the +practices that the project actually follows and the evidence those practices +produce. + +Open-source transparency can make this evaluation particularly informative. +However, transparency is not itself adequacy. +Publicly visible activity does not by itself demonstrate that every +practice is consistently applied. The evaluation should identify both the +strengths of the observed process and any gaps relevant to the intended tool use. + +Assess whether the practices: + +- address the qualified functions; +- are enforced rather than aspirational; +- leave objective, stable records; +- identify exceptions and emergency paths; and +- remain applicable to the claimed release. + +Different contributors reviewing one another can support independence, +but organizational, requirement, implementation, and infrastructure common cause +still need analysis. + +### Record findings and effects + +For each finding, record the unmet criterion, affected releases and functions, +whether the weakness could permit or conceal a relevant malfunction, corrective +or compensating action, closure evidence, and effect on the planned objective +and qualification conclusion. + +Do not treat every difference from a conventional safety lifecycle as a nonconformity +without analyzing the objective it serves. +Equally, do not treat community custom as a controlled process without evidence. + +## Development in accordance with a safety or assurance standard + +This approach is strongest when planned from the beginning of tool development. +Retrospectively reconstructing missing records is not equivalent to demonstrating +that the lifecycle was followed. + +### Define the applicable lifecycle + +Identify: + +- standard, edition, part, clauses, and integrity target; +- lifecycle and tailoring; +- organizational procedures and competence requirements; +- required verification, validation, independence, and confirmation measures; and +- required work products and traceability. + +### Demonstrate implementation + +Reference controlled evidence for requirements, architecture, design, +implementation, traceability, reviews, analyses, tests, configuration and +change management, issue and defect handling, release decisions, and +independent assessment or certification where applicable. + +### Treat tailoring and deviations explicitly + +For each material tailoring or deviation, state why the requirement or +objective is not applicable or is met differently, how the decision was +reviewed or accepted where required, what alternative evidence or control is +used, and how the decision affects the boundary. + +### Existing open-source tools + +An existing general-purpose open-source project is unlikely to have been developed +from inception under a functional safety standard. +That does not prevent use of other applicable qualification approaches. + +A safety-focused downstream distribution can introduce a controlled lifecycle +for its patches, builds, evidence, and releases. +It should not claim that this retrospectively changes the upstream development history. + +## Independent output verification, diversity, and effective controls + +The standard-specific role can differ: classification input, qualification +or conformance evidence, fallback control, supplementary measure, or a reason why +qualification credit is not claimed. State the precise basis. + +### Define the checked property + +Examples include: + +- semantic equivalence of generated code; +- compliance with a language, model, or output rule set; +- consistency between source and binary properties; +- correctness of a calculated metric; +- completeness of traceability; +- reproduction of a pass/fail decision; or +- absence of specified malfunction classes. + +“The output is independently checked” is too vague. + +### Establish coverage and timing + +Explain whether the control covers every artifact or a sample, every release +or only qualification runs, all relevant functions and configurations, +and each targeted TPL-003 malfunction. +State whether an error can propagate or become latent before the check occurs. + +### Analyze independence and common cause + +Consider shared requirements, formalized properties, parsers, libraries, models, +algorithms, code generators, input artifacts, configuration, developers, reviewers, +organizations, infrastructure, and result-processing scripts. + +### Control discrepancies + +Define how mismatches are preserved, investigated, and addressed before the +output is accepted. A warning no one reviews is not an effective control. + +### Verify the verifier + +The independent checker can introduce or miss errors. Establish sufficient confidence +in the parts on which the qualification evidence relies. + +This does not mean that every tool used to develop or test the tool under +qualification must itself be qualified. Qualification is not automatically +inherited or applied recursively. Confidence in a supporting tool may instead +be established through focused testing, review, manual checks, trusted reference +cases, a simple and transparent implementation, diversity, or other appropriate +controls. + +Consider how an error in the supporting tool could affect the qualification +evidence. If its output is relied upon without another activity capable of +detecting relevant errors, assess whether the applicable standard's tool +classification and qualification provisions apply. Otherwise, record the checks +or controls used to make the resulting evidence trustworthy. + +## Certification and its relationship with qualification + +### Separate four related concepts + +- **Conformity assessment** demonstrates whether specified requirements are fulfilled. + It can include testing, inspection, audit, evaluation, review, and surveillance. +- **Certification** is third-party attestation under a defined certification scheme + that the assessed object conforms to specified requirements. +- **Accreditation** is attestation concerning the competence and impartiality of a + conformity-assessment body. Accreditation of the body is not certification of the tool. +- **Tool qualification** is the applicable-standard process for establishing or claiming + confidence or certification credit for a bounded tool usage. + +### A certificate is not inherently a development-process audit + +Many functional-safety tool certification schemes do audit the tool-development +and maintenance process. They can also review validation results, requirements +traceability, known issues, service history, configuration management, and user +restrictions. In that common situation, certification can encapsulate several +underlying qualification methods or evidence routes. + +Certification is not inherently equivalent to evaluation of the tool development process. +The certification scheme decides what is assessed. A certificate can be based primarily +on product testing, process audit, lifecycle compliance, use history, a combination +of those elements, or another conformity-assessment design. + +The word *certified* alone therefore does not reveal what was demonstrated. + +| Underlying certification or assessment basis | Closest qualification contribution | +| --- | --- | +| Audit of actual tool-development and maintenance practices | Evaluation of the tool development process | +| Confirmation that the lifecycle conforms to a named safety or assurance standard | Development in accordance with a safety or assurance standard | +| Requirements-based functional, robustness, and unusual/error-condition testing | Tool validation or conformance evidence | +| Review of comparable operational history and known-issue data | Increased confidence from use or service-history evidence | +| Process audit plus product validation plus mandatory user restrictions | Combination of methods, evidence routes, and controls | +| Certification of the supplier's general QMS only | Indirect supporting evidence; not certification or qualification of the tool product | +| Assessment of a qualification kit | Independent review of reusable evidence; local applicability and activities remain necessary | + +### Interpret certification in four layers + +For every certificate or certification package, identify: + +1. **Underlying assurance activities:** What was actually examined—process, lifecycle, + validation, use history, controls, or a combination? +2. **Independent assessment:** Who performed the assessment, under which scheme, + with what competence, independence, sampling, and evidence access? +3. **Attestation:** What exact conformity claim does the certificate make, + and for which product, version, functions, environment, standard, integrity level, + and conditions? +4. **Applicability:** Does that claim cover the current tool baseline, usage, + configuration, target, dependencies, reliance, and required downstream activities? + +The assessment report, safety or qualification manual, certified version list, +known-issues list, and scheme rules are often necessary to interpret the +certificate correctly. + +### Standard-specific role of certification + +Apply the standards in the required comparison order: + +- **IEC 61508:** certified tools and translators are a named technique or measure. + The certificate does not remove the need to justify selection, identify the exact usage + and baseline, assess reliance and failure mechanisms, meet the applicable support-tool + provisions, and follow certificate conditions. +- **EN 50716:** a certificate is normally third-party evidence to be mapped to the Clause 6.7 + confidence and control argument. Confirm which functions, process areas, SIL, versions, + and local activities it covers. +- **ISO 26262:** certification is not a fifth software-tool qualification method. + Its underlying assessment can provide evidence for one or more of the four methods. + The ISO method-selection and qualification-report obligations remain applicable. +- **IEC 62304:** certification is not an IEC 62304 tool-qualification route. + Product, QMS, or supplier certificates can support the wider regulatory validation argument + only within their stated scope. +- **DO-178C/DO-330:** commercial certification or a vendor qualification package does not + by itself grant certification credit. The applicant must establish project applicability + and satisfy the applicable DO-330 objectives and authority expectations. + +### Avoid common certification errors + +Do not: + +- cite only the certificate front page; +- treat a QMS certificate as product qualification; +- assume that the highest SIL or ASIL mentioned covers every tool use; +- ignore certificate restrictions, validity, maintenance, and withdrawal status; +- assume that a source-level assessment covers a rebuilt or patched binary; +- assume that one certified configuration covers plugins, targets, or options not assessed; or +- describe receipt of a qualification kit as completion of local qualification activities. + +## Reuse certificates, qualification kits, and prior reports + +### Identify the complete package + +Record, where applicable: + +- certificate identifier, issuer, scheme, issue date, and validity; +- assessment or certification report; +- the applicable TPL-002 and qualification report; +- validation suite, procedures, and results; +- safety, qualification, or user manual; +- covered versions, builds, targets, functions, and usages; +- covered standards, editions, classes, levels, and TQLs; and +- required integrator activities, restrictions, and known issues. + +### Perform a structured match + +Compare the external package with the current tool identity, binary or source +baseline, usage, functions, configuration, environment, target, dependencies, +classification, integrity level, output reliance, complementary measures, +standard edition, known-issue status, and maintenance status. + +Classify each difference as covered, requiring additional evidence or local activity, +r outside the reusable boundary. + +### Complete local activities + +Qualification packages often require local installation verification, configuration checks, +est execution, target-specific validation, or project tailoring. +Record those activities and results in TPL-004. + +### Check maintenance and withdrawal status + +Define how the evidence package is updated for releases, corrections, security fixes, +certificate changes, qualification-package revisions, and withdrawn versions. +Assign responsibility for monitoring those changes and triggering impact analysis. + +## Activity-record checklist + +For every activity recorded in the **Detailed qualification activity results** +section of the template, check that: + +- [ ] the planned activity, objective, standard-specific basis, and acceptance + criteria are identifiable through TPL-002; +- [ ] the applicable TPL-002 objectives and TPL-003 concerns are identified; +- [ ] the applicable tool identity and the evidence, environment, and execution + context are identifiable and reviewable; +- [ ] actual performers, reviewers, independence, and execution dates are recorded; +- [ ] execution deviations and plan changes remain visible; +- [ ] the performed procedure is reproducible or reviewable; +- [ ] acceptance criteria were defined before interpreting results; +- [ ] TPL-002 or its referenced procedures define coverage measures, denominators, expected results, oracles, assessment criteria, and their trust basis; +- [ ] passed, failed, skipped, unavailable, and adverse evidence is visible; +- [ ] deviations and findings are linked to the common finding register; +- [ ] the activity result is separated from its contribution to the integrated qualification conclusion; +- [ ] common-cause and independence limitations are recorded; and +- [ ] follow-up actions, user conditions, and information for a safety manual or + other suitable user documentation are identified. + +The acronym list for the guide set is maintained in the [central guide](LLVM-QUAL-TPL-004-guide.md#acronyms). diff --git a/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-standards-guide.md b/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-standards-guide.md new file mode 100644 index 0000000..693c37f --- /dev/null +++ b/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004-standards-guide.md @@ -0,0 +1,336 @@ +# Standards Companion to LLVM-QUAL-TPL-004 — Qualification Concepts and Test Coverage + +> [!NOTE] +> **Guidance provided by:** [LLVM Qualification Group](https://llvm.org/docs/QualGroup.html) +> **Related template:** `LLVM-QUAL-TPL-004` +> **Central guide:** [`LLVM-QUAL-TPL-004-guide.md`](LLVM-QUAL-TPL-004-guide.md) +> **Qualification-approaches companion:** [`LLVM-QUAL-TPL-004-qualification-approaches-guide.md`](LLVM-QUAL-TPL-004-qualification-approaches-guide.md) +> This companion is informative. It compares concepts to support interpretation; +> it does not replace any standard, regulatory guidance, certification +> authority, or documented organizational interpretation. + +## Purpose and limits of this comparison + +The standards covered here do not all define software-tool qualification +in the same way: + +- IEC 61508 (E/E/PE systems) uses tool classes, conformance evidence, + SIL-graded techniques and measures, and effective controls; +- EN 50716 (railways) uses tool classes and an evidence-and-control model + focused on avoiding, detecting, or handling tool-induced failures; +- ISO 26262 (automotive) defines a tool-confidence classification and + named software-tool qualification methods; +- IEC 62304 (medical) does not define a comparable tool-qualification scheme; and +- DO-178C and DO-330 (airborne) define qualification as the process used + to obtain certification credit through a TQL-dependent, objective-based lifecycle. + +Accordingly, this guide set uses **qualification approach** as an umbrella +expression. A selected approach whose execution or evidence is recorded in +TPL-004 can be a: + +- qualification method; +- standard-recognized technique or measure; +- conformance-evidence route; +- effective control or mitigation route; +- objective-based qualification lifecycle; or +- reuse and applicability assessment for existing evidence. + +Retain the terminology of the applicable standard in the report. +Similar labels, such as "T3" or "validation", do not establish equivalence +across standards. + +## Meaning of tool qualification or confidence under each standard + +### IEC 61508 (E/E/PE systems) + +IEC 61508-3 does not organize tool confidence as a menu of qualification methods +comparable to ISO 26262. Its support-tool provisions are better understood as a +structured argument about selection, reliance, conformance, and control. + +For a bounded tool usage, the practical reasoning is: + +1. justify selection of the tool; +2. classify the usage as T1, T2, or T3; +3. define or identify the specification, product documentation, instructions, + and constraints needed for relevant T2 and T3 usages; +4. assess reliance on the tool and the failure mechanisms that could affect + the safety-related software; +5. for relevant T3 usage, establish evidence that the tool conforms to its + specification or documentation; +6. use an appropriate combination of tool validation and a history of successful + use in similar applications and environments where applicable; +7. where adequate conformance evidence is unavailable, implement effective measures + to control failures attributable to the tool; and +8. configuration-manage the tool, version, configuration, options, scripts, + and qualification status. + +IEC 61508-3 Annex A also identifies **certified tools and certified translators** +and **tools and translators with increased confidence from use** as techniques +or measures. The certificate still needs to be interpreted together with its scope, +supporting assessment, restrictions, and requirements. + +From an IEC 61508 perspective, a TPL-004 conclusion should therefore explain why +the conformance evidence and effective controls are adequate for the stated T class, +SIL context, tool baseline, functions, environment, and reliance. + +Relevant provisions include IEC 61508-3:2010, Clause 7.4.4 and Annex A, +including Table A.3. Consult the controlled copy used by the project. + +### EN 50716 (railways) + +EN 50716 uses an evidence-and-control model for support tools. Its central concern +is whether a tool can introduce an error or fail to detect an error in safety-related +software activities, and whether that failure is prevented, detected, or otherwise +adequately handled. + +For a bounded tool usage, the practical reasoning includes: + +1. justify the tool selection and T1, T2, or T3 classification; +2. identify potential erroneous outputs or failures to detect errors; +3. define the relevant tool specification, manual, usage instructions, and constraints; +4. establish output-conformance evidence, failure-detection evidence, or both; +5. select suitable evidence or controls for the applicable SIL and use; +6. validate the relevant tool functions and operating conditions where tool validation + is selected; +7. consider history of successful use in sufficiently comparable applications and environments + where applicable; +8. use independent output verification, manual-process equivalence, or tool diversity + where these form part of the accepted approach; +9. justify applicable process-compliance evidence or another appropriate method + where the standard permits it; and +10. configuration-manage the complete tool baseline and reassess changes. + +A TPL-004 conclusion under EN 50716 should state how the selected evidence and controls +demonstrate that the relevant tool-induced failures are acceptably avoided, detected, +or handled for the stated T class, SIL, usage, and conditions. + +Relevant provisions include EN 50716:2023, Clause 6.7 and related SIL-dependent +provisions. Consult the controlled copy used by the project. + +### ISO 26262 (automotive) + +ISO 26262-8 defines a distinct "confidence in the use of software tools" process. +TPL-003 records the Tool Impact, Tool Error Detection, resulting Tool Confidence Level, +and applicable ASIL boundary for each defined use case. +Where qualification is required, TPL-004 records the selected method or combination +and its results. + +ISO 26262:2018 identifies four software-tool qualification method families: + +- increased confidence from use; +- evaluation of the tool development process; +- validation of the software tool; and +- development in accordance with a safety standard. + +Method selection depends on the TCL and applicable ASIL. Consult the exact +recommendation tables and required method combinations in the applicable edition. + +Qualification remains usage-specific. The result applies to the defined tool functions, +version, configuration, environment, inputs, outputs, assumptions, and complementary measures. +A provider package or previous qualification can support the result only after applicability +has been demonstrated. + +A TPL-004 conclusion under ISO 26262 should state which qualification method combination +was applied, how it satisfies the TCL/ASIL-dependent expectation, what evidence supports it, +and which usage conditions remain mandatory. + +Relevant provisions include ISO 26262-8:2018, Clause 11. + +### IEC 62304 (medical) + +IEC 62304 is a lifecycle-process standard for medical-device software. +It does not define a peer software-tool qualification framework with: + +- development-tool classes; +- a tool-confidence decision flow; +- tool qualification levels; +- a catalogue of acceptable tool qualification methods; or +- a prescribed tool qualification report. + +Its software safety Classes A, B, and C apply to the medical-device software, +not to the development or verification tools. + +Confidence in tools used in medical-device work is normally established through +the wider quality-management and regulatory framework. +Depending on the jurisdiction and tool purpose, this can require risk-based validation +of software used in production or the quality management system, control of suppliers, +configuration and change control, and objective evidence that the software performs +as intended. + +For example, current FDA Computer Software Assurance guidance addresses software used +in medical-device production or the quality management system. +It supports a risk-based approach and a range of assurance activities, but it does not +turn those activities into IEC 62304 qualification methods and does not apply +as a complete tool-assurance framework for medical-device software development tools. + +If TPL-004 is used in a medical-device context, the report should therefore identify +the actual regulatory, QMS, or organizational requirement that creates the validation +obligation. + +### DO-178C / DO-330 (airborne) + +DO-178C defines tool qualification in terms of obtaining "certification credit" +for a software tool in the context of a specific airborne system. +Qualification becomes relevant when the project relies on a tool to eliminate, reduce, +or automate lifecycle processes and the relevant errors are not independently detected +by another process. + +The tool usage is evaluated against three criteria: + +- **Criterion 1:** the tool output is part of the airborne software and can introduce + an error; +- **Criterion 2:** the tool automates verification and its result is used to eliminate + or reduce other verification or development processes; and +- **Criterion 3:** the tool can fail to detect an error within the intended usage without + meeting the stronger Criterion 2 conditions. + +The tool criterion and airborne software level determine a Tool Qualification Level +from TQL-1 through TQL-5. Consult DO-178C and the applicable certification-authority +guidance for the exact mapping. + +DO-330 then supplies a complete, TQL-dependent set of qualification objectives, activities, +independence expectations, and lifecycle data. Qualification is therefore not performed by +choosing one alternative method such as validation or prior use. Operational requirements +and verification are central, while progressively stronger development and verification +objectives apply at higher TQLs. + +Independent verification of the relevant tool output can change the need for qualification +because it changes the certification credit claimed from the tool. If the independent process +fully performs the activity otherwise eliminated, reduced, or automated, qualification may be +unnecessary. This is a decision about reliance and credit, not merely a compensating test +added after qualification. + +A TPL-004 conclusion under DO-178C/DO-330 should state the intended use, criterion, +airborne software level, TQL, applicable DO-330 objective set, lifecycle data, deviations, +change-impact analysis, and certification-authority coordination. +Vendor qualification data is reusable only after the applicant establishes applicability +to the certification project. + +## Comparison of confidence approaches and concepts + +The following table is an interpretive map. + +| Confidence approach or concept | IEC 61508 | EN 50716 | ISO 26262 | IEC 62304 | DO-178C / DO-330 | +| --- | --- | --- | --- | --- | --- | +| Core assurance model | Tool selection, T1/T2/T3 classification, conformance evidence, SIL-graded techniques/measures, and effective controls | T1/T2/T3 classification plus evidence or controls that avoid, detect, or handle tool-induced failures | TI/TD classification produces TCL; TCL and ASIL drive named qualification methods | No tool-qualification model; medical-device software lifecycle plus surrounding QMS/regulatory validation obligations | Certification-credit decision, tool criterion, software level, TQL, and TQL-dependent objectives/lifecycle data | +| Tool validation | Explicit route for relevant conformance evidence | Explicit evidence route with validation-report expectations | Explicit qualification method | Not defined as an IEC 62304 method; risk-based validation may be required by the surrounding framework | Operational and requirements-based verification is central within the DO-330 objective set | +| Prior successful use | History of successful use in similar applications and environments; increased confidence from use is also an Annex A technique/measure | History in comparable applications and environments can contribute | Explicit increased-confidence-from-use method | Not a named method; operational history and supplier evidence may contribute to a risk-based validation argument | Not a general replacement for applicable objectives; prior qualification data can be reused after applicability and change-impact analysis | +| Evaluation of the tool development process | Can contribute to confidence or certification evidence, but is not an identically named standalone route | Process and SIL-compliance evidence can contribute under the applicable route | Explicit qualification method | Not a named method; supplier or process assessment may support the wider QMS validation argument | Development, verification, configuration-management, quality-assurance, and related objectives are built into the applicable DO-330 lifecycle | +| Development in accordance with a safety or assurance standard | Can support certification or other conformance evidence | Applicable process-compliance evidence can contribute | Explicit qualification method | Not defined as a tool-confidence route | DO-330 is itself a dedicated assurance lifecycle for the tool, scaled by TQL | +| Certified tool or translator | Explicitly recognized technique/measure; scope and underlying assessment still require applicability review | Third-party evidence that must be mapped to the Clause 6.7 argument | Evidence that may support one or more of the four methods; not a fifth method | Not an IEC 62304 method; third-party assessment may be supporting evidence | Not a separate method; certification credit depends on project-specific DO-330 compliance and authority acceptance | +| Independent output verification or tool diversity | Can be an effective control and can affect reliance on the tool | Recognized evidence/control route for relevant usages | Usually affects tool-error detection/classification unless tied to an accepted qualification method | Possible risk control, but not an IEC 62304 qualification route | Can remove the need for qualification when it independently performs the activity for which tool credit would otherwise be claimed | +| Effective downstream controls | Explicit fallback where sufficient T3 conformance evidence is unavailable | Explicit failure-detection/control route | Can affect TD and TCL and remain a mandatory usage condition | Product and process controls can support risk-based assurance | Changes the certification credit relied upon and therefore the qualification need | +| Other justified approaches | Suitable combinations of evidence and controls are possible within the IEC requirements | Relatively open to other appropriately justified methods that avoid, detect, or handle failures | Constrained by the method-selection framework | No prescribed catalogue; follow the applicable QMS/regulatory obligation | Alternative means require substantiation and agreement through the certification process | +| Dedicated tool qualification objectives and lifecycle | No separate TQL-style lifecycle | No separate TQL-style lifecycle | No separate tool-development lifecycle; the selected methods are applied | None for tools | Yes; complete TQL-dependent objectives, activities, independence, and lifecycle data | +| Reuse of prior evidence | Reuse requires matching tool, use, environment, SIL context, restrictions, and current known-issue status | Reuse requires a documented applicability assessment | Qualification remains bounded to the intended use and current baseline | Supplier evidence may be reused, but the manufacturer retains responsibility | Reuse requires matching intended use, operational environment, criterion, software level, TQL, baseline, and change impact | + +## Test-coverage expectations for software tools + +### Distinguish three kinds of coverage + +Tool-validation discussions often use the word *coverage* for different questions: + +1. **Requirements or behavior coverage** asks which required or expected tool behaviors + are addressed by tests and other evidence. +2. **Input and problem-space coverage** asks which use cases, feature + interactions, input classes, boundaries, configurations, targets, invalid or + unusual conditions, and known malfunction classes were exercised. +3. **Structural coverage of the tool implementation** measures which implementation + elements were exercised, such as statements, decision outcomes, conditions, or functions. + +These measures are complementary. A complete requirements-to-test matrix can still be weak +if the requirements are incomplete, the tests are superficial, or the oracle is wrong. +High structural coverage can still miss required behavior. +Neither metric alone establishes "adequate validation". + +### Cross-standard comparison + +| Standard and route | Requirements or behavior coverage | Structural coverage of the tool's own code | +| --- | --- | --- | +| IEC 61508 — tool validation or T3 conformance evidence | Evidence should address the relevant specification or product documentation, used functions, identified failure mechanisms, operating conditions, test cases, results, and discrepancies. Traceability should make the claimed conformance reviewable. | No universal statement, branch, function, or MC/DC target is imposed merely because support-tool validation is used. Structural coverage can become applicable through a selected development or certification basis, or can be added as justified supplementary evidence. | +| EN 50716 — support-tool validation or conformance evidence | Evidence should address the functions, outputs, conditions, and failure possibilities relevant to the classified usage, including the basis for deciding that output conforms or failures are detected. | No universal structural-coverage target is imposed merely by the support-tool validation route. It can apply through a development/process-compliance basis or as selected additional evidence. | +| ISO 26262 — qualification by validation of the software tool | Validation should be based on the tool requirements and qualified usage and should address the relevant functions, malfunctions and consequences, operating environment, and reaction to invalid, unusual, boundary, or other error-related conditions (called anomalous operating conditions in the standard). Requirements-to-test traceability is normally a key adequacy measure. | ISO 26262 does not apply the Part 6 product-software structural-coverage ladder universally to tool validation. Structural coverage can be supplementary or can enter through a development-process or development-in-accordance method whose selected basis requires it. | +| IEC 62304 | IEC 62304 itself specifies no development-tool qualification or coverage scheme. Requirements/behavior evidence for a tool depends on the applicable QMS, regulatory guidance, intended use, and risk. | Not specified for tools by IEC 62304. The surrounding validation framework may justify implementation evidence, but there is no IEC 62304 structural-coverage ladder for tools. | +| DO-178C / DO-330 | Tool Operational Requirements validation and requirements-based verification are central. The required evidence and independence depend on the TQL, intended use, and certification credit. | DO-330 applies TQL-dependent structural coverage to the tool software at the higher TQLs: statement coverage at TQL-3, decision coverage at TQL-2, and MC/DC at TQL-1. The corresponding structural-coverage objective is not generally applicable at TQL-4 or TQL-5. Consult the exact DO-330 Annex A tables and applicable authority guidance. | + +### What to record when structural coverage is not required + +Absence of a structural-coverage mandate does not reduce tool validation +to one row per written requirement. Record how the validation addresses: + +- the completeness and testability of the qualification requirements; +- every function and output within the qualified boundary; +- feature and option interactions relevant to the usage; +- input classes, boundaries, invalid inputs, and unusual or error-related + conditions; +- host, target, dependencies, configuration, and operational environment; +- TPL-003 malfunctions and error-propagation paths; +- known issues and defect-regression scenarios; +- the trustworthiness and independence of expected results or test oracles; +- supporting tools, harnesses, comparators, and result processing; +- passed, failed, skipped, blocked, flaky, and not-run tests; and +- remaining gaps, uncertainty, and downstream controls. + +If a percentage is reported, define its denominator and exclusions. +“100% requirements coverage” is not meaningful when requirements were omitted, +grouped too coarsely, or mapped to tests that do not contain effective checks. + +### Structural-coverage terminology + +Use structural-coverage terms precisely: + +- **statement coverage** measures executed statements and is not necessarily + identical to source-line coverage; +- **decision or branch coverage** measures decision outcomes and does not mean + that every possible execution path was tested; +- **MC/DC** demonstrates the independent effect of each condition on a decision + outcome under the applicable definition; and +- **function or method coverage** shows that functions were invoked but says little + about internal decisions or behavior adequacy. + +Structural coverage should be recorded when it is: + +- required by the applicable standard and qualification level; +- inherited from the selected development or certification basis; +- used to assess the completeness of requirements-based testing; or +- deliberately selected as supplementary evidence. + +When it is not applicable, state that explicitly and identify the standard-specific basis rather than reporting an unexplained zero or omitting the topic. + +## Using the same evidence under several standards + +Shared factual evidence can reduce duplicated work. For example, one controlled +validation execution may provide useful evidence under IEC 61508, EN 50716, and +ISO 26262. Reuse does not make the standards interchangeable. + +For each applicable standard: + +1. retain its own classification and integrity-level basis; +2. identify the exact method, technique, evidence route, control route, or objective set; +3. map the shared evidence to that basis; +4. identify additional activities or work products needed only by that standard; +5. assess evidence adequacy separately; and +6. state a separate conclusion. + +Do not create a combined “highest tool qualification level”. +SIL, ASIL, TCL, T1/T2/T3, software levels, and TQL express different concepts. + +## References to consult + +Use the exact editions adopted by the organization, project, regulator, +or certification authority. Relevant sources include: + +- IEC 61508-3:2010 — Software requirements; +- EN 50716:2023 — Railway applications — Requirements for software development; +- ISO 26262-8:2018 — Supporting processes; +- IEC 62304:2006 and Amendment 1:2015 — Medical device software lifecycle processes; +- FDA Computer Software Assurance for Production and Quality Management System Software, February 2026; +- RTCA DO-178C / EUROCAE ED-12C, *Software Considerations in Airborne Systems and Equipment Certification*; +- RTCA DO-330 / EUROCAE ED-215, *Software Tool Qualification Considerations*; +- FAA AC 20-115D; +- EASA AMC 20-115D; +- RTCA/EUROCAE FAS FTP1055 — Tool Qualification Impacts (an informational paper rather than regulatory guidance). + +The standards and authority-approved means of compliance remain authoritative +for definitions, required objectives, method selection, independence, lifecycle data, +confirmation measures, assessment, and approval. diff --git a/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004.md b/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004.md new file mode 100644 index 0000000..e68b95e --- /dev/null +++ b/fusa-qual-wg/tools/templates/LLVM-QUAL-TPL-004.md @@ -0,0 +1,218 @@ +> [!NOTE] +> **Template provided by:** [LLVM Qualification Group](https://llvm.org/docs/QualGroup.html) +> **Document identifier:** `LLVM-QUAL-TPL-004` +> **Usage guide:** [LLVM-QUAL-TPL-004-guide.md](LLVM-QUAL-TPL-004-guide.md) +> +> This template is provided for guidance and does not replace applicable +> safety standards. + +# Tool Qualification Report for `` + +> [!TIP] +> Text between angle brackets, such as ``, is a placeholder to be +> replaced with tool-specific information. +> +> Remove the instructional notes and tips when publishing a completed document. + +| Document information | Value | +|---|---| +| Organization / team name | `` | +| Version | `` | +| Date | `` | + +> [!TIP] +> List the roles involved in this document as applicable, for example authors, +> reviewers, or approvers. Add or remove rows as needed. + +| Role | Name | +|---|---| +| `` | `` | + +## Identification and inputs + +| Item | Value | +|---|---| +| Tool name | `` | +| Tool version (optional) | `` | +| Related Tool Usage Plan | `` | +| Related Tool Classification Report | `` | +| Qualification perspective | `` | + +> [!NOTE] +> The referenced Tool Usage Plan is the source for the tool identity, version, +> configuration, environment, intended or given usages, assumptions, constraints, +> planned approaches, independence arrangements, and acceptance criteria. +> +> The Tool Classification Report is the source for the use-case classifications, +> malfunction analyses, control assumptions, and identified qualification needs. +> +> Reference those sources instead of duplicating them here. + +> [!IMPORTANT] +> This template assumes that the qualification activities are performed in +> accordance with the Tool Usage Plan referenced above. +> +> One TPL-004 report is governed by one TPL-002 revision. If a different +> Tool Usage Plan applies to another tool version or usage boundary, prepare a +> separate TPL-004 report for that plan. +> +> If a change to any of these planned elements is required, TPL-002 should +> be updated before the affected qualification activity continues or the +> qualification conclusion is established. +> Changes affecting the classification basis, malfunction analysis, or +> required controls shall also be reflected in TPL-003. +> +> Unexpected deviations encountered during execution are expected to be +> recorded in this report, then either be resolved by performing the activity +> in accordance with the plan or addressed through a plan update before the +> qualification conclusion is established. + +## Qualification activity summary + +> [!TIP] +> Summarize each qualification activity and its current status here. Use the +> standard-specific activity name from the plan. Record the detailed execution +> and results in the corresponding activity record. + +| Activity ID and name | Reference | Status or result | +| --- | --- | --- | +| `QMA-` — `` | `` | `` | + +## Evidence inventory and applicability + +> [!NOTE] +> List the controlled evidence actually used by this report. A link to changing +> content, an unbounded branch, a transient dashboard, or an unsupported +> assertion is not a controlled evidence baseline. + +| Evidence ID | Evidence item and exact baseline | Source or owner | Activity, objective, or concern supported | Applicability and limitations | Controlled location | +| --- | --- | --- | --- | --- | --- | +| `QE-` | `` | `` | `` | `` | `` | + +**Evidence unavailable to downstream users or assessors:** +`` + +## Coverage and findings + +### Coverage + +> [!NOTE] +> An individual activity can pass while an overall qualification need remains +> only partially covered. Integrate the results against the objectives and +> the concerns established by TPL-003. + +| Objective or concern | Qualification activities | Evidence | Coverage conclusion and residual action | +| --- | --- | --- | --- | +| `` | `` | `` | `` | + +**Coverage gaps and cross-activity dependencies:** +`` + +### Issues and findings + +> [!TIP] +> Include adverse evidence. Do not report only favorable results. + +| Finding ID and type | Description and affected scope | Qualification impact and disposition | Required control or follow-up | Status and evidence | +| --- | --- | --- | --- | --- | +| `QF-` — `` | `` | `` | `` | `` | + +## Qualification conclusions + +> [!IMPORTANT] +> Complete a separate conclusion for each applicable standard. Do not infer +> compliance with one standard from qualification under another. +> Use "qualified" only when the applicable criteria have been satisfied +> for the stated boundary. + +### `` + +> [!TIP] +> Repeat this section for each applicable standard. + +**Qualification activities and principal evidence:** +`` + +**Requirements or interpretation mapping:** +`` + +**Conclusion:** `` + +**Rationale:** +`` + +**Mandatory conditions, limitations, and unresolved items:** +`` + +### Consolidated statement + +`` + +## Detailed qualification activity results + +> [!IMPORTANT] +> Copy the complete activity chapter below for each qualification activity. +> The Tool Usage Plan remains authoritative for the selected approach +> and intended activity. This report records what was actually executed, +> the evidence obtained, and the resulting conclusion. + +### `QMA-` — `` + +#### Execution identification + +**Activity and qualification-basis reference:** `` + +**Performed by:** `` + +**Reviewed by:** `` + +**Execution period or evidence date:** `` + +**Execution or assessment environment:** `` + +**Execution deviations:** `` + +#### Results and evidence + +**Results summary:** `` + +**Coverage achieved and gaps:** `` + +**Applicability of reused evidence or third-party assessment, if applicable:** `` + +**Issues and findings:** +`` + +**Detailed evidence references:** `` + +#### Activity conclusion + +**Activity result:** `` + +**Assessment against the acceptance criteria:** +`` + +**Residual limitations, dependent controls, and required follow-up:** +`` + +## Validity and user information + +This qualification conclusion is valid only for the defined usage and +classification boundaries, the executed tool and evidence baselines, and the +mandatory conditions identified by the referenced work products and this +report. + +**Report-specific limitations or additional conditions requiring review or re-qualification:** +`` + +**Planned review point or validity period, if applicable:** +`` + +**Information to communicate through safety manual or other user documentation:** + +- ``; +- ``; +- ``; +- ``; +- ``; +- ``.