Software test evidence
Requirements-based test evidence review for airborne software
A requirements-based test evidence review checks that an article's test set actually verifies its requirements: that every test traces to a requirement, that normal-range and robustness conditions are both exercised, and that the recorded results substantiate the pass each test claims under DO-178C. It is used by avionics and equipment suppliers building software verification evidence. You receive a reviewed test package and a list of requirements with thin coverage, tests with no requirement behind them, and results that do not support the verdict recorded.
When this review is needed
- Test evidence is approaching closure and the team needs traceability and robustness coverage read independently.
- Requirements changed after tests were written and the trace has to be reconciled with the current set.
- Tests pass but the recorded results are too thin to show the requirement was actually exercised.
- Robustness conditions, out-of-range and invalid inputs, are suspected to be under-covered.
The problem
Requirements-based testing is supposed to demonstrate that each requirement behaves as specified, including when inputs go out of range, but test sets drift away from that purpose. Tests get written to confirm what the code does rather than what the requirement demands, normal-range cases crowd out robustness cases, and a pass verdict is recorded with results too sparse to show what was checked. Low-level and high-level requirement coverage are also frequently uneven.
What gets reviewed
- Traceability from each requirement to the tests that verify it and back again
- Coverage of normal-range behavior and robustness conditions for each requirement
- Whether recorded results substantiate the pass or fail verdict assigned
- Balance of coverage across high-level and low-level requirements
- Test procedures sufficient for a reviewer to repeat the verification
- The expected result stated for each case and whether the recorded outcome matches it
What gets validated
- Every requirement traces to at least one test that verifies its behavior
- Every test traces back to a requirement and is not orphaned
- Robustness conditions are tested where the requirement implies input limits
- Recorded results contain enough detail to support the verdict assigned
- High-level and low-level requirements are both covered as the level expects
- Each test states an expected result the recorded outcome can be checked against
Evidence normally required
- The requirements set at both high and low level
- The test cases, procedures, and recorded execution results
- The requirements-to-test traceability data
- The software level assignment and the verification plan
Common discrepancies
- Tests written to confirm code behavior with no requirement behind them
- Missing robustness conditions where the requirement defines input limits
- Pass verdicts recorded against results too thin to show what was exercised
- Requirements with no test in the trace, or trace links that point nowhere
- Uneven coverage that leaves low-level requirements thinly tested
What is at stake
Test evidence that does not trace cleanly to requirements weakens both the verification claim and the structural-coverage argument that depends on it. Rebuilding the trace and adding robustness cases late means re-execution under schedule pressure, and the coverage analysis has to be redone behind it.
Move from findings to resolution
Identify gaps against the means of compliance.
How the work runs
Reconcile the trace
Confirm every requirement traces to a test and every test traces back to a requirement, flagging orphans on both sides.
Probe robustness
Check that out-of-range and invalid-input behavior is exercised where the requirement implies input limits.
Audit the verdicts
Confirm each recorded result carries enough detail and an expected result to support the pass or fail assigned.
Flag the gaps
Return the reviewed package with thin requirements and orphaned tests flagged, ordered by exposure.
What the buyer receives
- A reviewed test package with traceability confirmed or flagged per requirement
- A list of requirements with thin coverage and tests with no requirement behind them
- A robustness-gap note ordered by the requirements most exposed
Who uses the output
- Verification leads closing the test evidence
- Software engineering leads adding robustness cases and repairing the trace
- Certification leads confirming the test argument supports the coverage analysis
How the work fits into the transaction or program
The review sits upstream of structural coverage, since coverage is only meaningful when it is harvested from requirements-based test execution. Reconciling the trace and robustness here keeps the coverage analysis from inheriting gaps it cannot resolve later.
Start with a single asset
Confirm requirements trace through verification.
Regulatory limits
The review checks the applicant's test evidence for traceability, robustness, and result integrity. It does not make compliance findings, execute the tests, or guarantee the authority will accept the verification argument.
What this review does not cover
- Making compliance findings on the authority's behalf
- Writing or executing the missing tests
- Approving the requirements set or the verification plan
Specific to this review
- Requirements-based testing fails its purpose when tests confirm code behavior instead of requirement behavior, a confusion that recurs across programs.
- Robustness coverage, behavior under out-of-range and invalid inputs, is the dimension most often under-tested even when normal-range coverage looks complete.
- Thin test results undermine a pass verdict; the record has to show what was exercised, not merely that a test ran.
- A test with no stated expected result cannot substantiate a pass, because there is nothing the recorded outcome was checked against.
Sources
RTCA. Objectives and lifecycle data for airborne software assurance, by design assurance level (DAL A-E).
U.S. Government (eCFR). Type certificates, STCs (Subpart E), TSO authorizations (Subpart O), PMA (Subpart K), and export airworthiness approvals (Subpart L).
Frequently asked questions
Why review test evidence before structural coverage?
Coverage only counts when it comes from requirements-based test execution. Repairing the trace and robustness first keeps the later coverage analysis from carrying gaps it cannot close.
What counts as thin test results?
Results that record a pass without showing what was exercised or what outcome was expected. The record has to let a reviewer see the requirement behavior that was actually checked.
Relevant glossary terms
Related pages
Where this fits
Talk to an engineer who has done this work
We will walk through your current state, the records or evidence involved, and a scoped first engagement.
Walk through your situation with an engineer who has done this work.