Checking that each requirement’s tests run its code

CLI MCP Alpha

Finding code that traces to no need tells you which functions trace to a need. A traced function is not proof that it is tested, and an untested function is not visible there. This guide joins the coverage your test run already writes, one record per test, to the same functions, and reports per requirement:

  • the functions its tests executed, and whether each is traced to it;

  • the functions it traces that its tests did not execute (a scoped gap);

  • the untraced functions its tests executed: link suggestions to review, which ubCode lists and never writes anywhere;

  • the functions executed only by tests that verify no requirement.

ubCode does not run your tests. It reads what your CI produced: an llvm-cov export per test (clang for C and C++, rustc for Rust), a gcov JSON file per test or an lcov tracefile with a test name per section (GCC for C and C++), or one coverage.py report with test contexts (pytest). Coverage is counted per function: a function a test entered is executed by that test. It says nothing about lines, branches or MC/DC inside it. For llvm-cov and gcov, “the test” is the process that ran it: static initialisation, the test framework’s start-up and the fixtures’ set-up and tear-down count for every test, so give each test its own process (step 1) and expect shared code in every test’s list.

The requirements-based coverage is computed when you ask for it: by ubc codelinks coverage --test-coverage, and by the MCP tool requirements_coverage, which takes the same inputs (and its inventory_at_coverage parameter, step 4) relative to the project root and needs a [codelinks_coverage] table. It is added to the report as a section of its own; the rest of the report is unchanged.

Step 1: collect coverage per test

Each test needs its own record of what it executed.

C and C++ with clang, Rust with rustc. An llvm profile is per process, and nothing inside it names a test, so write one export per test you run, named after the test. Compile with -fprofile-instr-generate -fcoverage-mapping (clang) or -C instrument-coverage (rustc), then, for each test:

LLVM_PROFILE_FILE=prof/AdderTest.Add.profraw ./run_tests --gtest_filter=AdderTest.Add
llvm-profdata merge -sparse prof/AdderTest.Add.profraw -o prof/AdderTest.Add.profdata
llvm-cov export -format=text -instr-profile=prof/AdderTest.Add.profdata ./run_tests > cov/AdderTest.Add.json

The export’s name, without .json, is the test’s key. Inside a directory you pass whole, a subdirectory is part of the key: cov/calc/tests/add.json under cov is calc.tests.add. A merged export of the whole run is one “test” that executed everything: do not pass one. For cargo, run each test of the test binary on its own (<test binary> --exact <test>) with its own LLVM_PROFILE_FILE; a cargo nextest wrapper that does this per test is not verified end to end.

C and C++ with GCC. A .gcda file accumulates every process that runs, and nothing inside it names a test, so write one gcov JSON file per test you run, named after the test, and delete the .gcda files before the next one. Compile and link with --coverage, then, for each test:

./run_tests --gtest_filter=AdderTest.Add
gcov --stdout --json-format build/*.gcda > cov/AdderTest.Add.json
rm build/*.gcda

--stdout writes every translation unit’s document to one file (GCC 9 and later). Without it, gcov writes one <name>.gcov.json.gz per data file instead: concatenate a test’s files into one (cat *.gcov.json.gz > cov/AdderTest.Add.gcov.json.gz) and pass that. The name, without .gz, .json and .gcov, is the test’s key, as for an llvm-cov export. gcov names each file as the compiler saw it: absolute, or relative to the directory the translation unit was compiled in (current_working_directory in the file), to which ubCode joins it. When that directory is another machine’s checkout, map it with --path-map (step 3), naming the directory as gcov wrote it (zgrep reads the .gz files too):

zgrep -ho '"current_working_directory": "[^"]*"' cov/*.json* | sort -u   # "/ci/build/checkout"
ubc codelinks coverage --test-coverage cov --path-map /ci/build/checkout=.

-ffile-prefix-map and -fprofile-prefix-map rewrite the file names gcov writes, but not current_working_directory (GCC 13), so a file compiled by a relative path (gcc -c src/calc.c) is still named after the machine’s own directory: map that directory with --path-map. To name the same directory on every machine, compile by absolute path with a prefix map to an absolute directory (a relative target such as =. makes the name relative, and it is then read against current_working_directory, the build directory of an out-of-source build):

g++ --coverage -ffile-prefix-map="$PWD"=/ci/build/checkout -c "$PWD/src/calc.cpp" -o build/calc.o

Or with lcov: write a tracefile per test with its name, and join them if you like:

geninfo build --test-name AdderTest.Add -o cov/AdderTest.Add.info
lcov -a cov/AdderTest.Add.info -a cov/AdderTest.Sub.info -o tests.info

Each section of a tracefile keeps its TN: test name through lcov -a. lcov rewrites every character of the name other than a letter, digit or _ as _ (AdderTest.Add becomes AdderTest_Add), and ubCode compares the names the same way (step 2). A tracefile without test names (coverage.py’s coverage lcov writes none) is one test, named after the file, as an export is.

To run every test in one process instead, reset the counters before each test (__gcov_reset()) and write them after it (__gcov_dump()) with GCOV_PREFIX set to a directory of that test’s own, which libgcov reads at every dump; copy each .gcno beside its .gcda before running gcov there. A function already running at the reset (main, the runner’s loop) is then not counted for the test.

Python with pytest. Record which test executed each line, and write the report with its contexts:

pytest --cov=mypackage --cov-context=test --junitxml=junit.xml
coverage json --show-contexts -o coverage.json

A context names the test by its pytest node ID. coverage.py’s own [run] dynamic_context = test_function works too. On Python 3.14 and later, coverage.py records no test context unless [run] core = ctrace is set: the report then warns that it found none.

Step 3: run the report

Pass each export, directory or report with --test-coverage (repeatable):

ubc codelinks coverage --test-coverage cov --path-map /ci/build/checkout=.
ubc codelinks coverage --test-coverage tests.info --path-map /ci/build/checkout=.
ubc codelinks coverage --test-coverage coverage.json -f json

Coverage names files as the machine that ran the tests saw them. --path-map FROM=TO rewrites a path prefix (TO relative to the project root); when no file of the coverage lands in a codelinks project, the command warns and prints the --path-map that would fix it. Relative paths, as coverage.py and cargo write them, are read relative to the project root; when coverage ran in a subdirectory, map it (--path-map sub=.), since a relative path that happens to name another file of the project is joined to that file. Paths the report echoes (not_in_projects, the hint) are the inputs’ own, which can be absolute paths of the machine that collected them.

The section always covers every function of the selected projects, with or without a snapshot (step 4): --changed-since narrows the functions the report lists, not this section, and -f lcov refuses --test-coverage, since a tracefile has no place for it.

Step 4: keep the coverage usable after the code changes

Coverage describes the code it was collected on. Joined to a newer checkout, a function of the coverage is matched by its position, so an edit that leaves every function on a definition of the same span goes unseen (two functions of the same length swapped, a body changed without moving a line), and any edit that moves a function makes its whole file stale. To join it as it was, write the report in the job that collected the coverage, on that checkout, and keep it with the coverage:

# In the job that collected the coverage, on that checkout, after step 1:
ubc codelinks coverage --test-coverage cov -f json > requirements-coverage.json
# Keep cov/ and requirements-coverage.json together, as one artefact of that job.

# Later, on any newer checkout (CI or a laptop), with both files:
ubc codelinks coverage --test-coverage cov --inventory-at-coverage requirements-coverage.json \
  --path-map /ci/build/checkout=.

With --inventory-at-coverage (the snapshot), each function of the coverage is joined to the function that was at its position when the coverage was collected, and that function’s coverage counts for the current function with the same symbol ID only while its code is unchanged (the same content hashes: comments, markers and formatting aside). Inserted lines, comments and reordered functions no longer make a file stale, and two swapped functions are each credited with their own tests. A function whose code changed is stale; one renamed, moved, deleted or whose signature changed is listed as removed (with the new name when exactly one new function has its code, as a suggestion), and a function new since the coverage has no coverage data. Coverage is never carried across a rename, a move or a change of code.

The snapshot must be written with --test-coverage over the same coverage files, byte for byte, in the job that collected them: a re-exported or regenerated file is refused, and so is a snapshot written without --test-coverage, narrowed by --changed-since, cut by --status, or itself computed with --inventory-at-coverage (the refusals say what to do). That job needs ubc (and, over the file cap, its license, CI and machines without Internet access), and both runs should use the same ubc version, or the hashes of unchanged functions may differ and read as changed. Without a snapshot, the positional join of step 3 applies, with its limits.

Step 5: read the result

After the findings, the table adds the section (from the C++ fixture of the tests):

Requirements-based coverage: 1 input, 6 tests (4 linked, 0 not passed, 0 ambiguous, 1 with no requirement, 1 without a test need)
  records: 8 matched (1 nested), 0 outside the universe, 1 unmatched, 0 ambiguous, 0 stale, 1 not in the projects, 0 malformed
  units: 6 executed, 1 not executed, 0 stale, 0 without coverage data
  requirement  tests  executed  traced  traced+exec  scoped gap  suggestions
  REQ_ADD        2/2         2       2            2           0            0
  REQ_HELPER     2/2         3       3            1           2            2
  Scoped gaps (traced, not executed by the requirement's tests):
    REQ_HELPER   cpp:src/calc.cpp#calc::helper(int)  not_executed (run by other tests)
    REQ_HELPER   cpp:src/calc.cpp#calc::unused(int)  not_executed
  Link suggestions (executed by the requirement's tests, traced to nothing; review, never applied):
    REQ_HELPER   cpp:src/calc.cpp#calc::helper(double)  (only REQ_HELPER's tests run it)
    REQ_HELPER   cpp:src/calc.hpp#calc::twice(T)  (only REQ_HELPER's tests run it)
  Tests without a test need: HelperTest.Int

For each requirement:

executed

Functions its tests executed.

traced+exec

Of those, the ones that trace to it: the expected result.

scoped gap

Functions that trace to it that its tests did not execute. Each says why: not executed (and whether another test ran it), executed only by the requirement’s own tests that are not credited, stale coverage, or no coverage data at all (with a snapshot, new_since_coverage when it is new since the coverage).

suggestions

Untraced functions its tests executed: probably part of the requirement, but not linked. The fewer requirements’ tests run a function, the stronger the evidence, so a helper every test runs comes last. They are never applied: trace the ones you agree with by a marker or a symbol field.

Below the table, a line names each test case that matched but did not pass, and how many functions its test executed: those are not counted for its requirement.

-f json writes every list in full, under requirements: per requirement executed, traced_not_executed, suggestions and tests_not_passed, and executed_without_requirement for the functions only tests without a requirement ran.

With a snapshot (step 4), the section’s first lines name it and what became of its functions, with a note for each thing that differs between it and this run, and below the scoped gaps the table lists the functions whose code changed (with the tests whose coverage was dropped) and the functions removed since the coverage (here, the C++ fixture with its two one-line helper overloads swapped since the coverage, each credited with its own test):

Requirements-based coverage: 1 input, 6 tests (4 linked, 0 not passed, 0 ambiguous, 1 with no requirement, 1 without a test need)
  snapshot: requirements-coverage.json (written at 1a2b3c4d5e6f, working tree at 9f8e7d6c5b4a): 7 units carried, 0 changed, 0 removed, 0 new since the coverage
  records: 8 matched (1 nested), 0 outside the universe, 1 unmatched, 0 ambiguous, 0 stale, 1 not in the projects, 0 malformed, 0 not in the snapshot

-f json holds it all under requirements.snapshot (see Coverage from an older commit).

Reference

Inputs

The format of each input is read from its content, never from its name, and any input can be gzip-compressed (it is read decompressed, up to 1 GiB).

Input

Read

llvm-cov export

llvm-cov export -format=text (envelope version 2.x or 3.x; another is refused by name). One test per file. A function’s instances (template instantiations, monomorphisations) are one function; a test executed it when its entry count is not zero, so anything its process ran counts (static initialisation, the test framework, fixtures).

gcov JSON

gcov --stdout --json-format (format_version 1 or 2: GCC 9 and later; another is refused by name), plain or gzip-compressed (gcov’s own *.gcov.json.gz, one file or several concatenated). One test per file, every translation unit’s document in it. A relative file name is joined to its document’s compilation directory, and . and .. resolved, so a header two translation units name differently (src/calc.hpp, tests/../src/calc.hpp) is one file. A function’s instances (template instantiations, a function of a header in several translation units, a constructor’s or destructor’s variants) are one function; a test executed it when its entry count is not zero. gcovr’s JSON is refused: it merges each file’s functions across translation units and gives no end line.

lcov tracefile

TN, SF, FN (with or without an end line) and FNDA, and lcov 2.2’s FNL and FNA; every other record is skipped. A TN: names the test of every section after it, until the next one; a section without a test name is the file’s own test, named after the file. A test executed a function when its count is not zero. ubc codelinks coverage -f lcov’s own tracefile (test name codelinks_trace) is refused: it says which functions trace to needs, not which a test ran.

coverage.py JSON

coverage json --show-contexts (format 3). A report without contexts is refused. pytest-cov’s run phase is credited, never setup or teardown (a shared fixture’s setup is attributed to whichever test first asked for it). Code run at import is no test.

Each input is read once, however it is spelled (under its lexicographically first spelling, which an export’s test key comes from), and an input inside a directory input is read with that directory, with a note. Two inputs with the same name (job1/coverage.json and job2/coverage.json) are both read, each named in the report by enough of its path to tell them apart (numbered, (1), (2), when that name is another input’s), and a test two of them name is one test, with a note. Inside a directory, the files ending .json, .json.gz, .info and .info.gz are read, in any letter case, each one test (an llvm-cov export, a gcov JSON file, or a tracefile whose sections without a test name are its own), all of one format: a directory of two formats, or holding a coverage.py report, is refused. A symbolic link is not followed (a note says so), and two files naming one test (a/b.json and a.b.json) are joined, with a note.

How a function of the coverage is matched

A coverage function is matched with the function of the same file whose definition starts at or before its first line, and whose body starts at or after it (clang starts a function at its {, rustc at fn, gcc at its name, coverage.py at def; a tracefile at the line its producer wrote). Two definitions starting on one line are ambiguous, and listed, never guessed. A function inside another one’s body that the inventory does not list (a closure, a lambda) counts for the enclosing function, but only when every file the record was read from (each export is one build) also holds a fresh record of that enclosing function itself; otherwise it is stale (nested_without_function), since an edit may have moved another function’s record into that body. A function on no definition (one a macro makes) is unmatched. A function of test code, or of a file the coverage policy leaves out, is outside the universe, and so is one the Rust build configuration compiles out (Count the Rust code of one build configuration): the JSON counts those apart, as outside_universe.inactive, present when there is one. Such a record shows the run compiled code the configuration says is not built, so the configuration does not describe that run (a tool may have set a name: cargo-llvm-cov sets coverage): each is listed in compiled_out_records with the unit and its cfg, and the table names it with the remedy (list the name in cfgs, or set declared_cfgs = "unknown").

A function that no longer fits its definition is stale, and none of its coverage counts: for llvm-cov, gcov and a tracefile with end lines, when it ends on another line than the definition; for coverage.py, when one of its lines lies outside the definition or its name differs. Once one function of a file is stale, the file has changed since that input was collected, so none of that input’s functions of that file counts either (file_stale). So does a function of the coverage that starts on no definition, unless it starts on a #define (or one of its continuation lines) or on a macro invocation the inventory reports, where a function a macro makes is recorded, or, for gcov, it is recorded at a single point (start and end on the same line and column: how gcc records a function a macro expands, a function or lambda written inside a macro’s argument, and a = default one) where the source still holds, from that line and column, that function’s own = default declaration (starting there with its name, where gcc records it, and the ( of its parameters on the same line; as many parameters as the function has, each with the same & or &&, so another constructor of its class is not its own; then its = default on that line or one of the next seven, before any ;, { or }, as clang-format wraps a long one), or, when no = default declaration starts there, a name followed by ( (the macro’s invocation); and so does a stale function of test code or other code the report leaves out. A function on a line two definitions share (ambiguous) is no such sign. A shift that lands one function exactly on another of the same length is then not credited, as long as some other function of the file no longer fits.

Coverage from an older commit

With --inventory-at-coverage (step 4), a function of the coverage is matched against the snapshot instead: with the snapshot’s function whose definition holds its first line and that it lies in (for llvm-cov, gcov and a tracefile with end lines, it ends at or before the definition’s last line; for coverage.py, its lines are inside the definition and its name is the function’s or one nested in it). A function that is the definition’s own (it ends where the definition does, or has its name) is joined to it; a closure or a lambda inside it counts for it only with the function’s own record in the same file of the coverage, as above. Two candidates are ambiguous. A function inside a definition that it does not fit is stale (snapshot_mismatch), and so is the rest of its file from that input (file_stale): the coverage and the snapshot do not describe the same code. A function on no definition of the snapshot is unmatched and no sign of an edit: under a snapshot, unmatched also holds test code and other code the report leaves out inside a counted file (a Rust #[cfg(test)] module, a private function under visibility = "public") and functions a macro makes. Then each function’s coverage is carried to the current function with the same symbol ID and the same content hashes.

Since the coverage

Result

Nothing, or comments, markers, blank lines, formatting, reordered functions

Carried: counted as on the day the coverage was collected.

Two functions of the same length swapped

Each carried with its own tests.

A body changed, moved or not

Stale (changed_since_coverage); listed under changed with the tests whose coverage was dropped.

Renamed, moved to another file, signature changed, deleted, or no longer in the selected functions

Not carried (removed_since_coverage); listed under removed, with renamed_to when exactly one new function of the project has its code (a suggestion, never credited).

New, or now selected by the coverage policy

No coverage data; listed under new, and a scoped gap of reason new_since_coverage.

A file the snapshot does not list (a project it did not select)

Not counted (not_in_snapshot); listed under files_not_in_snapshot.

A file the snapshot’s universe left out (test code, generated, exclude), deleted since or not

Outside the universe, as without a snapshot.

The snapshot is refused (exit code 2, naming the file) when it is not a ubc codelinks coverage -f json report, was written under another symbol-ID scheme, was narrowed by --changed-since or cut by --status, was written without --test-coverage, was itself computed with --inventory-at-coverage (it then lists the functions of the commit it was written on, not those the coverage was collected on: keep the snapshot written in the coverage job, never a later report), or did not read one of the coverage files given now (compared by SHA-256: fewer files now than then is fine). It is never refused for anything that can only make a function read as changed or absent; the table notes each instead, and requirements.snapshot.differs says which: another ubc version or other grammar versions (unchanged functions may read as changed, and functions another ubc version names otherwise as removed and new), a symbol pack only one of them uses (that language’s functions are new or removed), another configuration (functions it left out, or put in other projects, are not carried). The snapshot’s commit and this run’s (git_head, current_git_head) are named when known, and stale_at_coverage counts the snapshot’s own stale functions: not zero when a function joined no definition when it was written, or when it was written on another checkout than the coverage’s (its own stale list tells which).

Test keys

Key

Matches a need when

A per-test file’s name (an llvm-cov export, a gcov JSON file, a tracefile’s sections without a test name)

It is the need’s ID, or the test case’s classname.case (case alone without a class name).

A tracefile’s test name (TN:)

It is the need’s ID or the test case’s classname.case, compared as lcov writes names (every character other than a letter, digit or _ as _). Two test cases (or needs) that compare equal (A_B.c and A.B_c) make the name ambiguous: listed, never credited.

A pytest node ID

It is the need’s ID, or, mangled as pytest writes a JUnit report (tests/test_ops.py::TestOps::test_add[1-2] is tests.test_ops.TestOps.test_add[1-2]), the test case’s classname.case.

A coverage.py test_function context

It is the need’s ID, or the test case’s classname.case_name, which every parametrisation shares.

Matching is exact. A test that matches no need is listed with the nearest test-case key, never linked. A matched test case credits its requirements only when its result is passed; a need without a result field always does. A test that matches several needs credits only when all of them do. A test without a class name (an export named test_init) that matches test cases of several suites or reports is ambiguous: listed, never credited. pytest’s --junit-prefix changes the JUnit class names, so node IDs no longer match.

What a test framework writes, and how to name its coverage:

Framework

Test report case

Coverage file or TN

GoogleTest

AdderTest and Add; parameterised Inst/AdderTest and Add/0

cov/AdderTest.Add.json, or --test-name AdderTest.Add (TN:AdderTest_Add); --test-name Inst/AdderTest.Add/0 (TN:Inst_AdderTest_Add_0) matches the parameterised case. As a file name, / is a directory under a directory argument, so name such a test’s file by its need’s ID instead.

Catch2, Unity, Ceedling

Not verified end to end

Name each test’s file or TN by its JUnit classname.case, or by its need’s ID.

Known limits

  • Per function only: no lines, branches or MC/DC.

  • gcov counts the entries of a function: with __gcov_reset() between tests in one process, a function already running at the reset is not credited to the test.

  • A tracefile without end lines (FN:<line>,<name>, which the format allows) finds no stale function by itself: only a function on no definition, or a stale neighbour from the same input, says its file changed.

  • A tracefile has no columns, so a function a macro makes at file scope (one line, on no definition) makes its file stale on every run without --inventory-at-coverage: read gcov’s JSON instead, or use a snapshot (step 4).

  • GCC lists the inline functions of system headers it compiled (/usr/include/...): they count as not_in_projects.

  • One export per test for llvm-cov; Windows file names cannot hold ::, so name an export by the test case’s classname.case or by its test need’s ID.

  • Without --inventory-at-coverage, staleness is positional: an edit is detected when some function of the coverage no longer fits the file (its span or lines no longer match a definition, or it starts on no definition outside a #define or a macro invocation). Not detected: an edit that leaves every function of the coverage on a definition of the same span (two functions of the same length swapped, a body changed without moving a line), or one that shifts a function onto a #define or macro-invocation line while the others land on same-span neighbours. For gcov, the same holds for a function recorded at a single point that an edit shifts onto another line holding, at its column, a name followed by ( (another macro invocation, a call or a declaration), or onto the = default declaration of a function of the same name and the same parameter shape (the same number of parameters, each with the same & or &&; a defaulted function’s overloads differ in that, but another class’s defaulted operator in the same file, such as operator==, operator<=> or operator=, can match both).

  • gcov records a function at a single point only where it knows the column: past column 4096 of a line it writes column 0, and such a one-line function is a span like any other. A single point whose line no longer holds what gcc recorded there, or holds another function’s = default declaration, or a line above its own (a comment, an access label, blanks), makes its file stale, and so, on every run without --inventory-at-coverage, does one that never did: a macro invocation or a defaulted function’s name whose ( is on the next line (including an out-of-class qualified name wrapped after :: or inside <...>, as clang-format wraps a long one), or a = default more than seven lines below the function’s name or separated from its parameters by more than qualifiers, macros and a trailing return type (a requires clause).

  • A file with a function that legitimately joins no definition (a lambda at namespace scope the inventory does not list, a compiler-made function) is stale on every run (without --inventory-at-coverage): the unmatched list names that function, and each stale entry its reason (file_stale).

  • With a snapshot, coverage is evidence about a function’s own code only: an edit to its callers, its callees, the tests, the headers and macros it uses or the build options is not seen. For release evidence, collect coverage on the commit you release.

  • A snapshot written on another checkout than the one the coverage was collected on (later, or with coverage files brought in from another commit) is not detected: write it in the job that collected the coverage. A report itself computed with --inventory-at-coverage is refused as a snapshot.

  • There is no ubproject.toml key or editor view for it yet: the editor’s coverage view (See it in the editor) shows trace coverage, not which tests ran the code.