Checking that each requirement’s tests run its code¶
CLI MCP Alpha
Finding code that traces to no need tells you which functions trace to a need. A traced function is not proof that it is tested, and an untested function is not visible there. This guide joins the coverage your test run already writes, one record per test, to the same functions, and reports per requirement:
the functions its tests executed, and whether each is traced to it;
the functions it traces that its tests did not execute (a scoped gap);
the untraced functions its tests executed: link suggestions to review, which ubCode lists and never writes anywhere;
the functions executed only by tests that verify no requirement.
ubCode does not run your tests. It reads what your CI produced: an llvm-cov export per test (clang for C and C++, rustc for Rust), a gcov JSON file per test or an lcov tracefile with a test name per section (GCC for C and C++), or one coverage.py report with test contexts (pytest). Coverage is counted per function: a function a test entered is executed by that test. It says nothing about lines, branches or MC/DC inside it. For llvm-cov and gcov, “the test” is the process that ran it: static initialisation, the test framework’s start-up and the fixtures’ set-up and tear-down count for every test, so give each test its own process (step 1) and expect shared code in every test’s list.
The requirements-based coverage is computed when you ask for it:
by ubc codelinks coverage --test-coverage, and by the MCP tool requirements_coverage,
which takes the same inputs (and its inventory_at_coverage parameter, step 4) relative to the project root
and needs a [codelinks_coverage] table.
It is added to the report as a section of its own; the rest of the report is unchanged.
Step 1: collect coverage per test¶
Each test needs its own record of what it executed.
C and C++ with clang, Rust with rustc.
An llvm profile is per process, and nothing inside it names a test,
so write one export per test you run, named after the test.
Compile with -fprofile-instr-generate -fcoverage-mapping (clang)
or -C instrument-coverage (rustc), then, for each test:
LLVM_PROFILE_FILE=prof/AdderTest.Add.profraw ./run_tests --gtest_filter=AdderTest.Add
llvm-profdata merge -sparse prof/AdderTest.Add.profraw -o prof/AdderTest.Add.profdata
llvm-cov export -format=text -instr-profile=prof/AdderTest.Add.profdata ./run_tests > cov/AdderTest.Add.json
The export’s name, without .json, is the test’s key.
Inside a directory you pass whole, a subdirectory is part of the key:
cov/calc/tests/add.json under cov is calc.tests.add.
A merged export of the whole run is one “test” that executed everything: do not pass one.
For cargo, run each test of the test binary on its own (<test binary> --exact <test>)
with its own LLVM_PROFILE_FILE; a cargo nextest wrapper that does this per test is not verified end to end.
C and C++ with GCC.
A .gcda file accumulates every process that runs, and nothing inside it names a test,
so write one gcov JSON file per test you run, named after the test, and delete the .gcda files before the next one.
Compile and link with --coverage, then, for each test:
./run_tests --gtest_filter=AdderTest.Add
gcov --stdout --json-format build/*.gcda > cov/AdderTest.Add.json
rm build/*.gcda
--stdout writes every translation unit’s document to one file (GCC 9 and later).
Without it, gcov writes one <name>.gcov.json.gz per data file instead:
concatenate a test’s files into one (cat *.gcov.json.gz > cov/AdderTest.Add.gcov.json.gz) and pass that.
The name, without .gz, .json and .gcov, is the test’s key, as for an llvm-cov export.
gcov names each file as the compiler saw it: absolute,
or relative to the directory the translation unit was compiled in (current_working_directory in the file),
to which ubCode joins it.
When that directory is another machine’s checkout, map it with --path-map (step 3),
naming the directory as gcov wrote it (zgrep reads the .gz files too):
zgrep -ho '"current_working_directory": "[^"]*"' cov/*.json* | sort -u # "/ci/build/checkout"
ubc codelinks coverage --test-coverage cov --path-map /ci/build/checkout=.
-ffile-prefix-map and -fprofile-prefix-map rewrite the file names gcov writes,
but not current_working_directory (GCC 13),
so a file compiled by a relative path (gcc -c src/calc.c) is still named after the machine’s own directory:
map that directory with --path-map.
To name the same directory on every machine, compile by absolute path with a prefix map to an absolute directory
(a relative target such as =. makes the name relative, and it is then read against current_working_directory,
the build directory of an out-of-source build):
g++ --coverage -ffile-prefix-map="$PWD"=/ci/build/checkout -c "$PWD/src/calc.cpp" -o build/calc.o
Or with lcov: write a tracefile per test with its name, and join them if you like:
geninfo build --test-name AdderTest.Add -o cov/AdderTest.Add.info
lcov -a cov/AdderTest.Add.info -a cov/AdderTest.Sub.info -o tests.info
Each section of a tracefile keeps its TN: test name through lcov -a.
lcov rewrites every character of the name other than a letter, digit or _ as _
(AdderTest.Add becomes AdderTest_Add), and ubCode compares the names the same way (step 2).
A tracefile without test names (coverage.py’s coverage lcov writes none) is one test, named after the file, as an export is.
To run every test in one process instead, reset the counters before each test (__gcov_reset())
and write them after it (__gcov_dump()) with GCOV_PREFIX set to a directory of that test’s own,
which libgcov reads at every dump; copy each .gcno beside its .gcda before running gcov there.
A function already running at the reset (main, the runner’s loop) is then not counted for the test.
Python with pytest. Record which test executed each line, and write the report with its contexts:
pytest --cov=mypackage --cov-context=test --junitxml=junit.xml
coverage json --show-contexts -o coverage.json
A context names the test by its pytest node ID.
coverage.py’s own [run] dynamic_context = test_function works too.
On Python 3.14 and later, coverage.py records no test context unless [run] core = ctrace is set:
the report then warns that it found none.
Step 2: link the tests to their requirements¶
A test of the coverage is matched with a test need:
a test case of a test report (a
testcaseneed, or the case type[test_reports]names), whose class name and name make the test’s key (AdderTestandAddmatch the exportAdderTest.Add;tests.test_opsandtest_add[1-2]match the node IDtests/test_ops.py::test_add[1-2]);or any need whose ID is the key. A tracefile’s test name is compared as lcov writes names:
AdderTest_Addmatches the test caseAdderTestAdd, and the needTC-ADDisTC_ADD. Only those are tests: a specification or another need with links is not listed as a test of the requirements it links to unless a key names it.
The requirements of a test need are the targets of its outgoing links:
every link type except the *_back ones.
A test case’s links to its test file and suites are not requirements.
A test report links its cases to requirements through test properties
(property_link_types):
[test_reports]
property_link_types = { verifies = "verifies" }
Only a test that passed counts.
A test case need carries its result:
the functions its test executed are credited to its requirements only when it is passed.
A matched test case that failed, errored, or was skipped or disabled
is listed under its requirement with the functions its test executed, and credits nothing.
A need that has no result field (one no test report made, matched by its ID) is credited:
its outcome is not known to ubCode, so make sure the coverage you pass comes from passing tests.
A test that matches several test cases credits only when every one of them passed:
a coverage.py test_function context covers every parametrisation of its function,
and the same test can be in two reports, so a failed run’s coverage would otherwise count.
Each such test case is listed under its requirement with why it is not credited.
To follow only some link types, pass --verifies-link-type (repeatable):
--verifies-link-type verifies ignores every other link of a test need.
A type the project does not have is an error that names the project’s link types,
so a misspelt one cannot quietly leave every test without a requirement.
The report names the link types it followed to each requirement.
Step 3: run the report¶
Pass each export, directory or report with --test-coverage (repeatable):
ubc codelinks coverage --test-coverage cov --path-map /ci/build/checkout=.
ubc codelinks coverage --test-coverage tests.info --path-map /ci/build/checkout=.
ubc codelinks coverage --test-coverage coverage.json -f json
Coverage names files as the machine that ran the tests saw them.
--path-map FROM=TO rewrites a path prefix (TO relative to the project root);
when no file of the coverage lands in a codelinks project,
the command warns and prints the --path-map that would fix it.
Relative paths, as coverage.py and cargo write them, are read relative to the project root;
when coverage ran in a subdirectory, map it (--path-map sub=.), since a relative path that happens to name another file of the project
is joined to that file.
Paths the report echoes (not_in_projects, the hint) are the inputs’ own, which can be absolute paths of the machine that collected them.
The section always covers every function of the selected projects, with or without a snapshot (step 4):
--changed-since narrows the functions the report lists, not this section,
and -f lcov refuses --test-coverage, since a tracefile has no place for it.
Step 4: keep the coverage usable after the code changes¶
Coverage describes the code it was collected on. Joined to a newer checkout, a function of the coverage is matched by its position, so an edit that leaves every function on a definition of the same span goes unseen (two functions of the same length swapped, a body changed without moving a line), and any edit that moves a function makes its whole file stale. To join it as it was, write the report in the job that collected the coverage, on that checkout, and keep it with the coverage:
# In the job that collected the coverage, on that checkout, after step 1:
ubc codelinks coverage --test-coverage cov -f json > requirements-coverage.json
# Keep cov/ and requirements-coverage.json together, as one artefact of that job.
# Later, on any newer checkout (CI or a laptop), with both files:
ubc codelinks coverage --test-coverage cov --inventory-at-coverage requirements-coverage.json \
--path-map /ci/build/checkout=.
With --inventory-at-coverage (the snapshot),
each function of the coverage is joined to the function that was at its position when the coverage was collected,
and that function’s coverage counts for the current function with the same symbol ID
only while its code is unchanged (the same content hashes: comments, markers and formatting aside).
Inserted lines, comments and reordered functions no longer make a file stale,
and two swapped functions are each credited with their own tests.
A function whose code changed is stale; one renamed, moved, deleted or whose signature changed is listed as removed
(with the new name when exactly one new function has its code, as a suggestion),
and a function new since the coverage has no coverage data.
Coverage is never carried across a rename, a move or a change of code.
The snapshot must be written with --test-coverage over the same coverage files, byte for byte,
in the job that collected them: a re-exported or regenerated file is refused, and so is a snapshot written without --test-coverage,
narrowed by --changed-since, cut by --status,
or itself computed with --inventory-at-coverage (the refusals say what to do).
That job needs ubc (and, over the file cap, its license, CI and machines without Internet access),
and both runs should use the same ubc version, or the hashes of unchanged functions may differ and read as changed.
Without a snapshot, the positional join of step 3 applies, with its limits.
Step 5: read the result¶
After the findings, the table adds the section (from the C++ fixture of the tests):
Requirements-based coverage: 1 input, 6 tests (4 linked, 0 not passed, 0 ambiguous, 1 with no requirement, 1 without a test need)
records: 8 matched (1 nested), 0 outside the universe, 1 unmatched, 0 ambiguous, 0 stale, 1 not in the projects, 0 malformed
units: 6 executed, 1 not executed, 0 stale, 0 without coverage data
requirement tests executed traced traced+exec scoped gap suggestions
REQ_ADD 2/2 2 2 2 0 0
REQ_HELPER 2/2 3 3 1 2 2
Scoped gaps (traced, not executed by the requirement's tests):
REQ_HELPER cpp:src/calc.cpp#calc::helper(int) not_executed (run by other tests)
REQ_HELPER cpp:src/calc.cpp#calc::unused(int) not_executed
Link suggestions (executed by the requirement's tests, traced to nothing; review, never applied):
REQ_HELPER cpp:src/calc.cpp#calc::helper(double) (only REQ_HELPER's tests run it)
REQ_HELPER cpp:src/calc.hpp#calc::twice(T) (only REQ_HELPER's tests run it)
Tests without a test need: HelperTest.Int
For each requirement:
executedFunctions its tests executed.
traced+execOf those, the ones that trace to it: the expected result.
scoped gapFunctions that trace to it that its tests did not execute. Each says why: not executed (and whether another test ran it), executed only by the requirement’s own tests that are not credited, stale coverage, or no coverage data at all (with a snapshot,
new_since_coveragewhen it is new since the coverage).suggestionsUntraced functions its tests executed: probably part of the requirement, but not linked. The fewer requirements’ tests run a function, the stronger the evidence, so a helper every test runs comes last. They are never applied: trace the ones you agree with by a marker or a symbol field.
Below the table, a line names each test case that matched but did not pass, and how many functions its test executed: those are not counted for its requirement.
-f json writes every list in full, under requirements:
per requirement executed, traced_not_executed, suggestions and tests_not_passed,
and executed_without_requirement for the functions only tests without a requirement ran.
With a snapshot (step 4), the section’s first lines name it and what became of its functions,
with a note for each thing that differs between it and this run,
and below the scoped gaps the table lists the functions whose code changed (with the tests whose coverage was dropped)
and the functions removed since the coverage
(here, the C++ fixture with its two one-line helper overloads swapped since the coverage, each credited with its own test):
Requirements-based coverage: 1 input, 6 tests (4 linked, 0 not passed, 0 ambiguous, 1 with no requirement, 1 without a test need)
snapshot: requirements-coverage.json (written at 1a2b3c4d5e6f, working tree at 9f8e7d6c5b4a): 7 units carried, 0 changed, 0 removed, 0 new since the coverage
records: 8 matched (1 nested), 0 outside the universe, 1 unmatched, 0 ambiguous, 0 stale, 1 not in the projects, 0 malformed, 0 not in the snapshot
-f json holds it all under requirements.snapshot (see Coverage from an older commit).
Reference¶
Inputs¶
The format of each input is read from its content, never from its name, and any input can be gzip-compressed (it is read decompressed, up to 1 GiB).
Input |
Read |
|---|---|
llvm-cov export |
|
gcov JSON |
|
lcov tracefile |
|
coverage.py JSON |
|
Each input is read once, however it is spelled (under its lexicographically first spelling, which an export’s test key comes from),
and an input inside a directory input is read with that directory, with a note.
Two inputs with the same name (job1/coverage.json and job2/coverage.json) are both read,
each named in the report by enough of its path to tell them apart (numbered, (1), (2), when that name is another input’s),
and a test two of them name is one test, with a note.
Inside a directory, the files ending .json, .json.gz, .info and .info.gz are read, in any letter case,
each one test (an llvm-cov export, a gcov JSON file, or a tracefile whose sections without a test name are its own),
all of one format: a directory of two formats, or holding a coverage.py report, is refused.
A symbolic link is not followed (a note says so),
and two files naming one test (a/b.json and a.b.json) are joined, with a note.
How a function of the coverage is matched¶
A coverage function is matched with the function of the same file whose definition starts at or before its first line,
and whose body starts at or after it
(clang starts a function at its {, rustc at fn, gcc at its name, coverage.py at def;
a tracefile at the line its producer wrote).
Two definitions starting on one line are ambiguous, and listed, never guessed.
A function inside another one’s body that the inventory does not list (a closure, a lambda) counts for the enclosing function,
but only when every file the record was read from (each export is one build) also holds a fresh record of that enclosing function itself;
otherwise it is stale (nested_without_function), since an edit may have moved another function’s record into that body.
A function on no definition (one a macro makes) is unmatched.
A function of test code, or of a file the coverage policy leaves out, is outside the universe,
and so is one the Rust build configuration compiles out (Count the Rust code of one build configuration):
the JSON counts those apart, as outside_universe.inactive, present when there is one.
Such a record shows the run compiled code the configuration says is not built,
so the configuration does not describe that run (a tool may have set a name: cargo-llvm-cov sets coverage):
each is listed in compiled_out_records with the unit and its cfg, and the table names it
with the remedy (list the name in cfgs, or set declared_cfgs = "unknown").
A function that no longer fits its definition is stale, and none of its coverage counts:
for llvm-cov, gcov and a tracefile with end lines, when it ends on another line than the definition;
for coverage.py, when one of its lines lies outside the definition or its name differs.
Once one function of a file is stale, the file has changed since that input was collected,
so none of that input’s functions of that file counts either (file_stale).
So does a function of the coverage that starts on no definition,
unless it starts on a #define (or one of its continuation lines) or on a macro invocation the inventory reports,
where a function a macro makes is recorded,
or, for gcov, it is recorded at a single point (start and end on the same line and column:
how gcc records a function a macro expands, a function or lambda written inside a macro’s argument, and a = default one)
where the source still holds, from that line and column, that function’s own = default declaration
(starting there with its name, where gcc records it, and the ( of its parameters on the same line;
as many parameters as the function has, each with the same & or &&, so another constructor of its class is not its own;
then its = default on that line or one of the next seven, before any ;, { or }, as clang-format wraps a long one),
or, when no = default declaration starts there, a name followed by ( (the macro’s invocation);
and so does a stale function of test code or other code the report leaves out.
A function on a line two definitions share (ambiguous) is no such sign.
A shift that lands one function exactly on another of the same length is then not credited,
as long as some other function of the file no longer fits.
Coverage from an older commit¶
With --inventory-at-coverage (step 4), a function of the coverage is matched against the snapshot instead:
with the snapshot’s function whose definition holds its first line and that it lies in
(for llvm-cov, gcov and a tracefile with end lines, it ends at or before the definition’s last line;
for coverage.py, its lines are inside the definition and its name is the function’s or one nested in it).
A function that is the definition’s own (it ends where the definition does, or has its name) is joined to it;
a closure or a lambda inside it counts for it only with the function’s own record in the same file of the coverage, as above.
Two candidates are ambiguous.
A function inside a definition that it does not fit is stale (snapshot_mismatch),
and so is the rest of its file from that input (file_stale): the coverage and the snapshot do not describe the same code.
A function on no definition of the snapshot is unmatched and no sign of an edit:
under a snapshot, unmatched also holds test code and other code the report leaves out inside a counted file
(a Rust #[cfg(test)] module, a private function under visibility = "public") and functions a macro makes.
Then each function’s coverage is carried to the current function with the same symbol ID and the same content hashes.
Since the coverage |
Result |
|---|---|
Nothing, or comments, markers, blank lines, formatting, reordered functions |
Carried: counted as on the day the coverage was collected. |
Two functions of the same length swapped |
Each carried with its own tests. |
A body changed, moved or not |
Stale ( |
Renamed, moved to another file, signature changed, deleted, or no longer in the selected functions |
Not carried ( |
New, or now selected by the coverage policy |
No coverage data; listed under |
A file the snapshot does not list (a project it did not select) |
Not counted ( |
A file the snapshot’s universe left out (test code, generated, |
Outside the universe, as without a snapshot. |
The snapshot is refused (exit code 2, naming the file) when it is not a ubc codelinks coverage -f json report,
was written under another symbol-ID scheme,
was narrowed by --changed-since or cut by --status,
was written without --test-coverage,
was itself computed with --inventory-at-coverage (it then lists the functions of the commit it was written on,
not those the coverage was collected on: keep the snapshot written in the coverage job, never a later report),
or did not read one of the coverage files given now (compared by SHA-256: fewer files now than then is fine).
It is never refused for anything that can only make a function read as changed or absent;
the table notes each instead, and requirements.snapshot.differs says which:
another ubc version or other grammar versions (unchanged functions may read as changed,
and functions another ubc version names otherwise as removed and new),
a symbol pack only one of them uses (that language’s functions are new or removed),
another configuration (functions it left out, or put in other projects, are not carried).
The snapshot’s commit and this run’s (git_head, current_git_head) are named when known,
and stale_at_coverage counts the snapshot’s own stale functions:
not zero when a function joined no definition when it was written,
or when it was written on another checkout than the coverage’s (its own stale list tells which).
Test keys¶
Key |
Matches a need when |
|---|---|
A per-test file’s name (an llvm-cov export, a gcov JSON file, a tracefile’s sections without a test name) |
It is the need’s ID, or the test case’s |
A tracefile’s test name ( |
It is the need’s ID or the test case’s |
A pytest node ID |
It is the need’s ID, or, mangled as pytest writes a JUnit report
( |
A coverage.py |
It is the need’s ID, or the test case’s |
Matching is exact. A test that matches no need is listed with the nearest test-case key, never linked.
A matched test case credits its requirements only when its result is passed;
a need without a result field always does.
A test that matches several needs credits only when all of them do.
A test without a class name (an export named test_init) that matches test cases of several suites or reports is ambiguous:
listed, never credited.
pytest’s --junit-prefix changes the JUnit class names, so node IDs no longer match.
What a test framework writes, and how to name its coverage:
Framework |
Test report case |
Coverage file or |
|---|---|---|
GoogleTest |
|
|
Catch2, Unity, Ceedling |
Not verified end to end |
Name each test’s file or |
Known limits¶
Per function only: no lines, branches or MC/DC.
gcov counts the entries of a function: with
__gcov_reset()between tests in one process, a function already running at the reset is not credited to the test.A tracefile without end lines (
FN:<line>,<name>, which the format allows) finds no stale function by itself: only a function on no definition, or a stale neighbour from the same input, says its file changed.A tracefile has no columns, so a function a macro makes at file scope (one line, on no definition) makes its file stale on every run without
--inventory-at-coverage: read gcov’s JSON instead, or use a snapshot (step 4).GCC lists the inline functions of system headers it compiled (
/usr/include/...): they count asnot_in_projects.One export per test for llvm-cov; Windows file names cannot hold
::, so name an export by the test case’sclassname.caseor by its test need’s ID.Without
--inventory-at-coverage, staleness is positional: an edit is detected when some function of the coverage no longer fits the file (its span or lines no longer match a definition, or it starts on no definition outside a#defineor a macro invocation). Not detected: an edit that leaves every function of the coverage on a definition of the same span (two functions of the same length swapped, a body changed without moving a line), or one that shifts a function onto a#defineor macro-invocation line while the others land on same-span neighbours. For gcov, the same holds for a function recorded at a single point that an edit shifts onto another line holding, at its column, a name followed by((another macro invocation, a call or a declaration), or onto the= defaultdeclaration of a function of the same name and the same parameter shape (the same number of parameters, each with the same&or&&; a defaulted function’s overloads differ in that, but another class’s defaulted operator in the same file, such asoperator==,operator<=>oroperator=, can match both).gcov records a function at a single point only where it knows the column: past column 4096 of a line it writes column 0, and such a one-line function is a span like any other. A single point whose line no longer holds what gcc recorded there, or holds another function’s
= defaultdeclaration, or a line above its own (a comment, an access label, blanks), makes its file stale, and so, on every run without--inventory-at-coverage, does one that never did: a macro invocation or a defaulted function’s name whose(is on the next line (including an out-of-class qualified name wrapped after::or inside<...>, as clang-format wraps a long one), or a= defaultmore than seven lines below the function’s name or separated from its parameters by more than qualifiers, macros and a trailing return type (arequiresclause).A file with a function that legitimately joins no definition (a lambda at namespace scope the inventory does not list, a compiler-made function) is stale on every run (without
--inventory-at-coverage): theunmatchedlist names that function, and eachstaleentry its reason (file_stale).With a snapshot, coverage is evidence about a function’s own code only: an edit to its callers, its callees, the tests, the headers and macros it uses or the build options is not seen. For release evidence, collect coverage on the commit you release.
A snapshot written on another checkout than the one the coverage was collected on (later, or with coverage files brought in from another commit) is not detected: write it in the job that collected the coverage. A report itself computed with
--inventory-at-coverageis refused as a snapshot.There is no
ubproject.tomlkey or editor view for it yet: the editor’s coverage view (See it in the editor) shows trace coverage, not which tests ran the code.