Test reports: the directives¶

sphinx-test-reports turns JUnit-style XML test reports into needs. ubCode supports six of its directives: the three that create needs — test-file, test-suite and test-case — and the two that render a report as tables, test-results and test-env (see Results tables), in reStructuredText and MyST; and test-report, which renders a report through a template, in reStructuredText (see A whole report: test-report). Each one reads a report and creates one need — and a test-file with :auto_suites: (and :auto_cases:) one need per suite (and per case) besides, see A need per suite and per case. Each need renders as a need card where the directive is written and is a need like any other everywhere else: in needs.json, in a needtable, needflow or needextract, in a filter, in :need: references.

The directives are available when the project’s ubproject.toml has a [test_reports] section — an empty table will do; it also adds the testfile, testsuite and testcase need types and the fields their needs carry.

Not supported yet: renaming the directives, tracked in #3845.

A first example¶

With the reports in a reports directory beside the documentation:

[source]
dir = "docs"

[test_reports]
rootdir = "reports"

a page can document a report, one of its suites and one of its cases:

.. test-file:: googletest report
   :file: gtest_data.xml
   :id: TF_GTEST

   The unit tests of the math library.

.. test-suite:: MathTest
   :file: gtest_data.xml
   :suite: MathTest
   :id: TS_MATH

.. test-case:: Subtraction
   :file: gtest_data.xml
   :suite: MathTest
   :case: Subtraction
   :id: TC_SUB
   :links: TS_MATH

   Why this case matters.

The same in MyST:

```{test-case} Subtraction
:file: gtest_data.xml
:suite: MathTest
:case: Subtraction
:id: TC_SUB
:links: TS_MATH

Why this case matters.
```

The TC_SUB card shows the case’s fields (suite, case, classname, result, time, the report file, …), the body written under the directive, and after it the case’s evidence: each failure, error or skip message and text, and its captured System-out and System-err.

The directives¶

All three take the need’s title as their argument, an optional body (the need’s content), and these options:

:file: (required): The JUnit XML report: a path relative to [test_reports] rootdir (see Where the report is read from).
:id:: The need’s id. Without it the id is generated (see Ids).
:status:, :tags:, :links:: As for any need.
:collapse:: true or false (yes and no too, in any case): whether the card’s fields start collapsed.
Every extra_options name of the section: Its value becomes the need’s field of that name.
test-file: One need for the whole report. It also takes the :auto_suites: and :auto_cases: flags (see A need per suite and per case).
test-suite: One need for a suite of the report, named by :suite: (required): the first top-level suite of that name, in report order, or, when no top-level suite has it, the first suite of that name at any depth (a suite before the suites nested in it). A name used under two parents finds the first; there is no way to write a path.
test-case: One need for a test case of a suite, named by :suite: (required, found as for test-suite), and by :case: (the case’s name), :classname: (its class), or both — at least one of them. The first case of the suite that matches, in report order, is the one documented: by name alone when no class name is given, by class name alone when no name is given, otherwise by both.

The fields each need carries¶

test-file: suites (the report’s top-level suites), cases, passed, skipped, failed and errors, summed over them, and the report path as written in :file:.
test-suite: suite (the suite’s name), cases, passed, skipped, failed and errors, the report path, and every suite property whose name is listed in extra_options.
test-case: suite, case and classname (as the directive gives them; absent when not given), case_name and case_parameter (a pytest name test_x[a-b] gives test_x and a-b), result (passed, failed, error, skipped or disabled), time (seconds, 0.0 when the report gives none), style, tr_ and the result — the card’s CSS class, ub-need-style-tr_failed for a failed case —, the report path, the test’s source file and line, and every case property whose name is listed in extra_options.

Like any need, each also records where it is written: the titles of the sections it is under (sections, section_name), the need it is written inside, if any (parent_needs), and the markup it is written in (doctype, .rst or .md).

The report path and the source location are stored under the field names file_option, source_file_option and source_line_option select (file, case_file and case_line by default). A property that property_link_types maps to a link field adds its comma-separated ids to that field — to links beside the ids written in :links:. An extra_options value given on the directive is replaced by a report property of the same name.

The six counters count the cases where the report leaves an attribute out (a googletest report has no skipped attribute), and are never negative.

A need per suite and per case¶

A test-file with :auto_suites: creates, after its own need, one need per suite of the report; with :auto_cases: as well, one need per test case besides. :auto_cases: alone creates nothing more than the file’s need (and is reported, see below).

.. test-file:: googletest report
   :file: gtest_data.xml
   :id: TF_GTEST
   :tags: ci
   :auto_suites:
   :auto_cases:

The page shows the TF_GTEST card, then, for each top-level suite in report order, the suite’s card and the cards of its cases in report order. A suite that holds nested suites is followed by its own cases, then by each nested suite the same way. Every card is a card of the page, anchored at its need id (#TF_GTEST_F64), as a need written by hand would be.

The needs

A suite need has the suite’s name as its title and the fields of a test-suite need; a case need has the case’s name as its title and the fields of a test-case need, with case and classname the case’s own. Each takes the test-file’s :tags:, :status:, :collapse: and extra_options values, and is placed where the test-file is written (sections, parent_needs). A suite need has no body; a case need’s body is its evidence alone. A suite property listed in extra_options is a field of the suite need only, not of its cases.

The links

Every suite and case links to the test-file’s :links:, to the test-file’s need and to every suite it is in; a case also to its own suite. A property that property_link_types maps to links adds its ids, as for a written directive.

The ids

A suite’s id is its parent’s id, _, and the upper-case SHA-1 of its name cut to [test_reports] suite_id_length characters (3 by default): TF_GTEST_F64. The parent is the test-file’s need for a top-level suite and the suite it is nested in otherwise: in pytest_nested_example.xml under a test-file TF_NEST, the suite nested_example nested in parent_testsuite (TF_NEST_42F) is TF_NEST_42F_41F. A case’s id is its suite’s id, _, and the SHA-1 of its class name followed by its name, cut to [test_reports] case_id_length characters (5 by default): TF_GTEST_F64_423A1; with [test_reports] deterministic_case_ids = true it is the deterministic id of the case instead, testcase__MathTest__Subtraction_srmht (see Ids). A test-file without :id: uses its generated id as the parent.

Two needs of one expansion with the same id

The first one is created and the later one is not — for a suite, nor any case or suite inside it — and needs.test_report_duplicate_id says which, with what would separate them:

two suite names whose hashes agree: raise suite_id_length;
two case names whose hashes agree: raise case_id_length, or use the deterministic ids;
two cases whose deterministic ids agree although their names differ (a deterministic id keeps only the last part of the class name, joined to the name with __): switch the deterministic ids off;
the same case in two suites, under the deterministic ids (a merged report of several test runs): switch the deterministic ids off, whose suite-scoped ids keep them apart;
a case whose deterministic id is the test-file’s own :id:: give the file another :id:;
the same suite name twice, or the same case twice in one suite: nothing separates them, and only the first is created.

A suite or case id that another directive also writes is needs.duplicate, as for any two needs. When the test-file’s own id is the one another need takes (a req with that id), the page shows the generic directive box where the test-file is written, followed by the cards of its expansion, which still link to that id — so to the other need — as their parent.

Large reports

A report of tens of thousands of cases expands into one page holding a card per case, which ubc build html writes in seconds. A report of more than a few dozen suites needs a longer suite_id_length: two suite ids are as likely as not to collide from about 75 suites at the default 3 characters, and from about 300 at 4. Above 30,000 search-index records (about one per card), the search index passes its size threshold and build.search_index_large is reported, which fails a default ubc build html: switch the search off ([build.html.search] enabled = false) or ignore the code (ignore = ["build.search_index_large"] under [lint]).

Results tables¶

test-results renders a JUnit report where it is written:

Unit tests
----------

.. test-results:: pytest_data.xml

The argument is the report, read relative to [test_reports] rootdir and kept inside it, exactly as a test-file’s :file: is (see Where the report is read from); a path with spaces is fine. The directive takes no option, and a body under it is ignored.

For every suite of the report it renders a section titled with the suite’s name (unknown when the suite has none), nested under the section the directive is written in; a nested suite is a section inside its suite’s, at every depth, after the suite’s own cases. They are sections of the page: each has an id — the page’s anchor, listed in .. contents:: and the page’s “On this page” list — but no reference reaches it: a MyST [](#id) fragment and sphinx.ext.autosectionlabel do not see a generated section (it has no source line), so it never becomes an autosectionlabel label; a reStructuredText `Suite name`_ on the same page does reach it. Each section holds:

a paragraph Tests: 10, Failures: 0, Errors: 0, Skips: 10 — each count is the suite’s attribute when the report gives one, and otherwise counted from the suite’s cases, its nested suites’ included (sphinx-test-reports prints -1 for a missing attribute);
a paragraph Time: 0.054, the suite’s time as Python prints a number — left out when the suite has no time;
a table with the columns class, name, status and reason, one row per test case of the suite itself, in report order. status is the case’s result (passed, failed, error, skipped, disabled). reason is the message of the failure, error or skip, as a paragraph, then its text — a traceback, say — as a literal block; a case with neither has an empty reason.

Every body row, its cells and the paragraphs in them carry the class tr_<result> (tr_passed, tr_failed, tr_error, tr_skipped, tr_disabled), and every body cell also its column’s name (class, name, status, reason) — the classes sphinx-test-reports gives them, so its common.css keeps working. The table itself has the class ub-test-results. ubCode colours them out of the box, in the light and the dark theme, in the built site and the preview: a tinted row per result, and the status as a coloured pill. To restyle them, write rules for those classes in a stylesheet of your own ([build.html] extra_css, see HTML site); the pill’s colours are the custom properties --ubc-tr-passed, --ubc-tr-failed, --ubc-tr-error, --ubc-tr-skipped and --ubc-tr-disabled. A test-report need’s card (test-file, test-suite, test-case) is tinted the same way, through its tr_<result> style. The PDF draws the tables without row colours.

A report is part of the page that names it: an edited, deleted or newly created report re-indexes and re-renders the page (a path that leaves rootdir is not read and not watched: re-pointing a symlink into the root needs a cold build). Each page reads the report once, however many directives on it name it; two pages naming one report read it once each. There is no row limit. A page showing a report of 50 000 cases (2 000 suites of 25) builds in about 2 seconds and is about 21 MB of HTML (4 200 pages in the PDF), which is what the reader’s browser then has to load; consider splitting such a report.

Environment tables¶

test-env renders a tox-style environment report — a JSON object of environments, each an object of variables:

.. test-env:: tox-report.json
   :env: py312, docs
   :data: platform, python

For every environment it renders a section titled with the environment’s name and a table of its variables, Variable and Data. A number, a string, true, false or null is written as JSON writes it (1.0, 1e+16, false); a list or an object is a JSON block, indented by four spaces, its keys in the file’s order. The table has the class ub-test-env.

:env:: The environments to show, comma-separated, in that order; without it, every environment, in the file’s order.
:data:: The variables to show, comma-separated, in that order; without it, every variable, in the file’s order. A dotted name such as python.version is a variable called that, not a path into an object.
:raw:: One JSON block per environment, of the variables :data: selects, instead of the table.

A name listed twice in :env: or :data: counts once, and an empty entry (a,,b) is skipped. The file is read as UTF-8; a byte-order mark at its start is fine. Like test-results, the argument is read under rootdir and the file is part of the page.

Diagnostics of the tables¶

sphinx-test-reports stops the build on most of these; ubCode reports each one and carries on. A table directive whose report cannot be read keeps the generic directive box; its report is the finding.

needs.test_report_missing (warning): The file does not exist: The given file does not exist: nope.xml.
needs.test_report_unreadable (warning): A report that is not well-formed XML; an environment file that is not valid JSON, or not UTF-8.
needs.test_report_path_outside_root (warning): The argument leaves rootdir; the file is not read.
needs.test_report_json_unsupported (warning): A .json argument to test-results, which reads JUnit XML.
needs.test_report_note (info): A report with no suite (nothing is rendered), or something in it read past, such as a time that is not a number.
needs.test_report_env_shape (warning): An environment file that is not an object of objects (nothing is rendered), or one environment in it that is not an object (it is skipped).
needs.test_report_env_not_present (warning): An :env: name the file does not hold, on the option’s value.
needs.test_report_env_key_not_present (warning): A :data: name that a shown environment does not hold, once per name: option 'nope' is not present in JSON file when no environment holds it, option 'host' is not present in 'py27' environment file naming the environments that lack it otherwise.
needs.test_report_option_invalid (warning): :env: or :data: with no value, or a value given to :raw:; the directive is not rendered.

An option the directive does not take is block.directive_unknown_option, and the directive is not rendered. With sphinx_test_reports switched off in [parse] extensions, both directives are placeholders reported as directive.extension_disabled.

A whole report: `test-report`¶

test-report documents a JUnit report in one go: it fills a report template with its own values and puts the result where it is written — the counts of the report’s suites and cases, tables of its cases, and a test-file with :auto_suites: and :auto_cases:, so a need per suite and per case.

.. test-report:: Unit tests
   :id: REP_UNIT
   :file: pytest_data.xml
   :tags: nightly
   :links: REQ_1

   The nightly run of the unit tests.

:id:: The id of the test-file need the template creates, and the tag its needs are found by. Required: a report has no generated id.
:file:: The report, read relative to [test_reports] rootdir as a test-file’s :file: is (see Where the report is read from). Required.
:tags:: Tags of the generated needs; the report’s :id: is added after them, which is how the template’s counts and tables find its needs.
:links:: Links of the generated test-file need.
:status:, :collapse:: Accepted and not used, as in sphinx-test-reports (:collapse: must still be true or false, which sphinx-test-reports checks too).

The body becomes the generated test-file’s content. The directive itself creates no need: the test-file in its template does, as one written by hand would, so the report’s needs, their links, its missing-report warning and its duplicate ids are a test-file’s. Two test-reports with one :id:, or a test-report and a test-file with one, are needs.duplicate.

A test-report is written in a reStructuredText page: its template is reStructuredText, and a Markdown page cannot host it (needs.test_report_markdown_host; nothing is rendered). A section in the template nests one level under the section the directive stands in, as a section in an .. include::d file does.

The template¶

Unset, [test_reports] report_template is the template ubCode ships, which is sphinx-test-reports’ own with :max_items: 0 under each of its three tables, so they list every case, as a Sphinx build’s do (a project’s own template follows [needs] views_max_items):

**Test file**: ``{file}``

**Template used**: ``{template_path}``

**Statistics**

| Test suites: :need_count:`'{id}' in tags and type=='{suite_need}'`
| Test cases: :need_count:`'{id}' in tags and type=='{case_need}'`
| Failed test cases: :need_count:`'{id}' in tags and 'failed' == result and type=='{case_need}'`
| Skipped test cases: :need_count:`'{id}' in tags and 'skipped' == result and type=='{case_need}'`

**Test cases**:

.. needtable::
   :filter: '{id}' in tags and type == '{case_need}'
   :columns: id, title, result
   :style_row: tr_[[copy('result')]]
   :max_items: 0

**Failed test cases**:

.. needtable::
   :filter: '{id}' in tags and 'failed' == result
   :columns: id, title, result
   :style_row: tr_[[copy('result')]]
   :max_items: 0

**Skipped test cases**:

.. needtable::
   :filter: '{id}' in tags and 'skipped' == result
   :columns: id, title, result
   :style_row: tr_[[copy('result')]]
   :max_items: 0

**Imported data**


.. {file_type}:: {title}
   :id: {id}{links_string}
   :tags: {tags}
   :file: {file}
   :auto_suites:
   :auto_cases:

   {content}

report_template names a template of your own, relative to the ubproject.toml that sets it, decoded in [test_reports] import_encoding (utf8 unset; a byte-order mark is dropped). The path is trusted as the configuration is: a template outside the project root is read. An edited, deleted or newly created template re-renders the pages that use it.

A template is text with ten keys, each written {key}:

{file}: :file: as written.
{id}: :id:.
{title}: The argument.
{content}: The body. Its first line replaces the key, every further line takes the indentation of the line the key is on, so {content} inside a directive stays inside it; a blank line stays empty.
{tags}: :tags: followed by ; and the id, or the id alone.
{links_string}: A :links: option line at three spaces — a line break, three spaces, :links: and the value — or nothing, so :id: {id}{links_string} adds the option to the generated directive only when there are links.
{file_type}: test-file.
{suite_need}, {case_need}: The testsuite and testcase need types ([test_reports] suite / case).
{template_path}: The template, relative to the ubproject.toml that set it; test_report_template.txt for the shipped one.

{{ and }} write a brace. That is the whole language: ubCode reads nothing else of Python’s str.format. A format spec ({id:>8}), a conversion ({id!r}), an index or an attribute ({content[0]}, {id.upper}), a numbered or empty field ({0}, {}) or a brace without its pair is needs.test_report_template_invalid, naming the construct, its line and column, and nothing of the template is rendered.

Note

A template that uses a format spec, a conversion, an index or an attribute renders in a Sphinx build and is refused here: write the value plainly. A template language with conditions and loops is proposed for sphinx-test-reports in sphinx-needs#2142.

Diagnostics of `test-report`¶

Each is reported on the directive (an option’s on the option), and the directive keeps the generic box:

needs.test_report_template_missing (warning): The configured template does not exist.
needs.test_report_template_unreadable (warning): The template cannot be read, is not valid in import_encoding, or import_encoding names no encoding ubCode knows (the WHATWG encoding labels: latin1, not latin-1).
needs.test_report_template_invalid (warning): The template uses a construct outside its language (above), or writes a test-report itself (one in the directive’s body renders, as a nested report).
needs.test_report_markdown_host (warning): A test-report in a Markdown page.
needs.test_report_option_missing (warning): No :id: (ID must be set for test-report.), or no :file:.
needs.test_report_option_invalid (warning): An option written with no value, or a :collapse: that is not true or false.

An option the directive does not take is block.directive_unknown_option. The template’s own reStructuredText mistakes are reported too: one in a line the template wrote names the template and its line (in the test-report template templates/report.txt:12: Unknown directive …), at the directive; one in the body is at the body’s own line. The report the template’s test-file reads is that test-file’s: its warnings are a test-file’s. With sphinx_test_reports switched off in [parse] extensions, test-report is a placeholder reported as directive.extension_disabled.

Ids¶

A directive without :id: gets the id sphinx-test-reports gives it: the need type, then the upper-case SHA-1 of the title, cut to [needs] id_length characters — testfile86527 for a test-file titled My test data. (The type, not the type’s prefix: that is sphinx-test-reports’ own rule.) With [needs] id_from_title, the upper-cased title comes before the hash.

With [test_reports] deterministic_case_ids = true, a test-case without :id: instead gets the id of the case it found — testcase__MathTest__Subtraction_srmht — the id the test-reports build needs converter writes for the same case. An :id: written on the directive always wins.

Both generated forms contain lower-case letters, so a generated id is not checked against [needs] id_regex. An :id: written on the directive is.

Where the report is read from¶

:file: is read relative to [test_reports] rootdir, which is relative to the directory of the ubproject.toml that sets it, and is [source] dir when it is not set. The report may live outside the documentation, as in the example above, but :file: must stay inside rootdir: a path that leaves it (../secret.xml, an absolute path elsewhere, a link pointing out of it) is not read (needs.test_report_path_outside_root). To read reports from somewhere else, point rootdir there.

A report is part of the page that reads it: when the report changes, the page is indexed and rendered again, so a ubc build html after a new test run shows the new results. Each report is read once per index pass, however many directives name it.

A :file: ending in .json is not read: JSON reports are not supported yet ([test_reports] json_mapping is accepted but not applied).

Diagnostics¶

sphinx-test-reports stops the build on most mistakes in these directives. ubCode reports each one on the directive, and carries on:

needs.test_report_option_missing (warning): No :file:; a test-suite or test-case without :suite:; a test-case with neither :case: nor :classname:. No need is created.
needs.test_report_option_invalid (warning): A :collapse: that is not true or false; an option written with no value (every built-in option but the two flags needs one); a value given to the :auto_suites: or :auto_cases: flag. No need is created. Also :auto_cases: without :auto_suites:, where the test-file need is still created and nothing is expanded.
needs.test_report_path_outside_root (warning): A :file: that leaves rootdir. No need is created.
needs.test_report_missing (warning): The report does not exist, or is not a file that can be read (a directory, say). No need is created.
needs.test_report_json_unsupported (warning): The report is a JSON report (its :file: ends in .json), which is not read yet. No need is created.
needs.test_report_unreadable (warning): The report is not well-formed XML, or not valid in its encoding. No need is created.
needs.test_report_suite_not_found (warning): No suite, at any depth, has the :suite: name. No need is created.
needs.test_report_case_not_found (warning): No case of the suite matches :case: and :classname:. No need is created.
needs.test_report_need (warning): The need could not be built from the values read, for example a field the project declares with a type the value cannot take; no need is created. Also a field default whose predicate could not be evaluated on a need that was created: the field takes its plain default.
needs.test_report_note (info): Something in the report was read past, such as a time that is not a number. The need is created. A note is about the report, so it is reported once, on the first directive that reads the report, in the order the pages sort by path.
needs.test_report_duplicate_id (warning): Two needs of one :auto_suites: / :auto_cases: expansion have the same id: the later one is not created (see A need per suite and per case).
needs.test_report_id_shadows_section (warning): A need a directive created has the same id as the anchor of a section on its page — a section titled testcase__usage_test__usage_test_sxckt beside that case’s need, which only the lower-case ids can match. The built page gives the id to the need card and <id>-1 to the section (and -2 to any later section with the same title), while :ref: links and objects.inv still point at <id> (and <id>-1), so each lands one element early. Rename the section; or, for the directive’s own need, give it an :id: of your own; for a suite of its expansion, give the test-file an :id: of its own (a suite’s id is built on it); and for a case of its expansion, switch [test_reports] deterministic_case_ids off. Reported until the index reserves these ids as the build does (#3903).

An option the directive does not take is block.directive_unknown_option, and a directive with no title is block.directive_no_argument; either way no need is created — unlike a need directive, which keeps the need and drops an unknown option. An :id: that id_regex refuses is needs.id_regex. Two directives with one id are needs.duplicate.

A directive that creates no need keeps the generic directive box on the page; the finding above is its only report.

Everywhere ubCode differs from a Sphinx build on these directives is listed in The test-report directives.