Test reports

The [test_reports] section holds the settings of sphinx-test-reports, which turns JUnit-style XML test reports into needs. sphinx-test-reports (2.0 and later) and its converter, the test-reports build needs command of the ub-test-reports package, read the same section, so one ubproject.toml describes the project to all three tools.

When the section is present — even as an empty [test_reports] table — ubCode adds the three sphinx-test-reports need types and the fields their needs carry to the project’s needs configuration, so a needs.json written by the converter imports with nothing else declared. The section’s presence is the switch: ubCode does not read conf.py, so a project that loads sphinx-test-reports there needs a [test_reports] table (an empty one will do) for ubCode to add the same types and fields — or the extension’s name in [parse] extensions, extensions = ["sphinx_test_reports"], which means the same as an empty table when there is none (its old name, sphinxcontrib.test_reports, is read as the new one and reported at information level).

"sphinx_test_reports" = false in [parse.extensions] switches the extension off and keeps the section, including one a project inherits through extend; an explicit false under either name beats an explicit true under the other. Switched off, the extension is withdrawn whole, as an unloaded Sphinx extension is: its need types and fields are not added, so a needs.json written by the converter imports with Invalid need type; its five directives still parse with their options, and each renders as a labelled placeholder with a directive.extension_disabled warning naming the entry to set back to true — unless a [[needs.types]] entry or a [parse.extend_directives] entry of the same name owns it, as it then does. The switch itself is reported once, at information level (config.test_reports_switched_off).

The five directive names belong to sphinx-test-reports while it is switched on: a [[needs.types]] entry whose directive is test-file, test-suite, test-case, test-results or test-env keeps its need type, but .. test-case:: reads a test report, and the configuration says so (config.conflicting_directive_registration). Give such a type another directive. Switched off, or without the section, the project’s type owns the name. The directives are read under their own spelling only: .. Test-Case:: is not a test-report directive (it keeps the generic placeholder), where a Sphinx build stops on it.

Note

Of the sphinx-test-reports directives, test-file, test-suite and test-case are supported, one need each, and a test-file’s :auto_suites: / :auto_cases: expansion into a need per suite and case, and so are the test-results and test-env tables and the test-report directive with its report template: see Test reports: the directives. ubCode reads the section, adds the vocabulary, and checks the settings.

Minimal example

[test_reports]
extra_options = ["TestType"]

[test_reports.build.needs]
project = "demo"
link_properties = { PartiallyVerifies = "links" }

This is enough to import the converter’s output: the testcase need type, the test-report fields and the TestType field come from the section, and the converter reads extra_options and link_properties from the same file.

What the section adds

Need types. The file, suite and case entries each add a need type, named by the entry’s type (testfile, testsuite and testcase by default), with its name as the title and its prefix, color and style. They are added after the project’s own types — the default types, or the [[needs.types]] the project declares instead — and never re-enable a default type the project replaced. Each also makes a need directive of that name (.. testcase::), as in a Sphinx build.

A type the project already has wins. If [[needs.types]] already has an entry whose directive is one of these names (or one of the default types has the name, when the project declares none), that entry is used unchanged, and if its title, prefix, color or style differs from the section’s, config.test_reports_type_declared names both values. (A Sphinx build stops here with “already exists as need type”.)

Fields. Every field a test-report need carries is added as an extra field:

Field	Type	Holds
suite, case, case_name, case_parameter, classname	string	the test’s suite, name, parameter and class
result, result_text	string	the result (passed, failed, error, skipped, disabled) and a one-line failure message
time	string	the execution time in seconds, as the report gives it
remote_url	string	the test’s source in the remote repository
suites, cases, passed, skipped, failed, errors	integer	the counters of a test-file or test-suite need, so failed > 0 compares numbers
file, case_file, case_line	string	the report path and the test’s source location, under the names file_option, source_file_option and source_line_option choose
each extra_options entry	string	the test property of that name

A field the project already declares is kept as declared. Declared with another type, it is kept too, and config.test_reports_field_type_mismatch says the test-report values may not fit it — except a numeric field declared as a type its values fit: a number (time, [needs.fields.time] schema = { type = "number" }) or a number or an integer (the source-line field, case_line). Their values are numeric strings, which the import converts, so time > 0.1 compares numbers; time declared as an integer is reported, since its values are fractional seconds. Any other string field the section adds, declared as a number, is reported: its values cannot convert, and every imported need would be refused. The same code reports a field the section adds whose name the project uses for a link type.

An extra_options name the section adds itself (failed, suite, …) changes nothing. A name that cannot be a field — an empty one, a name [needs.fields] itself refuses or warns about (id, type, content, var), a link type, or a core field of another type such as tags — is ignored and reported as config.test_reports_extra_option_reserved.

The fields are added before the [needs] features that name a field are read, so [needs.global_options], string links and card-layout field: elements can name them with nothing else declared.

Link fields are not added. links always exists; any other field named as a property_link_types target must be a declared link type ([needs.links]), and config.test_reports_link_target_undeclared reports one that is not. (A Sphinx build’s test-report directives stop there with InvalidNeedException.)

ubc config shows the types and fields the section added, under needs, and the resolved section itself under test_reports.

Options

file

Type: table or array of 6 strings (default: directive = "test-file", type = "testfile", name = "Test-File", prefix = "TF_", color = "#ffffff", style = "node")

The test-file directive and the need type it creates. Two spellings are accepted: a table, where each key left out takes its default,

[test_reports.file]
type = "testfile"
prefix = "TF_"

or the six strings in the order directive, type, name, prefix, color, style, as tr_file is written in a conf.py:

[test_reports]
file = ["test-file", "testfile", "Test-File", "TF_", "#ffffff", "node"]

An array of any other length, a table key other than these six, or a value that is not a string is an error. The table is the clearer spelling, since six strings in a row cannot be told apart.

Renaming the directive is not supported yet: the setting is kept, the directive keeps its default name, and config.test_reports_rename_unsupported says so.

suite

Type: table or array of 6 strings (default: test-suite, testsuite, Test-Suite, TS_, #cccccc, folder)

The test-suite directive and need type, spelled as file is.

case

Type: table or array of 6 strings (default: test-case, testcase, Test-Case, TC_, #999999, rectangle)

The test-case directive and need type, spelled as file is. When [test_reports.build.needs] is present, its type must equal that table’s need_type (each compared at its default when unset): otherwise the configuration is refused, because the converter’s needs.json and the build would describe different need types.

file_option

Type: string (default: "file")

The need field holding the path of the test report.

source_file_option

Type: string (default: "case_file")

The need field holding the test’s source file.

source_line_option

Type: string (default: "case_line")

The need field holding the test’s line in its source file.

The three field names must each be a field of their own: naming a field every test-case need already has (case, result, status, links, …), or two of them naming the same field, refuses the configuration. An empty string means the default.

rootdir

Type: string (default: the source directory)

The directory relative report paths are read from. A relative path is taken from the directory of the ubproject.toml that sets it. Unset, ubCode uses [source] dir (or the ubproject.toml’s own directory when that is unset); a Sphinx build uses its configuration directory — the one holding conf.py — instead. Set it whenever conf.py is not in the source directory, so that both tools read reports from the same place.

report_template

Type: string (default: the template shipped with sphinx-test-reports)

The template a test-report directive renders (A whole report: test-report). A relative path is taken from the directory of the ubproject.toml that sets it (a -c value from the working directory, as every -c path is). The template’s {template_path} names it relative to the ubproject.toml that set it — the project’s own ubproject.toml for a -c value — so a page reads the same wherever it is built. The path is trusted, as the configuration is: a template outside the project root is read. Unset or "", the template shipped with ubCode — sphinx-test-reports’ own, listing every case. ubCode formats a template with {key} for its ten keys and the escapes {{ and }}, and nothing else of Python’s str.format; A whole report: test-report lists the keys.

suite_id_length

Type: integer (default: 3)

How many characters of a suite’s hash an automatically generated suite id keeps: the id of each suite need of a test-file’s :auto_suites: expansion (A need per suite and per case). Raise it when needs.test_report_duplicate_id reports two suite names with one id.

case_id_length

Type: integer (default: 5)

How many characters of a test case’s hash an automatically generated case id keeps: the id of each case need of an :auto_cases: expansion, unless deterministic_case_ids is on.

Both lengths are non-negative integers; a negative one is refused (sphinx-test-reports accepts it).

import_encoding

Type: string (default: "utf8")

The text encoding of the report template.

extra_options

Type: array of strings (default: [])

Test properties that become need fields: each name is added as a string field, and a <property> of that name is copied into it.

property_link_types

Type: table of strings (default: {})

Test properties whose values become links, as a table from a property name to the link field it fills:

[test_reports]
property_link_types = { PartiallyVerifies = "links" }

A comma-separated property value such as "REQ_1, REQ_2" gives one link per id. links is always available; any other target must be a declared link type.

json_mapping

Type: table (default: the sphinx-test-reports mapping)

How the fields of a JSON test report are found. Accepted and kept, but JSON test reports are not read yet: a directive whose :file: ends in .json reports needs.test_report_json_unsupported.

deterministic_case_ids

Type: boolean (default: false)

Whether automatically generated test-case ids use the deterministic scheme, <type>__<class>__<name>_<five letters> (the one the converter always writes, such as testcase__MathTest__Addition_hcuyy), instead of the legacy hashed one — for a test-case without :id: and for each case need of an :auto_cases: expansion.

[test_reports.build.needs]

How test-reports build needs writes a needs.json. The converter reads this table; ubCode checks it, so a mistake is caught by whichever tool reads the file first.

project

Type: string (default: "")

The project written into the needs.json.

version

Type: string (default: "1.0")

The version the needs are written under.

need_type

Type: string (default: "testcase")

The need type every test case gets. Must equal the type of case.

tags

Type: array of strings (default: [])

Tags added to every test-case need.

link_properties

Type: table of strings (default: unset)

The converter’s spelling of property_link_types. ubCode reads it as an alias, deprecated from the start: when property_link_types is unset, link_properties is used in its place; when both are set and differ, property_link_types is used and config.test_reports_link_properties_differ reports the difference. sphinx-test-reports is merging the two keys into property_link_types (sphinx-needs#2055); until a converter release reads that key, write the mapping here, where the converter reads it too. Both names in each pair must be non-empty.

remote_url

Type: string (default: "")

The repository URL the converter makes source links from.

commit

Type: string (default: "")

The commit the source links point at. A CI job usually keeps remote_url in the file and passes the commit on the converter’s command line (--commit), which is why ubCode does not ask for the two together.

url_pattern

Type: string (default: "{base}/blob/{commit}/{file}#L{line}")

The source-link pattern, made from the placeholders {base}, {commit}, {file} and {line}.

ubCode never makes a source link itself. It checks url_pattern with the converter’s own rule — a pattern that cannot make a link is wrong whatever the command line passes — and reports a problem as config.test_reports_url_pattern, a warning, where the converter refuses to run. Whether remote_url and commit are given together is the converter’s to check, with its flags in hand.

How the section is read

Unknown keys

A key ubCode does not read, in [test_reports], [test_reports.build] or [test_reports.build.needs], is ignored and reported as config.test_reports_unknown_key, one finding per table naming every such key and the supported ones. The rest of the table is read, so a ubproject.toml written for a newer sphinx-test-reports still loads. sphinx-test-reports reports unknown keys the same way.

Wrong types

A known key of the wrong type — case_id_length = "5", extra_options = "TestType" — fails the configuration, with the position of the value. So do the two rules above that span keys: the field names of file_option, source_file_option and source_line_option, and case’s type against need_type. The messages for the file, suite and case spellings and for the two rules are sphinx-test-reports’ own; other keys report the configuration reader’s own type message.

Paths

rootdir and report_template are taken from the directory of the ubproject.toml that sets them, also when it is a file another configuration extends; a -c value is taken from the working directory.

Shared files

Through extend the section is inherited key by key, and a file/suite/case table folds key by key too (a six-string array is one value). sphinx-test-reports and its converter read only the nearest ubproject.toml, so repeat the section there when the project is also built with Sphinx.

Precedence

A -c override (or UBCODE_CONFIG_OVERRIDE) wins over the ubproject.toml, which wins over the default:

$ ubc check -c 'test_reports.case_id_length = 8'

A Sphinx build reads the same section with -D in place of -c, and a tr_* value in its conf.py between the two; ubCode reads no conf.py, so move such values into the section.

The section’s eight findings are listed with the other configuration codes in Linting, and The [test_reports] section lists where ubCode and a Sphinx build read the section differently.