Templates: generating content from data¶

VS Code Any LSP editor CLI Experimental

The ub-template directive renders a Jinja template with data you declare, and inserts the result into the page as if you had written it there. Its commonest use is generating needs from a data file: the generated needs are ordinary needs — in needs.json, linkable with :need:, listed by a needtable — and they sit in the page’s section that holds the directive. A template you use often can also become a directive of your own (Declare a template directive of your own).

Both are ubCode’s own, and the sphinx-ub-directives Sphinx extension reads the same ubproject.toml tables, so the same project builds with Sphinx too; see Using the same project with Sphinx.

Switch it on¶

The ub-template directive renders only when it is switched on, because rendering a template is something a project opts into:

[parse]
ub_template_enabled = true

Until then a ub-template directive renders nothing and warns (template.switched_off). Declared template directives need no switch: declaring one is the opt-in.

Generate needs from a data file¶

Put the data in a JSON or TOML file under the source directory, with a table (an object) at its top level, for example data/reqs.json:

{"reqs": [
  {"id": "REQ_001", "title": "Boot in under a second", "status": "open"},
  {"id": "REQ_002", "title": "Never lose data", "status": "closed"}
]}

Name it as a context in ubproject.toml:

[parse.template_contexts.reqs]
file = "data/reqs.json"

A context can also be written inline, as a table of values:

[parse.template_contexts.team]
lead = "Ada"
members = ["Bob", "Cy"]

Write the template, in the page itself (the directive’s body) or in a file (:file:, relative to the page, or to the source directory with a leading /):

.. ub-template:: reqs

   {% for r in reqs %}
   .. req:: {{ r.title }}
      :id: {{ r.id }}
      :status: {{ r.status }}

   {% endfor %}

In a MyST page, write it as a fence; the rendered text is then parsed as MyST:

````{ub-template} reqs
{% for r in reqs %}
```{req} {{ r.title }}
:id: {{ r.id }}
```
{% endfor %}
````

Editing the data file, the template, or a template it includes re-renders every page that uses it, in ubc build and in the editor (after the next re-index).

What a context holds¶

A context is data only: strings, booleans, integers, floats, arrays and tables. An array holds values of one type ([1, 2], ["a", "b"]) or tables (rows, as reqs above); a null, a date or time, a float that is not a finite number (TOML’s inf and nan), an integer outside the signed 64-bit range, data nested more than 64 levels deep, an array mixing types ([1, "a"], [1, 1.5]) and an array of arrays are refused, naming where (template.context_invalid), and the directive renders nothing. YAML files are not read: use JSON or TOML. A file that starts with a byte-order mark is not read either.

A template meets a table’s keys in the order the data writes them ({% for key, value in d|items %}), whether the table comes from a file, an inline context or :ctx:, and a table the template builds itself keeps the order it was built in; only dictsort sorts.

Every template also sees var: the project’s variant data, the same values a .. if:: directive evaluates (Variant data), as var.cpu or var["cpu"]. A key it does not define is an error; test for one with {% if var.cpu is defined %}. A context may therefore not be named var (config.template_context_reserved), and a ub-template context or :ctx: may not define var, loop or self at its top level (template.context_invalid: the engine gives every template those names); a top-level key no template can name (true, false, none, True, False, None, in) is skipped. Unlike a table in a context, var iterates its keys in sorted order.

Options¶

The directive’s argument names a context; it is optional.

:file:: Render this template file instead of the body. Relative to the page, or to the source directory with a leading /; it must be inside the source directory.
:ctx:: More variables, as an inline JSON object; they override the named context’s.
:debug:: Also show the rendered text, as a literal block after the output.
:raw:: Do not parse the rendered text: emit it as raw content of this output format, for example :raw: html, as a raw directive would.

A template’s {% include %}, {% import %} and {% extends %} name files relative to the source directory.

Every file name a template directive reads — :file:, a declared template, a context file, an include — follows one rule: it is /-separated (a \ is refused), .. is resolved as the file system resolves it (sub/../t.j2 names a file only when sub/ exists), and the result must lie inside the source directory, symbolic links followed.

Declare a template directive of your own¶

A template you use often can become a directive with a name, an argument and options of its own, declared in ubproject.toml. It needs no switch: declaring it is the opt-in. A template directive takes three pieces: the template, its declaration, and a page that uses it.

1. Write the template, a file under the source directory:

_templates/req_row.rst.j2¶

.. req:: Owned by {{ options.owner }}
   :id: {{ argument }}
   :status: {{ options.status }}

2. Declare the directive, naming that file:

ubproject.toml¶

[parse.directive_templates.req-row]
template = "_templates/req_row.rst.j2"   # under the source directory; or body = """…"""
description = "One requirement row"
argument = { required = true, description = "Requirement id" }
content = false                           # whether the directive takes a body

[parse.directive_templates.req-row.options]
owner = { required = true, description = "Who owns it" }
status = { choices = ["open", "closed"], default = "open" }
draft = { flag = true }

The template sees exactly argument, options (the declared options in the order they are declared, defaults applied; a flag is true or false), body (when content is true), contexts (every [parse.template_contexts] entry, by name, in the order ubproject.toml writes them) and var.

3. Use it in a page, like any directive, with completion and hover for its options in the editor:

index.rst¶

.. req-row:: REQ_010
   :owner: Ada

which renders as if the page held (the status is open, the option’s default):

.. req:: Owned by Ada
   :id: REQ_010
   :status: open

The signature is closed: an option that is not declared, a missing required option or argument, a value outside choices, a flag written with a value, or a body on a directive that takes none is reported on the use (template.signature), which then renders nothing. Directive names are case-insensitive, as everywhere: .. Req-Row:: is .. req-row::; in a reStructuredText page option names are too (:Owner: is :owner:), as in docutils, so a declared option name may not have capitals. A declaration whose name, in any case, is already a directive (built in — Note is the built-in note —, a need type, a [parse.extend_directives] entry, a directive preset, or a sphinx-test-reports directive), that differs only in case from another declaration, that gives both or neither of template and body, or whose fields have the wrong shape (template, body, markup and description are strings, content is a boolean, argument a table, and in an option default is a string, choices an array of strings, required and flag booleans; a flag cannot be required), is reported when the configuration loads (config.directive_template_invalid) and ignored; the rest of the configuration still loads. A key no declaration has (a typo such as contnet) is reported (config.directive_template_unknown_key), and the declaration is kept.

A declared template directive has the ub-template directive’s limits; its findings in the rendered text are reported on the use, naming the template line.

Markup, and no headings in the output¶

A template’s output is parsed in the template’s own markup. A declared template’s markup is markup = "rst" or markup = "md", or, unset, its template file’s name (*.rst.j2, *.md.j2), else reStructuredText; a ub-template directive’s :file: named *.rst.j2 or *.md.j2 is that markup, and any other template is the page’s.

So one template can serve both kinds of page: a reStructuredText need-row template generates its need in a MyST page too. Cross-markup output may not hold the constructs that register in the whole document, though: a substitution definition, a footnote (numbered by hand, auto-numbered, or a MyST [^1]) or a citation in output used in a page of the other markup is reported (template.markup_construct), and that use renders nothing. Markdown output may not start with front matter either, in any page (only a page’s own source holds it): it is reported the same way.

Template output may not contain a heading, in either markup, from any template, anywhere in the output — inside a note, a need or a list item it writes included. A use whose output holds a section title is reported (template.heading) and renders nothing. The output is body content where the directive stands, so a need it generates sits in the page’s section that holds the directive. A rubric is no section, and is allowed: use one for a generated title. (Headings inserted into a page reshape its sections differently in each tool’s splice; they are left for a later version, with a design of their own.)

Where findings are reported¶

A template error in a body (an undefined value, a syntax error) is reported on the body line it is on.
A template error in a :file: template is reported on the directive, naming the template file and line.
A finding in the rendered text — an unknown role, a malformed directive — is reported on the directive, naming the line of the rendered text it is on; in a body template, on the body line that wrote it.
Needs the template writes are positioned on the directive (on the body line that wrote them, in a body template).

In a body template a line’s position is exact up to its first {{ … }}: text after a value is positioned as if the value were as long as the expression that wrote it. A line a macro, a {% filter %} or a {% call %} block wrote is positioned on the call.

A Sphinx build of the same project reports every finding about a use at the directive’s line, and positions the needs the template generates there too (see Using the same project with Sphinx).

Use :debug: to see the rendered text.

The template language¶

Templates are minijinja 2.24, configured exactly as the sphinx-ub-directives Sphinx extension configures it, so a template renders the same text with either tool:

undefined values are errors (test with is defined); values are not escaped; a template file’s one trailing newline is dropped;
Python methods on values work (d.get(k), d.items(), s.upper()), except that s.count('') (an empty substring) is an error;
{% break %} and {% continue %};
the filters filesizeformat, urlencode, truncate, pluralize, striptags and wordwrap (with a positional width, |wordwrap(40), as well as width=), and the functions joiner() and cycler();
names may be non-ASCII ({% set größe = 2 %});
there is no debug(), and nothing reads the clock, randomness or a file (now, random and lipsum are unknown).

Limits¶

Security. A template directive is rendered whenever its page is read: by ubc check and ubc build, and in the editor in the whole-project index pass that runs when a project is opened and in the analysis of the page on every keystroke, whether or not a preview is open. So opening a repository renders the templates it contains, with no trust prompt. What makes that safe is that a template is

sandboxed by construction: it reads only the declared data and the template files under the source directory — no environment variables, no clock, no network, no programs, no other documents — because the engine is given nothing else to reach;
bounded, by the limits below, which are the same in the editor and in a build.

Rendering is also opt-in per project (ub_template_enabled is off until ubproject.toml switches it on, and a declared directive exists only once declared), which keeps templates from running by accident, but the switch lives in the repository itself, so it is no defence against a hostile one. VS Code keeps the extension disabled in a workspace you have not trusted.

Each limit is reported on the directive, which then renders nothing:

the rendered text may be at most 1 MiB (template.output_limit);
no single value a template builds — a string, a list, a map — may be larger than 1 MiB (template.value_limit); operations that can multiply their inputs (~, + and * on strings, join, replace, indent, batch, slice, format widths, wordwrap, and the Python string methods replace, join and format) are refused before the value is built;
a render may spend at most ten million units of work — one per template instruction, plus one per 4 bytes of every value it builds (template.compute_limit); a template generating 300 needs from a data file spends about 14,000. Calls nest at most 500 deep, minijinja’s own limit (template.compute_limit);
the template and data files it reads must be regular files of at most 16 MiB (template.template_invalid, template.context_invalid);
in one page, after a template has hit a limit with three different inputs (different data, or a different :ctx:), its later uses in that page with yet other inputs are not rendered (template.template_disabled, naming the first hit on the same page). This can skip a use that would have rendered; a use with inputs that already failed reports its limit again.

These limits are bounds, not part of what a template means: which template a Sphinx build stops, and how fast, can differ, but a template under them renders the same text with either tool. They are the same in the editor and in every build, so the editor reports what ubc check reports. The editor additionally stops a single render that runs for more than two seconds (template.cancelled, editor only; ubc check renders it in full).

A template used with the same inputs on many pages is rendered once per build (or per index pass) and its result reused, so a runaway template costs one render for each different input it is given, and at most three in any one page. Not charged before it is built: tojson with a large indent (its output is charged once built, so it still counts towards the work limit).

Output reads only what its template declared. The rendered text may not contain another template directive (template.nested_expansion) or a directive that reads a file or other outside content (template.output_reads_file): include, literalinclude, csv-table and raw with :file: or :url:, needimport, needreport with :template:, src-trace, mermaid, uml, plantuml and graphviz given a file, needservice, the auto* directives of sphinx.ext.autodoc, the sphinx-test-reports directives, and a need directive given :template:, :pre_template:, :post_template: or :jinja_content:. A directive preset is judged as its base written with the preset’s options: a preset of include, or one that defaults a need’s :template:, is refused too. (A template directive is no preset base: base = "ub-template" is refused when the configuration loads.) Such a directive renders nothing and is reported; the rest of the output renders. A diagram written inline, an image and a figure are allowed: an image is linked as it would be on any page, not read into it.

The other findings are template.render_error (the template could not be rendered), template.context_unknown and template.context_invalid (a context that is not declared, or whose file is missing, malformed, not JSON or TOML, not plain data, not a regular file, too large, or outside the source directory), template.context_unavailable (see below), template.template_missing, template.template_invalid and template.template_outside_root (the :file:, or a declared template), template.heading and template.markup_construct (see Markup, and no headings in the output), template.option_invalid, and template.signature (a declared directive used outside its declaration).

Values are inserted as they are. A value containing markup — **bold**, or a line starting .. raw:: — becomes markup in the page, so render only data you would be willing to write into the page yourself.

Using the same project with Sphinx¶

Add the sphinx-ub-directives extension to the Sphinx build (extensions = ["sphinx_ub_directives"] in conf.py). It reads the same [parse.directive_templates], [parse.template_contexts] and [parse] ub_template_enabled from ubproject.toml, and the two implementations are held to one contract by a shared conformance corpus: the same rendered text and the same needs, sections included, for every use. What may differ is where a finding is reported (Sphinx reports every finding about a use at the directive’s line, and positions the needs it generates there too) and its message.

The Sphinx extension can also take these settings from conf.py (ub_directive_templates, ub_template_contexts, ub_template_enabled), which ubCode does not read. So keep them in ubproject.toml when both tools build the project; a key set there wins over conf.py in the Sphinx build.

Where the two differ today:

extend: ubCode resolves a ubproject.toml extend chain for these tables; the Sphinx extension reads the one project file.

Coming from the sphinx-jinja2 or sphinx-jinja extensions? The directive is ub-template rather than jinja, contexts move to [parse.template_contexts] (data only: no env, no Python objects, filters or tests), output may not write headings (sphinx-jinja2 lets them join the page; use a rubric), and the extension’s documentation has a migration page: Migrating from sphinx-jinja2 (and sphinx-jinja). A template that names env reports template.context_unavailable: sphinx-jinja2 hands a template the live Sphinx build environment, and neither tool here has one to give.