Conformance scenarios

Generated from the module documentation of crates/canon/src/conform/mod.rs.

Conformance scenarios and their runner. Pure: the caller reads the scenario files and fixtures and passes their text in; canon conform run is that caller.

The canon-conformance/1 scenario format

A scenario file is a YAML document holding exactly one named scenario:

format: canon-conformance/1
id: CANON-CLAIM-001                       # the scenario id, unique within the registry
covers: [CANON-CLAIM-001, CANON-CLAIM-002] # the CANON-* requirements it covers; may be empty
fixture: fixtures/investigation/protocol.yaml
steps:
  - id: compile                           # the step id, unique within the scenario
    compile: {}
    expect:
      canon-ir: |
        { ... the exact canon-ir/1 bytes `canon compile` prints ... }
  - id: no-evidence
    evaluate:
      case: { ... a canon-case/1 document ... }
      evidence: [ ... canon-evidence/1 records ... ]
      authority: [ ... canon-authority/1 decisions ... ]  # optional
      decisions: [ ... canon-decisions/1 decisions ... ]  # optional
      at: 2026-10-04T00:00:00Z                           # optional evaluation instant
    expect:
      decision: |
        { ... the exact canon-decision/1 bytes ... }
      # or, instead of `decision`:
      # refusal: <the identifier the refusal names>

• fixture is the protocol/1 document the scenario compiles, as a path relative to the working directory canon conform run runs in (the repository root, by convention). An absolute path (a leading / or \, or a drive prefix such as C:) or a path with a .. component, split on / or \, is refused: the scenario does not parse. A path that passes that check but resolves outside the working directory through a symbolic link is refused too, unread: the scenario is reported unreadable with fixture <path> resolves outside the working directory.
• covers entries are CANON- followed by an identifier, each listed once.
• steps is an ordered, non-empty list. Each step is exactly one of compile or evaluate, with exactly one expectation of the matching kind: a compile step expects canon-ir, an evaluate step expects decision or refusal.
• An expectation is compared byte for byte with the step's output. A YAML literal block (|) keeps one trailing newline, which is the one canon compile prints.
• Unknown keys are refused: a file with one does not parse as a scenario.

The registry

The registry is the directory conformance/scenarios/: one file per story, named <story-name>.yaml. There is no index file. canon conform run reads every *.yaml file of the directory (--scenarios <dir>, default conformance/scenarios) in sorted file-name order and runs each scenario's steps in order, stopping a scenario at its first failing step.

The report

canon conform run prints one line per file, in the order read, then one summary line, all on standard output:

passed: scenario `<id>`
failed: scenario `<id>` step `<step>`: <what differed>
unreadable: <path>: <why>
conform: <n> passed, <n> failed, <n> unreadable

For a compile step whose output differs, <what differed> is canon-ir/1 differs from the expectation at line <n>, <n> being the first differing line, counted from 1. A fixture that does not parse is reported by kind and position only (fixture does not parse: not well-formed YAML at line <l> column <c>, or not a protocol/1 document at …), never by its content. So is a registry file that does not deserialize as a scenario (unreadable: <path>: not well-formed YAML at …, or not a canon-conformance/1 scenario at …). <path> is the scenario directory as given, joined with the file name.

A registry with no *.yaml file runs no scenario and is refused: the line no scenario ran: the registry holds no scenario file comes before the summary.

Exit status: 0 when at least one scenario ran and every scenario passed; 1 when any scenario failed, any file was unreadable, or no scenario ran; 2 when the registry directory itself cannot be read, with error[unreadable]: <dir>: <why> on standard error and nothing on standard output. The same registry produces byte-identical output on every run.

An evaluate step compiles the fixture, reads case as a canon-case/1 document and each evidence entry as a canon-evidence/1 document, and calls eval::evaluate_with, passing authority and decisions (each as YAML text) and at through unread: the evaluator reads them as canon-authority/1 decisions, canon-decisions/1 explicit decisions and the evaluation instant, and refuses each when it is not one (malformed-input, invalid-instant, …). The authority and decisions lists are each written back as YAML that reads as the same list (in flow style where block style cannot write an entry); a list that cannot be fails the step as <list> entry <n> cannot be serialized: <why> (authority entry 2 …, decisions entry 1 …), and the registry run goes on.

A step expecting decision compares section by section. The expectation's top-level keys are the sections it lists; only those are compared, each on its own and byte for byte: written one section per top-level member, as canonical JSON writes them, in any order, the step passes when each listed section's lines equal the decision's for that section (the comma that separates members aside). An expectation not laid out that way passes only when it equals, byte for byte, the decision's canonical JSON holding only the listed sections (eval::render_sections). A section the decision carries and the expectation does not list is not compared, so a later story that adds a section leaves earlier expectations passing; a listed section the decision does not carry is a difference. An expectation that is not a JSON object is compared with the whole decision (eval::render); one that lists no section fails as the expected canon-decision/1 lists no section. Otherwise <what differed> is canon-decision/1 section <section> differs from the expectation at line <n>: <section> is the first section the expectation writes whose lines differ from the decision's, and <n> the expectation's line, counted from 1, where it first differs (the line that opens it, when the decision does not carry it). When no single section differs, as when the expectation is not laid out as canonical JSON, it is canon-decision/1 differs from the expectation at line <n>, <n> the first differing line. A refused evaluation is evaluation refused: <code>: <why>.

A step expecting refusal passes when the evaluation is refused with exactly that code (a case or evidence entry that is not a document of its format is refused as malformed-input); otherwise <what differed> is expected refusal <code>, the evaluation gave a decision or expected refusal <code>, the evaluation refused with <code>: <why>.

The requirements catalogue

The registry answers for a catalogue of the normative requirements Canon implements, conformance/requirements.yaml, a canon-requirements/1 document:

format: canon-requirements/1
requirements:
  CANON-CLAIM-001:
    status: implemented        # implemented | planned
    statement: A required positive predicate evaluating to UNKNOWN MUST NOT be treated as satisfied.
    source: design § 32        # where the requirement is stated

Keys are CANON- followed by an identifier, each listed once; status, statement and source are required, and the statement and source are not empty. Unknown keys are refused. A planned requirement is stated but not yet implemented.

canon conform run --requirements <catalogue> reads the catalogue before the registry; one that cannot be read exits 2 with error[unreadable]: <path>: <why> on standard error, and one that does not parse as a catalogue exits 1 with error[catalogue]: <path>: <why>, the catalogue reported by kind and position only, as a registry file is. Either way nothing is printed on standard output. Then the registry runs as above, and these checks follow:

1. Coverage. Every implemented requirement is covered by at least one scenario that passed; otherwise uncovered: requirement <id>. A `planned` requirement needs no scenario; one a passing scenario covers is reported `planned requirement `<id>` is covered`, so the catalogue is moved forward rather than drifting. Both in catalogue order. 2. **Unlisted.** A scenario that covers a requirement the catalogue does not list, passed or not, is reported `unlisted: requirement `<id>` covered by scenario `<scenario>, in registry order.
2. Determinism. Every evaluate step of every scenario whose fixture compiles runs three times: as written, again as written, and with its evidence, authority and decisions lists in reverse order, an equivalent input, since none of the three is ordered by position (CANON-DETERMINISM-001). The three outputs, each the canonical canon-decision/1 document or the refusal's code, must be byte-identical; otherwise nondeterministic: scenario <id>step<step>: <rerun|permuted> differs at line <n>, <n> the first line, counted from 1, where that run's output differs from the first. In registry and step order.

The check lines come after the registry's lines and before its summary, and a second summary line follows it:

requirements: <n> implemented, <n> covered, <n> planned

covered counts the implemented requirements a passing scenario covers. The run exits 0 only when the registry passed and no check failed, and 1 otherwise. Without --requirements the report is the registry's alone, byte for byte.