eval.gated_change/1
Decide one change an automated loop made to an agent's text or configuration (guidance, a system prompt, tool caps) against an evaluation set: keep it only on evidence about that exact candidate revision, from a producer the case lists as another agent run than the one that wrote it, that the evaluation design passed review, the change stays inside the surface the loop may change and carries no leaked or forbidden content, it improved on the tuning split and the improvement reproduced, and the held-out split did not regress; reject it when every one of those is decided and one is false; propose it to a person when it leaves the surface; or close the case inconclusive on an explicit decision. The candidate's revision hashes its content together with its basis (baseline, question set, data revision and design), so a moved basis is a new candidate revision and evidence about the old one no longer counts. Missing, stale or non-independent evidence leaves a claim unknown, and unknown neither keeps nor rejects.
- Id:
eval.gated_change - Revision: 1
- Source:
protocols/eval-gated-change/1.yaml
Dependency graph
Which actions may produce which evidence, which claims that evidence establishes, and which outcomes and obligations rest on those claims. Hover or focus a node to trace what it rests on.
Protocol · revision 1
eval.gated_change/1Decide one change an automated loop made to an agent's text or configuration (guidance, a system prompt, tool caps) against an evaluation set: keep it only on evidence about that exact candidate revision, from a producer the case lists as another agent run than the one that wrote it, that the evaluation design passed review, the change stays inside the surface the loop may change and carries no leaked or forbidden content, it improved on the tuning split and the improvement reproduced, and the held-out split did not regress; reject it when every one of those is decided and one is false; propose it to a person when it leaves the surface; or close the case inconclusive on an explicit decision. The candidate's revision hashes its content together with its basis (baseline, question set, data revision and design), so a moved basis is a new candidate revision and evidence about the old one no longer counts. Missing, stale or non-independent evidence leaves a claim unknown, and unknown neither keeps nor rejects.
- Actions6
- Evidence kinds4
- Claims7
- Outcomes4
- Claim supports claim
- Precondition, traced on hover
- Needs authority
Hover or focus a node to trace what it rests on and what rests on it. Press Enter to pin it, Escape to clear.
Protocol as text
| Name | Description | Rule | Leads to |
|---|---|---|---|
candidate.evaluate | check and run the current candidate revision: the guard check, the tuning comparison and the held-out comparison | effect read | produces guard_check; produces heldout_comparison; produces tuning_comparison |
change.apply_outside_surface | apply a candidate that leaves the surface the loop may change; needs a capability never granted to a loop | claim is false: change.within_surface needs loop_change.apply_outside_surface effect write | — |
change.keep | keep the candidate; needs every keep claim to hold for the current candidate revision and authority to keep a loop's change | claim change.supported needs loop_change.keep effect write | — |
change.propose | hand a candidate that leaves the surface the loop may change to a person as a proposal, applying nothing | claim is false: change.within_surface effect none | — |
change.reject | discard the candidate, leaving the surface as it was; needs every keep claim decided and one of them false, so an unknown claim never rejects | all of claim is false: change.supported not claim is unknown: change.within_surface not claim is unknown: content.clean not claim is unknown: experiment.valid not claim is unknown: heldout.not_regressed not claim is unknown: result.reproducible not claim is unknown: tuning.improved effect none | — |
design.review | review the evaluation design, under authority | needs eval_design.review effect none | produces design_review |
More columns: swipe horizontally, or focus the table and use the arrow keys.
| Name | Description | Rule | Leads to |
|---|---|---|---|
design_review | a review of the evaluation design; its result is passed or failed | — | establishes experiment.valid (passed about eval_design, independent on different_agent_run) |
guard_check | a check of the candidate, in order, against the surface the loop may change, for expected answers inside the candidate and for new forbidden content in answers; its result is outside, leaked or clean | — | establishes change.within_surface (not outside about candidate, independent on different_agent_run); establishes content.clean (clean about candidate, independent on different_agent_run) |
heldout_comparison | a run of the candidate against the baseline on the held-out split; its result is held or regressed | — | establishes heldout.not_regressed (held about candidate, independent on different_agent_run) |
tuning_comparison | a first and a second run of the candidate against the baseline on the tuning split; its result is reproduced (both better), not_reproduced (the first better, the second not) or not_better (the first not better) | — | establishes result.reproducible (reproduced about candidate, independent on different_agent_run); establishes tuning.improved (not not_better about candidate, independent on different_agent_run) |
More columns: swipe horizontally, or focus the table and use the arrow keys.
| Name | Description | Rule | Leads to |
|---|---|---|---|
change.supported | every keep claim holds for the current candidate revision: the design is valid, the change is within the surface and clean, it improved on the tuning split reproducibly, and the held-out split did not regress | all of claim change.within_surface claim content.clean claim experiment.valid claim heldout.not_regressed claim result.reproducible claim tuning.improved | requires kept; requires rejected (false); gates change.keep; gates change.reject (false) |
change.within_surface | an independent guard check of the current candidate revision did not find it outside the surface the loop may change | not evidence = outside: guard_check | supports change.supported; requires proposed (false); requires rejected; gates change.apply_outside_surface (false); gates change.propose (false); gates change.reject (not unknown) |
content.clean | an independent guard check of the current candidate revision found it inside the surface, with no expected answers inside it and no new forbidden content in answers | evidence = clean: guard_check | supports change.supported; requires rejected (not unknown); gates change.reject (not unknown) |
experiment.valid | the current evaluation design passed a review by a producer the case lists as another agent run than the loop's | evidence = passed: design_review | supports change.supported; requires rejected (not unknown); gates change.reject (not unknown) |
heldout.not_regressed | an independent run of the current candidate revision on the held-out split did not regress against the baseline | evidence = held: heldout_comparison | supports change.supported; requires rejected (not unknown); gates change.reject (not unknown) |
result.reproducible | an independent second run of the current candidate revision on the tuning split reproduced the first run's improvement | evidence = reproduced: tuning_comparison | supports change.supported; requires rejected (not unknown); gates change.reject (not unknown) |
tuning.improved | an independent first run of the current candidate revision on the tuning split was better than the baseline | not evidence = not_better: tuning_comparison | supports change.supported; requires rejected (not unknown); gates change.reject (not unknown) |
More columns: swipe horizontally, or focus the table and use the arrow keys.
| Name | Description | Rule | Leads to |
|---|---|---|---|
inconclusive | the case closed on an explicit decision that the evaluation settles nothing | — | — |
kept | the case closed with the candidate kept; needs every keep claim to hold | claim change.supported | — |
proposed | the case closed with the candidate proposed to a person, because it leaves the surface the loop may change | claim is false: change.within_surface | — |
rejected | the case closed with the candidate rejected; needs it within the surface, every keep claim decided and one of them false. Evidence that is missing, about another candidate revision or not independent rejects nothing: it leaves its claim unknown, not false | all of claim is false: change.supported claim change.within_surface not claim is unknown: content.clean not claim is unknown: experiment.valid not claim is unknown: heldout.not_regressed not claim is unknown: result.reproducible not claim is unknown: tuning.improved | — |
More columns: swipe horizontally, or focus the table and use the arrow keys.
Source: protocols/eval-gated-change/1.yaml
Actions
What can be done in a case: what it needs first, which authority it requires, the class of effect it has and the evidence it may produce.
| Action | Description | Precondition | Requires capability | Effect | May produce |
|---|---|---|---|---|---|
candidate.evaluate | check and run the current candidate revision: the guard check, the tuning comparison and the held-out comparison | none | no authority | read | guard_check, heldout_comparison, tuning_comparison |
change.apply_outside_surface | apply a candidate that leaves the surface the loop may change; needs a capability never granted to a loop | change.within_surface is FALSE | loop_change.apply_outside_surface | write | no evidence |
change.keep | keep the candidate; needs every keep claim to hold for the current candidate revision and authority to keep a loop's change | change.supported is TRUE | loop_change.keep | write | no evidence |
change.propose | hand a candidate that leaves the surface the loop may change to a person as a proposal, applying nothing | change.within_surface is FALSE | no authority | none | no evidence |
change.reject | discard the candidate, leaving the surface as it was; needs every keep claim decided and one of them false, so an unknown claim never rejects | all of (change.supported is FALSE; not (change.within_surface is UNKNOWN); not (content.clean is UNKNOWN); not (experiment.valid is UNKNOWN); not (heldout.not_regressed is UNKNOWN); not (result.reproducible is UNKNOWN); not (tuning.improved is UNKNOWN)) | no authority | none | no evidence |
design.review | review the evaluation design, under authority | none | eval_design.review | none | design_review |
More columns: swipe horizontally, or focus the table and use the arrow keys.
Evidence kinds
The kinds of evidence this protocol admits.
| Evidence kind | Description | Produced by | Used by claims |
|---|---|---|---|
design_review | a review of the evaluation design; its result is passed or failed | design.review | experiment.valid |
guard_check | a check of the candidate, in order, against the surface the loop may change, for expected answers inside the candidate and for new forbidden content in answers; its result is outside, leaked or clean | candidate.evaluate | change.within_surface, content.clean |
heldout_comparison | a run of the candidate against the baseline on the held-out split; its result is held or regressed | candidate.evaluate | heldout.not_regressed |
tuning_comparison | a first and a second run of the candidate against the baseline on the tuning split; its result is reproduced (both better), not_reproduced (the first better, the second not) or not_better (the first not better) | candidate.evaluate | result.reproducible, tuning.improved |
More columns: swipe horizontally, or focus the table and use the arrow keys.
Claims
Each claim is TRUE, FALSE or UNKNOWN, established from evidence by its predicate. Missing or stale evidence leaves a claim UNKNOWN, never FALSE.
| Claim | Description | True when |
|---|---|---|
change.supported | every keep claim holds for the current candidate revision: the design is valid, the change is within the surface and clean, it improved on the tuning split reproducibly, and the held-out split did not regress | all of (change.within_surface is TRUE; content.clean is TRUE; experiment.valid is TRUE; heldout.not_regressed is TRUE; result.reproducible is TRUE; tuning.improved is TRUE) |
change.within_surface | an independent guard check of the current candidate revision did not find it outside the surface the loop may change | not (evidence guard_check with result outside about candidate, from a producer independent on different_agent_run) |
content.clean | an independent guard check of the current candidate revision found it inside the surface, with no expected answers inside it and no new forbidden content in answers | evidence guard_check with result clean about candidate, from a producer independent on different_agent_run |
experiment.valid | the current evaluation design passed a review by a producer the case lists as another agent run than the loop's | evidence design_review with result passed about eval_design, from a producer independent on different_agent_run |
heldout.not_regressed | an independent run of the current candidate revision on the held-out split did not regress against the baseline | evidence heldout_comparison with result held about candidate, from a producer independent on different_agent_run |
result.reproducible | an independent second run of the current candidate revision on the tuning split reproduced the first run's improvement | evidence tuning_comparison with result reproduced about candidate, from a producer independent on different_agent_run |
tuning.improved | an independent first run of the current candidate revision on the tuning split was better than the baseline | not (evidence tuning_comparison with result not_better about candidate, from a producer independent on different_agent_run) |
More columns: swipe horizontally, or focus the table and use the arrow keys.
Obligations
What must be done before a case can be complete, and what discharges it. An obligation stays open until its predicate is TRUE; its predicate being UNKNOWN does not discharge it.
This protocol declares no obligations.
Outcomes
The legitimate terminal interpretations of a case, and what each requires.
| Outcome | Description | Requires |
|---|---|---|
inconclusive | the case closed on an explicit decision that the evaluation settles nothing | decision explicitly_inconclusive |
kept | the case closed with the candidate kept; needs every keep claim to hold | change.supported is TRUE |
proposed | the case closed with the candidate proposed to a person, because it leaves the surface the loop may change | change.within_surface is FALSE |
rejected | the case closed with the candidate rejected; needs it within the surface, every keep claim decided and one of them false. Evidence that is missing, about another candidate revision or not independent rejects nothing: it leaves its claim unknown, not false | all of (change.supported is FALSE; change.within_surface is TRUE; not (content.clean is UNKNOWN); not (experiment.valid is UNKNOWN); not (heldout.not_regressed is UNKNOWN); not (result.reproducible is UNKNOWN); not (tuning.improved is UNKNOWN)) |
More columns: swipe horizontally, or focus the table and use the arrow keys.
Artifacts
The things whose revisions evidence is bound to.
| Artifact | Description |
|---|---|
candidate | the change under test; its revision hashes the changed text or configuration together with its basis: the baseline, the question set, the data revision and the evaluation design |
eval_design | the evaluation design the loop is tuned under; its splits, metrics and guards |
More columns: swipe horizontally, or focus the table and use the arrow keys.
Independence dimensions
What an evidence match may demand of a record's producer (independent_on). The case lists, for each producer, the dimensions on which it is independent of the case's work; a match that demands a dimension reads only records whose producer the case lists on it. A record that fails the demand neither establishes nor contradicts the match, so a match only such records would decide is UNKNOWN.
| Dimension | Description |
|---|---|
different_agent_run | the producer is another agent run than the one that wrote the candidate |
More columns: swipe horizontally, or focus the table and use the arrow keys.