ml.experiment/1

Test a hypothesis by experiment: run a candidate against a baseline on a frozen dataset snapshot under a reviewed design, and close the case supported when the experiment is valid and its result reproducible, beats the baseline and is robust; refuted when a valid, reproducible result does not beat the baseline; invalid when the experiment itself is not valid; or inconclusive or superseded on an explicit decision.

• Id: ml.experiment
• Revision: 1
• Source: protocols/ml-experiment/1.yaml

Dependency graph

Which actions may produce which evidence, which claims that evidence establishes, and which outcomes and obligations rest on those claims. Hover or focus a node to trace what it rests on.

Protocol · revision 1

ml.experiment/1

Test a hypothesis by experiment: run a candidate against a baseline on a frozen dataset snapshot under a reviewed design, and close the case supported when the experiment is valid and its result reproducible, beats the baseline and is robust; refuted when a valid, reproducible result does not beat the baseline; invalid when the experiment itself is not valid; or inconclusive or superseded on an explicit decision.

• Evidence kinds5
• Claims5
• Outcomes5
• Claim supports claim

Hover or focus a node to trace what it rests on and what rests on it. Press Enter to pin it, Escape to clear.

Protocol as text

Evidence kinds

Name	Description	Rule	Leads to
comparison	a comparison of a candidate with the baseline on the design's metrics; its result is candidate_better or baseline_better	—	establishes candidate.beats_baseline (candidate_better about candidate)
design_review	a review of an experiment design; its result is passed or failed	—	establishes experiment.valid (passed about experiment_design)
integrity_check	a check of the runs behind a set of results for leakage, contamination or other integrity violations; its result is clean or violated	—	establishes experiment.valid (clean about results)
reproduction	a rerun on the same dataset snapshot and run configuration; its result is matched or diverged	—	establishes result.reproducible (matched about results)
seed_sweep	a sweep of a candidate's effect over seeds or ablations; its result is stable or unstable	—	establishes effect.robust (stable about candidate)

Claims

Name	Description	Rule	Leads to
candidate.beats_baseline	a comparison of the current candidate revision found it better than the baseline	evidence = candidate_better: comparison	supports hypothesis.supported; requires refuted (false)
effect.robust	a seed sweep of the current candidate revision found its effect stable	evidence = stable: seed_sweep	supports hypothesis.supported
experiment.valid	the current experiment design passed review and the current results carry no integrity violation	all of evidence = passed: design_review evidence = clean: integrity_check	supports hypothesis.supported; requires invalid_experiment (false); requires refuted
hypothesis.supported	the experiment is valid, its result reproducible, and the candidate beats the baseline with a robust effect	all of claim candidate.beats_baseline claim effect.robust claim experiment.valid claim result.reproducible	requires supported
result.reproducible	a reproduction of the current results matched them	evidence = matched: reproduction	supports hypothesis.supported; requires refuted

Outcomes

Name	Description	Rule	Leads to
inconclusive	the case closed on an explicit decision that the experiment settles nothing	—	—
invalid_experiment	the case closed because the experiment itself is not valid	claim is false: experiment.valid	—
refuted	the case closed with the hypothesis refuted; needs a valid experiment and a reproducible result in which the candidate does not beat the baseline. A comparison that is missing, or about another candidate revision, refutes nothing: it leaves the comparison unknown, not false	all of claim is false: candidate.beats_baseline claim experiment.valid claim result.reproducible	—
superseded	the case closed because another case explicitly superseded it	—	—
supported	the case closed with the hypothesis supported	claim hypothesis.supported	—

Source: protocols/ml-experiment/1.yaml

Actions

What can be done in a case: what it needs first, which authority it requires, the class of effect it has and the evidence it may produce.

This protocol declares no actions.

Evidence kinds

The kinds of evidence this protocol admits.

Evidence kind	Description	Produced by	Used by claims
comparison	a comparison of a candidate with the baseline on the design's metrics; its result is candidate_better or baseline_better	no action	candidate.beats_baseline
design_review	a review of an experiment design; its result is passed or failed	no action	experiment.valid
integrity_check	a check of the runs behind a set of results for leakage, contamination or other integrity violations; its result is clean or violated	no action	experiment.valid
reproduction	a rerun on the same dataset snapshot and run configuration; its result is matched or diverged	no action	result.reproducible
seed_sweep	a sweep of a candidate's effect over seeds or ablations; its result is stable or unstable	no action	effect.robust

Claims

Each claim is TRUE, FALSE or UNKNOWN, established from evidence by its predicate. Missing or stale evidence leaves a claim UNKNOWN, never FALSE.

Claim	Description	True when
candidate.beats_baseline	a comparison of the current candidate revision found it better than the baseline	evidence comparison with result candidate_better about candidate
effect.robust	a seed sweep of the current candidate revision found its effect stable	evidence seed_sweep with result stable about candidate
experiment.valid	the current experiment design passed review and the current results carry no integrity violation	all of (evidence design_review with result passed about experiment_design; evidence integrity_check with result clean about results)
hypothesis.supported	the experiment is valid, its result reproducible, and the candidate beats the baseline with a robust effect	all of (candidate.beats_baseline is TRUE; effect.robust is TRUE; experiment.valid is TRUE; result.reproducible is TRUE)
result.reproducible	a reproduction of the current results matched them	evidence reproduction with result matched about results

Obligations

What must be done before a case can be complete, and what discharges it. An obligation stays open until its predicate is TRUE; its predicate being UNKNOWN does not discharge it.

This protocol declares no obligations.

Outcomes

The legitimate terminal interpretations of a case, and what each requires.

Outcome	Description	Requires
inconclusive	the case closed on an explicit decision that the experiment settles nothing	decision explicitly_inconclusive
invalid_experiment	the case closed because the experiment itself is not valid	experiment.valid is FALSE
refuted	the case closed with the hypothesis refuted; needs a valid experiment and a reproducible result in which the candidate does not beat the baseline. A comparison that is missing, or about another candidate revision, refutes nothing: it leaves the comparison unknown, not false	all of (candidate.beats_baseline is FALSE; experiment.valid is TRUE; result.reproducible is TRUE)
superseded	the case closed because another case explicitly superseded it	decision explicitly_superseded
supported	the case closed with the hypothesis supported	hypothesis.supported is TRUE

Artifacts

The things whose revisions evidence is bound to.

Artifact	Description
baseline	the reference model or method the candidate is compared with
candidate	the model or method under test
dataset_snapshot	the frozen data the runs read
experiment_design	the metrics, splits, seeds and success criterion the experiment is run under
results	the metrics the runs produced
run_configuration	the hyperparameters, environment and code revision of the runs