incident.response/1

Respond to an incident through two obligations that progress independently: restore_service, on which emergency mode is left, and investigate_cause, which stays open after emergency mode ends until a cause analysis of a current revision of the service or the release identifies the cause. A rollback of the release is verified only on a complete rollback result from a verifier the case lists as another principal than the one who rolled it back. The service is healthy only on an observation at most 30 minutes old.

• Id: incident.response
• Revision: 1
• Source: protocols/incident-response/1.yaml

Dependency graph

Which actions may produce which evidence, which claims that evidence establishes, and which outcomes and obligations rest on those claims. Hover or focus a node to trace what it rests on.

Protocol · revision 1

incident.response/1

Respond to an incident through two obligations that progress independently: restore_service, on which emergency mode is left, and investigate_cause, which stays open after emergency mode ends until a cause analysis of a current revision of the service or the release identifies the cause. A rollback of the release is verified only on a complete rollback result from a verifier the case lists as another principal than the one who rolled it back. The service is healthy only on an observation at most 30 minutes old.

• Actions6
• Evidence kinds4
• Claims4
• Obligations2
• Precondition, traced on hover
• Needs authority

Hover or focus a node to trace what it rests on and what rests on it. Press Enter to pin it, Escape to clear.

Protocol as text

Actions

Name	Description	Rule	Leads to
emergency.leave	leave emergency mode; admissible once restore_service is discharged and the impact is bounded, whatever is known of the cause	all of claim impact.bounded claim service.healthy	—
logs.search	search the system's logs	effect read	produces cause_analysis
metrics.inspect	read the system's metrics	effect read	produces impact_assessment; produces operational_observation
release.inspect	read what a release contains	effect read	produces cause_analysis
release.rollback	return to a previous release	needs release.rollback effect write	produces rollback_result
traffic.shift	move traffic between deployments	needs traffic.shift effect write	—

Evidence kinds

Name	Description	Rule	Leads to
cause_analysis	an analysis of why an incident happened	—	establishes cause.identified (identified)
impact_assessment	an assessment of who and what an incident affects	—	establishes impact.bounded (bounded about service)
operational_observation	an observation of the running system; its result is healthy or unhealthy, and it applies for 30 minutes after it is observed, after which it no longer says anything	—	establishes service.healthy (healthy about service)
rollback_result	the result of returning a release to the one before it; its result is complete or partial	—	establishes rollback.verified (complete about release, independent on different_principal)

Claims

Name	Description	Rule	Leads to
cause.identified	the incident's cause is known	evidence = identified: cause_analysis	requires investigate_cause
impact.bounded	the incident's impact is known and contained	evidence = bounded: impact_assessment	gates emergency.leave
rollback.verified	the release's rollback is complete, by a rollback result from a producer the case lists as another principal than the one who rolled it back	evidence = complete: rollback_result	—
service.healthy	the service is running healthily	evidence = healthy: operational_observation	requires restore_service; gates emergency.leave

Obligations

Name	Description	Rule	Leads to
investigate_cause	establish why the incident happened; stays open after emergency mode ends until a cause analysis of a current revision of the service or the release identifies the cause	claim cause.identified	—
restore_service	urgent; bring the affected service back to health	claim service.healthy	—

Source: protocols/incident-response/1.yaml

Actions

What can be done in a case: what it needs first, which authority it requires, the class of effect it has and the evidence it may produce.

Action	Description	Precondition	Requires capability	Effect	May produce
emergency.leave	leave emergency mode; admissible once restore_service is discharged and the impact is bounded, whatever is known of the cause	all of (impact.bounded is TRUE; service.healthy is TRUE)	no authority	not declared	no evidence
logs.search	search the system's logs	none	no authority	read	cause_analysis
metrics.inspect	read the system's metrics	none	no authority	read	impact_assessment, operational_observation
release.inspect	read what a release contains	none	no authority	read	cause_analysis
release.rollback	return to a previous release	none	release.rollback	write	rollback_result
traffic.shift	move traffic between deployments	none	traffic.shift	write	no evidence

Evidence kinds

The kinds of evidence this protocol admits. A kind with a maximum age applies for that long after a record's observed_at: evaluated at a later instant, the record is excluded as expired.

Evidence kind	Description	Applies for	Produced by	Used by claims
cause_analysis	an analysis of why an incident happened	no limit	logs.search, release.inspect	cause.identified
impact_assessment	an assessment of who and what an incident affects	no limit	metrics.inspect	impact.bounded
operational_observation	an observation of the running system; its result is healthy or unhealthy, and it applies for 30 minutes after it is observed, after which it no longer says anything	30m after it is observed	metrics.inspect	service.healthy
rollback_result	the result of returning a release to the one before it; its result is complete or partial	no limit	release.rollback	rollback.verified

Claims

Each claim is TRUE, FALSE or UNKNOWN, established from evidence by its predicate. Missing or stale evidence leaves a claim UNKNOWN, never FALSE.

Claim	Description	True when
cause.identified	the incident's cause is known	evidence cause_analysis with result identified
impact.bounded	the incident's impact is known and contained	evidence impact_assessment with result bounded about service
rollback.verified	the release's rollback is complete, by a rollback result from a producer the case lists as another principal than the one who rolled it back	evidence rollback_result with result complete about release, from a producer independent on different_principal
service.healthy	the service is running healthily	evidence operational_observation with result healthy about service

Obligations

What must be done before a case can be complete, and what discharges it. An obligation stays open until its predicate is TRUE; its predicate being UNKNOWN does not discharge it.

Obligation	Description	Discharged when
investigate_cause	establish why the incident happened; stays open after emergency mode ends until a cause analysis of a current revision of the service or the release identifies the cause	cause.identified is TRUE
restore_service	urgent; bring the affected service back to health	service.healthy is TRUE

Outcomes

The legitimate terminal interpretations of a case, and what each requires.

This protocol declares no outcomes.

Artifacts

The things whose revisions evidence is bound to.

Artifact	Description
release	a versioned, deployable unit of the system
service	the running service a case concerns

Independence dimensions

What an evidence match may demand of a record's producer (independent_on). The case lists, for each producer, the dimensions on which it is independent of the case's work; a match that demands a dimension reads only records whose producer the case lists on it. A record that fails the demand neither establishes nor contradicts the match, so a match only such records would decide is UNKNOWN.

Dimension	Description
different_principal	the producer is another principal than the one who did the case's work