Compatibility: Pactmark 0.1.x. The
delegated-incident-boundary example proves the delegation
and fencing properties described here. It does not claim durable resume — see
Limits.The situation
A platform team wants an assistant that helps during incidents: gather signals, correlate them, propose a containment action, and — under supervision — execute it. Rate-limit a noisy tenant, disable a leaking integration, roll back a feature flag. This is the scenario with the least margin for error, because it runs precisely when everything else is already going wrong.It runs unattended
At 3am, triggered by an alert, with nobody watching the console.
It acts on production
The blast radius of a wrong action is the thing you were trying to protect.
Infrastructure is unhealthy
Timeouts and lost responses are the normal case, not the exception.
Everything is reviewed afterwards
In a post-incident review, by people who want a precise timeline.
Delegated authority
A run started by an alert has no interactive human. It still needs an authority — and that authority must be derived, not invented.DelegatedRunAuthority binds a system worker to one run, one scheduler receipt, one lease and one
fencing token. A newer fence invalidates the older delegation — that property is exactly what the
delegated-incident-boundary example demonstrates.
Figure 1. During an incident, workers restart. Fenced leases are what make a restart safe
rather than a second source of damage.
The policy is deliberately narrow
R4 is denied outright here, not gated behind an approval. During an incident, an approval prompt
for an irreversible production action is a decision made under time pressure by a tired person.
Some doors should simply not be in the room.
Reading is the majority of the work
Most incident value is in correlation, not action. Read tools at R1 with tight egress allowlists — metrics, logs, deployment history, feature-flag state — give an assistant the ability to build a timeline in seconds without the ability to change anything.1
Gather
R1 reads across the declared allowlist.
2
Correlate
Produce a timeline artifact, content-addressed.
3
Propose
A containment action with a normalized preview.
4
Wait
waiting_for_approval. The run parks — possibly for a long time — without holding anything open.When the containment action is uncertain
Figure 2. The infrastructure is already unhealthy. Uncertain effects are the expected case, and the strategy has to have been registered before the incident started. A rate-limit call times out. Was the limit applied?1
Do not retry
A doubled rate-limit during an incident can be worse than none.
2
Park and reconcile
If the control plane supports lookup, the strategy is
reconcilable: query current state and
record what is actually true.3
If it cannot be looked up, say so
abandon_uncertain records that the effect may have occurred. The incident timeline shows an open
question rather than a false resolution.Compensation is a new run
Rolling back a containment action is not an undo — it is a new business action:CompensationRequested starts a separate compensation run with its own WorkOrder, its own
authority and its own evidence. EffectCompensated then binds the original effect digest to the
compensating one.
In a post-incident review this reads correctly: this action was taken at this time under this
authority, and that action reversed it at that time under that authority.
The post-incident record
What you still own
The runbook
Which actions are even eligible for automation is a judgement about your systems.
Someone awake
Parked runs need a human. Automation that parks and is never resolved is worse than no
automation.
Control-plane semantics
reconcilable is your assertion. If the control plane cannot answer reliably, it is wrong.Not claiming causation
A timeline shows sequence. Causation is a conclusion someone has to defend.
Reliability and recovery
Running the incident, step by step.
Durability
Fenced leases, atomic commands and what to back up together.