Compatibility: Pactmark 0.1.x.

​
Design for at-least-once

Durable command records

Idempotency scoped by authority and request digest.

Append-only events

With optimistic sequencing, so concurrent appends conflict instead of interleaving.

Transactional wake-ups

Committed with the events that caused them.

Database-time leases

Not application time. Clock skew cannot extend ownership.

Fencing tokens

A returning zombie cannot commit over a newer owner.

Pessimistic reservations

Uncertain model calls retain their maximum charge.
Figure 1. Recovery starts from persisted truth in a fresh process. Nothing depends on the original process, the original request, or the original machine.

​
Monitor parked work, not just active work

A dashboard showing “3 runs in progress” and nothing else will report a healthy system while unresolved effects accumulate behind it.

​
Uncertain effects

Figure 2. The only three honest resolutions: it was acknowledged, it definitely did not dispatch, or it is uncertain and a registered strategy decides what happens next. Never retry solely because a response was lost. A lost response is the definition of not knowing. Follow the registered strategy, or start a separately authorised compensation run.

​
Running an incident

1

Contain access first

Revoke credentials and grants, activate digest kill switches, stop workers or egress where required, and preserve tenant boundaries. Do this before diagnosis.
2

Do not rewrite run truth

Append-only means append-only, including during an incident. Preserve events, command and effect records, release manifests and relevant metadata logs — without copying secret values or hidden model reasoning.
3

Classify every effect

Acknowledged, definitely not dispatched, or uncertain. There is no fourth category and no “probably fine”.
4

Resolve only through the registered strategy

Retry where the strategy proves it is safe. Reconcile by lookup where the target supports it. Otherwise start a separately authorised compensation WorkOrder. Human guesses are not receipts.
5

Rotate and rebuild

Rotate exposed secrets, rebuild projections, restore protected stores if required, and verify artifact and evidence bindings still resolve.
6

Notify and record

Affected owners, timeline in UTC, scope, residual risk and the follow-up tests you added.
Verify recovery explicitly: the old digest is denied, the old credential is unusable, the next high-risk action re-checks revocation, and no canary value appears in any persisted or output surface. Absence of alerts is not verification.

​
Setting RPO and RTO

Derive them from the deployed database, artifact, key and backup systems — then prove them:
1

Restore a backup into a clean environment

Time it. That number is your RTO floor, not your target.
2

Replay a terminal run from the restored data

pactmark replay <runId> verifies from stored events and artifacts without executing anything.
3

Resume an in-flight run in a fresh process

This is the one that finds missing pieces in your backup set.
4

Measure the data gap

The distance between the last durable commit and the restore point is your real RPO.
Local Postgres crash tests are design evidence. They demonstrate that the mechanism is correct. They are not a promise about your managed database’s failover behaviour, your storage layer, or your network.

PostgreSQL profile

Migrations, leases, protected context and what to back up together.

Incident response use case

A worked scenario with delegated authority and compensation.