This is
- An honest boundary map for a receipt layer
- Evidence that transport faults ≠ identity faults
-
A reason to run
afterstate identity previewbefore production
Findings Adversarial study
Adversarial study · 16 Jul 2026
Where Afterstate breaks
The scale study showed 100% safe on transport faults. That is not the whole story. We deliberately misconfigured identity and delayed the observer — then published the scorecard. Blended afterstate safety on this matrix is 92%. Every unsafe cell has one cause.
Headline
Three arms, five faults, same τ³ write tools. Safe means no integrity
failure. Completion means verified_success.
| Arm | Safe | Completion |
|---|---|---|
| naive blind retry | 39% 101 / 260 | 27% 70 / 260 |
| observe_first prompt: check before retry | 85% 220 / 260 | 50% 129 / 260 |
| afterstate claim → observe → receipt | 92% 240 / 260 | 69% 180 / 260 |
The 8% unsafe afterstate cells are all
identity_collision (20 / 20). Strip that fault and
afterstate is 100% safe on the remaining 240 cells.
By fault — where the 8% lives
Read this table before the blended rate. Transport faults are solved. The two adversarial faults are the product boundary.
| Fault | afterstate safe | afterstate complete | What it means |
|---|---|---|---|
timeout_after_send |
100% 60/60 | 100% 60/60 | Write lands; response lost → observe → applied |
network_down |
100% 60/60 | 100% 60/60 | Write never lands → not_applied → safe retry |
double_ask |
100% 60/60 | 100% 60/60 | Identical second call → replay, no second write |
observer_lag |
100% 60/60 | 0% 0/60 | Safe-but-stuck: escalate / wait — resolves 60/60 with reconciler (see Break #2) |
identity_collision |
0% 0/20 | 0% 0/20 | Coarse identity → second distinct intent suppressed |
Break #1 — identity collision
Afterstate dedupes by a business-operation identity you declare.
If that identity is too coarse, two different intents hash to the
same action_id. The second call is treated as a
duplicate — and the customer’s real second intent never runs.
-
Book SFO→JFK for Alice lands once
-
Book SFO→JFK for Bob same action_id
-
Bob’s booking suppressed
suppressed_intent
0 / 20 safe on this fault.
This is a configuration failure, not a transport failure.
We induced it on purpose.
-
Decorate-time warning no scope-anchor → loud UserWarning
-
afterstate identity previewcompare two param sets offline -
Add an anchor field passenger_id / booking_id / *_id
Publish the break, then ship the lint.
The same misconfig that scored 0% now fails loudly at
integrate time — including in the
QuickServe demo.
Break #2 — observer lag
If the system of record has not yet reflected the write when we
observe, Afterstate refuses to guess. Verdict:
escalated (or WAIT once durable reconcile is
on the path). That is safe — no blind retry, no duplicate —
and incomplete until a later observation resolves it.
| Arm on observer_lag | Safe | Complete |
|---|---|---|
| naive | 0% 0/60 | 0% 0/60 |
| observe_first | 100% 60/60 | 50% 30/60 |
| afterstate | 100% 60/60 | 0% 0/60 |
Follow-up: rerun with the reconciler wired in
The durable reconciler (afterstate reconcile) re-observes
pending claims after the lag window and resolves them to
applied / not_applied. We reran the full observer_lag slice — same
models, scenarios, and fault — with that worker on the path.
| Arm on observer_lag | Safe | Complete |
|---|---|---|
| afterstate + reconcile | 100% 60/60 | 100% 60/60 |
Run id: world_bench_lag_reconcile · 60 graded cells ·
1 rate-limited cell retried as infra_failed. Every
pending claim resolved to verified_success once the
system of record caught up — no escalation left open, no blind
retry issued while waiting. The 0% completion above is the
published matrix without the worker; this is the same fault with
the shipped fix.
Verdict mix (afterstate)
| Verdict | n | Read |
|---|---|---|
verified_success |
180 | Transport faults, resolved |
escalated |
60 | Observer lag — safe stop |
suppressed_intent
|
20 | Identity collision — the only unsafe mode |
How to read this
This is not
- “We are always 100% safe”
- A reason to skip identity design
- An endorsement by Sierra Research
Methods
- Tools and databases from sierra-research/tau2-bench (τ³), including telecom.
-
Arms:
naive,observe_first,afterstate. -
Adversarial faults:
observer_lag(bounded false-negative observe),identity_collision(coarse identity fields on airline book). -
Empty transcripts and API outages graded as
infra_failed— excluded from safety stats (0 in this run). -
Run id:
world_bench_adversarial· 780 graded cells.
If you ship write tools
The useful question is not “are you 100%?” — it is “where do you refuse to guess, and where does bad identity still hurt?” We are building that layer with design partners.