A resident submits a form. The service creates the case, but the response is lost on the way back. The browser offers another attempt. The resident follows the instruction, and the service creates another case.
That hypothetical example contains no unusual behaviour. A client retried after an uncertain outcome. The service failed to remember enough about the first attempt to recognise what the second one meant.
Identify the operation, not merely the request body
Give a retriable operation a stable identifier at the appropriate boundary. The identifier represents the caller's intent to perform this operation once. Keep it across retries of that operation; create a new one for a genuinely new submission.
Do not assume that matching request bodies necessarily represent a duplicate. Two legitimate applications can contain the same fields. Conversely, do not accept a reused identifier with materially changed content as though it were the original request. Decide how the service detects that mismatch and responds.
Scope the identifier to the authenticated caller or tenant and operation type. Otherwise one caller may collide with another or use a guessed identifier to retrieve an outcome it should not see. Record how long the service remembers the operation and what a retry beyond that period means.
The Amazon Builders' Library article on idempotent APIs explains why explicit caller intent and parameter mismatch matter. The exact contract still belongs in the service design.
Commit the operation record with the business change
Checking a cache, creating a case and then writing the idempotency key leaves a window for duplicate work. Two workers may both observe that the key is absent. Or a process may commit the case and fail before recording the key.
In a relational design, an appropriate transaction and uniqueness constraint can bind the operation record to the business change. Decide which result is retained for a repeated call: the case identifier, the operation status and enough response information to honour the API contract. Do not store sensitive request bodies merely because they make comparison convenient.
| Failure point | Required observable behaviour |
|---|---|
| Before the transaction commits | Retry may perform the operation |
| After commit, before the response arrives | Retry returns the recorded outcome rather than creating another case |
| Two attempts race | One logical operation is committed; the other follows the duplicate/in-progress contract |
| Same identifier, different material parameters | Reject the mismatch explicitly |
| Retained operation record has expired | Apply the documented expiry/reconciliation contract; do not imply indefinite protection |
This table is a design example. Implementations need to account for transaction isolation, crash recovery, access control and the actual data store. A mutex inside one application process does not coordinate every instance of a distributed service.
Include the message after the database write
Creating the case may also require notifying another service. A database transaction cannot by itself make a separate network call happen atomically. If the process fails after the case commits but before the notification, the case exists and the recipient knows nothing about it.
An outbox design records the business change and the intent to publish in one database transaction. A separate publisher sends committed outbox records. Publication can still be repeated, for example if acknowledgement is lost after a successful send, so the consumer needs its own duplicate-handling contract.
AWS's transactional outbox guidance describes the dual-write problem and the need to consider duplicate messages and ordering. It does not turn an entire cross-organisation workflow into an unconditional exactly-once system.
Give uncertainty a state someone can inspect
Useful states might distinguish accepted, processing, completed and awaiting reconciliation. Do not invent completion merely because a retry budget ran out. If a downstream system timed out after receiving a request, repeating it without a stable operation identifier can create the same problem at the next boundary.
Where the downstream API supports idempotency, carry an appropriate stable identifier across that call. Where it does not, decide how to query the result or reconcile the ambiguity. Sometimes an operational queue with a named owner is more honest than automatic retries that multiply side effects.
Bound retry attempts, use an appropriate delay policy and consider the total retry load across layers. A browser, API gateway, worker and client library can each be retrying the same dependency. The combined behaviour matters more than the setting in any one component.
Test the crash points, not only repeated success
Sending the same successful request twice is a useful start. Also test concurrent attempts, a timeout after commit, a publisher restart, repeated notification delivery and a different payload under the same identifier.
For each case, inspect the actual durable records. Count business effects, not just HTTP responses. If a test uses a mock that always executes transaction steps in order, it cannot establish the real database's behaviour under a race.
Use a test environment and safe data. The purpose is to expose failure handling deliberately, not to discover it by duplicating real resident actions. Keep the evidence tied to the application and schema revision that was exercised.
Leave an operating route for the cases automation cannot settle
An operator should be able to find the operation by a safe correlation identifier, see the last durable state, identify any pending outbox work and follow the reconciliation procedure. Metrics should distinguish retries that resolved uncertainty from repeated attempts that remain blocked.
The design has succeeded when a lost response does not become a second case, and when unresolved outcomes remain visible to someone authorised to resolve them. That requires both durable engineering and a service contract people can understand.
See software engineering or discuss a service or integration.
Further reading
Amazon Builders' Library: making retries safe with idempotent APIs
