Field note 03 / Integrations
Before you retry that request.
Handling lost API responses and duplicate records in CRM integrations.
A timeout can leave the result uncertain. Before retrying, establish whether the other system may already have applied the change.
Consider a request to create a record. The receiving system creates it, but the response never reaches the integration. Sending the same create request again could produce a second record. The retry policy needs to account for that uncertainty.
Identify repeated work
Depending on the receiving API, the design might use an idempotency key, an upsert with a stable identifier, or a check of the existing result before deciding what to do next. Each approach depends on the API's guarantees.
An idempotency key identifies an operation so the receiving system can recognize a retry. Reuse it for the same operation only when the API supports that behavior. A later, intentional update is a separate operation.
A lookup before creating a record can help reconcile an uncertain result, but another worker may act between the lookup and the write. Check the API's uniqueness and concurrency guarantees before treating that sequence as duplicate prevention.
Amazon Builders' Library: making retries safe with idempotent APIs
Decide which failures should be retried
Temporary failures and invalid data need different treatment. A retry may help with an unavailable service. A rejected field value usually needs investigation or correction.
For operations that are safe to retry, use bounded retries with increasing delays and jitter to spread requests out. Respect the receiving API's retry guidance. Repeated immediate attempts can add load to a service that is already struggling.
Amazon Builders' Library: timeouts, retries, and backoff with jitter
Make the retry behavior testable
Suppose an integration creates a target record for source order ORD-104. In a test environment, let the receiving system commit the create, then drop the response before the caller receives it. These are illustrative acceptance checks for a design that supports safe retries.
| Test step | Expected result | Evidence to inspect |
|---|---|---|
| Create succeeds, but its response is lost | Treat the result as uncertain, rather than assuming the create failed. | The target record exists. The integration retains the source ID and operation identity. |
| Recover the same operation | Use the API's documented idempotency behavior, or reconcile the existing result. Keep one target record for ORD-104. | Inspect the target records and recovery log. A successful retry status alone is insufficient. |
| The result cannot be established safely | Hold the operation for review without issuing another blind create. | The exception identifies ORD-104, the attempted operation, and the person responsible. |
| A later intentional change arrives | Process it as a new operation against the same order. | The existing target record reflects the change. The new operation has its own identity. |
Give unresolved work an owner
When retries stop, someone needs enough context to act. Record the affected source ID, the attempted operation, and the failure. Agree who investigates, where corrections are made, and how to resume without blindly repeating completed work.
- Test a repeated request and inspect the resulting records.
- Test a lost response after the receiving system applies the change.
- Test invalid data and confirm it reaches the person responsible for correction.
- Test recovery after the retry limit is reached.
The ownership rules still come first. Decide which value should reach the other system before deciding how to retry its delivery.
Notes from building integrations,
backend systems, and custom applications.