| Author: Abdullah Ahmed | Category: API Development and Integration
An order submission times out after the customer presses confirm. The application does not know whether the supplier accepted the order, so it shows an error and invites another attempt. If the first request succeeded, that helpful-looking retry can create a second order. The difficult part of API failure handling is often uncertainty about the outcome.
Reliable integrations distinguish failures that can be corrected, failures that may be retried, and operations whose result must first be established. They also give customers and staff a clear account of what is known. A graceful response preserves business correctness while giving the system a controlled path back to normal operation.
Classify failures by the decision they require
Start with categories that lead to different actions. Invalid input needs correction. Missing permission needs an access decision. An expired credential may need renewal. A temporary capacity problem may justify waiting. An unknown result after a write may require reconciliation before any repetition.
HTTP status codes provide useful information, but the application's contract and operation semantics matter too. A successful HTTP response can describe a business request that is still pending or has been rejected by a domain rule. Conversely, a network interruption may occur after the server committed the action.
RFC 9110 defines HTTP semantics, including status codes and idempotent methods. Use those semantics as a foundation, then document the additional business behaviour a client needs to interpret the specific API safely.
Set time limits around the whole operation
An integration should not wait indefinitely for a dependency. Define connection and response limits appropriate to the operation, and consider an overall deadline for the complete user request. Several individually reasonable waits can add up to an unacceptable experience when calls occur in sequence.
Choose limits using observed behaviour and business needs rather than copying an arbitrary value. A background export can tolerate a different waiting strategy from an interactive stock check. Review the limits when the dependency or workload changes, and measure how often legitimate work exceeds them.
Decide what happens when the caller stops waiting. The downstream operation may continue even if the user-facing request is cancelled. Where cancellation is supported, understand its guarantees. Where it is not, preserve enough context to discover the eventual outcome instead of assuming the work disappeared.
Make retries conditional and bounded
Retry only when the failure and the operation make repetition appropriate. A validation error will not improve through repeated identical requests. A permission failure may require a human or credential-management action. Even a temporary server failure does not automatically make a consequential write safe to repeat.
For eligible operations, use a bounded retry policy with increasing delays and random variation where appropriate. Respect documented waiting instructions and rate limits. Set a maximum attempt count or elapsed-time budget so a failed dependency does not keep work alive indefinitely.
Avoid stacking independent retry loops across every layer. A browser, application service, client library, and background worker can multiply attempts unexpectedly. Identify which layer owns the retry policy and make the others cooperate. Monitor actual attempt counts to confirm that production behaviour matches the intended design.
Protect consequential writes from duplication
An idempotency mechanism lets a caller repeat a logical operation without causing the same effect again, within the API's documented guarantees. When supported, create a stable key for the business operation and reuse it for retries of that operation. Generating a new key on every attempt defeats the purpose.
Persist the key and relevant request context before the outcome can become uncertain. Understand the provider's key scope, retention period, and response to reuse with different data. These details determine whether a later retry is still protected and whether the caller needs another recovery path.
If the API lacks suitable duplicate protection, look for a supported external reference or status query. Design a reconciliation process around it. For a high-consequence action with no reliable way to establish the outcome, automated repetition may need to stop and place the case in a controlled review queue.
Represent pending and uncertain states honestly
A user interface should distinguish a confirmed failure from an operation still being checked. “We could not confirm the result yet” can be more accurate than “Your order failed”. Explain what the user should do next and whether another submission could cause a duplicate.
Give the operation a reference and a stable place to view progress. Preserve entered information where appropriate. If the system can continue safely in the background, communicate that behaviour and notify the user when the outcome is known. Avoid a spinner that continues indefinitely without context.
Define state transitions explicitly. A request might move from received to submitted, then confirmed, rejected, or requiring review. Record the evidence behind transitions, especially when external notifications and polling can both update the same record. Conflicting messages should trigger a deliberate resolution rule rather than whichever update arrives last.
Use error responses that machines and people can understand
Return a stable error category that clients can act on, a useful human explanation, and appropriate contextual details. For input problems, identify the relevant field without exposing sensitive content. For support investigations, provide a reference that connects to internal diagnostics.
RFC 9457 specifies problem details for HTTP APIs, including fields such as type, title, status, detail, and instance. A consistent structure helps clients handle errors, while the API's own documentation must explain the meaning and recovery policy for each problem type.
Do not expose stack traces, credentials, internal queries, or unrestricted downstream responses. Keep diagnostic detail in appropriately protected logs. Human-readable wording may evolve, so clients should not depend on parsing an English sentence to decide whether an operation can be retried.
Prevent one dependency from exhausting the service
A slow external API can consume connections, worker capacity, and memory even when the rest of the application is healthy. Limit concurrent work directed at the dependency and consider separate resource pools for unrelated functions. The goal is to keep one failing connection from blocking every customer journey.
A circuit breaker can temporarily stop calls after a defined pattern of failures and later allow controlled probes for recovery. Its thresholds and state transitions need to match the workload. An overly sensitive breaker may interrupt a usable service; an ineffective one may add complexity without reducing pressure.
Decide the business response while calls are restricted. Some reads may use clearly labelled recent data. Some optional features may be unavailable. Essential writes may need a durable queue or a clear refusal. A fallback is acceptable only when its information and consequences remain suitable for the task.
Queue work with an operating plan
Background processing can separate a customer's immediate request from a slower external action. Accept work only when it has been stored durably enough for the promised behaviour. A success message should not depend on a temporary in-memory task that may disappear when the process restarts.
Define retry, expiry, and failure handling for queued items. Some work becomes invalid with age: a reservation request may no longer make sense after its intended time. Avoid replaying an old backlog without checking the business context. Include a controlled path for items that cannot be processed automatically.
Give operations visibility into queue age and affected business records. A count of failed messages is a starting point, but staff need to know which customers or tasks require action. Document who can retry, cancel, or repair an item and how those actions avoid duplicate effects.
Reconcile instead of relying on delivery alone
Notifications can be delayed, duplicated, or unavailable. A reconciliation process compares local understanding with an authoritative external state and identifies disagreements. It provides a second path to correctness when an individual request or event does not produce a dependable final result.
Choose the reconciliation scope and frequency based on consequence and volume. A payment-related workflow may need prompt investigation of uncertain outcomes, while a noncritical catalogue update can tolerate a different schedule. Keep the process bounded so it does not overload the dependency during recovery.
Record what was compared, which differences were found, and how they were resolved. Some discrepancies can be repaired automatically under clear rules; others need review. Make the distinction explicit and preserve an audit trail for actions that change business records.
Observe the customer impact
Collect technical signals such as latency, error categories, retry counts, and queue age, then connect them to business outcomes. A small number of failed requests may matter greatly if they block every approval in a critical workflow. A large number of harmless polling errors may have less immediate impact.
Use correlation identifiers across internal steps and external references where supported. Avoid placing secrets or unnecessary personal information in logs. Give support staff a safe way to locate the relevant operation without requiring customers to provide raw technical payloads.
Alert on conditions that have an owner and an actionable response. Include a link to the recovery procedure and relevant dashboards. Review noisy alerts and missing signals after incidents so monitoring evolves with the service rather than accumulating unattended notifications.
Test uncertainty, not only obvious errors
Exercise a timeout before the server receives a request and a lost response after the server accepts it. Those cases can look identical to the client while requiring different reasoning. Test duplicate notifications, out-of-order updates, expired credentials, malformed responses, and rate-limit behaviour in a controlled environment.
Check the user experience and the stored business state after each exercise. It is not enough for the application to avoid crashing. Verify that no duplicate order was created, pending work remains discoverable, and staff can resolve the exception through supported controls.
Test recovery as well as failure. Restore the dependency, release queued work gradually, and confirm that normal traffic remains healthy. A backlog can create a second incident if every waiting operation retries at once. Rehearse the operating procedure with the people who will actually use it.
Walk through an uncertain order from end to end
Consider a procurement application sending an order to a supplier. Before submission, it stores a local operation record, a stable external reference, and the intended order details. It then sends the request using the supplier's documented duplicate-protection mechanism where available. A timeout moves the local operation into an unresolved state rather than immediately creating a new attempt with a new identity.
The application next uses the supported status or lookup method to establish whether the supplier accepted the order. If the result is still unavailable, it schedules a bounded follow-up and shows the requester that confirmation is pending. The user can inspect the same operation later instead of submitting an unrelated duplicate.
If an authenticated supplier notification arrives during that process, the application associates it with the existing operation and applies the documented state rules. A later polling result should not reverse a confirmed state merely because it reflects an older snapshot. Where the evidence conflicts and the contract provides no safe automatic resolution, the case enters a review queue.
An operator reviewing the case needs the external reference, the attempted time, the known responses, and supported actions. They should be able to confirm the outcome or cancel further attempts without editing database rows directly. Record their decision and preserve enough history to explain it later.
Test this sequence with a simulated lost response after acceptance. Verify that the supplier receives only the intended business order, the requester sees a coherent status, and the operation eventually reaches a final state or an owned exception. This is stronger evidence than merely checking that an exception handler returns a friendly message.
Finally, define what happens if the unresolved state lasts beyond the normal operating window. The appropriate response may be escalation, customer contact, or a controlled cancellation process, depending on the supplier contract. A pending state is useful only when it has a path toward resolution and someone responsible for cases that do not progress.
Document a failure policy for each important operation
For each consequential API call, record its business purpose, timeout budget, retry eligibility, duplicate protection, and reconciliation method. Include the customer-facing state and the team responsible for unresolved cases. This compact policy connects implementation details to operational ownership.
Review the policy when an API version, provider, or workflow changes. A newly asynchronous response or a shorter idempotency retention window can alter the safety of an existing client. Keep assumptions linked to the provider's current contract and verify them in relevant integration checks.
Begin with the operation whose uncertain outcome would cause the most harm. Trace what happens if the response disappears after success, then prove that the system can establish the result without repeating the effect. That exercise is a practical foundation for graceful failure handling across the rest of the integration.