Retrying a render safely, and which failures are safe to retry

Why the default answer is wrong here

A retry is safe when the operation is idempotent and pointless when the failure is deterministic. Document pipelines routinely get both wrong: they retry a validation error that will fail identically every time, burning attempts and delaying the real error, and they retry a timeout without idempotency, which is the case most likely to have partially succeeded. The distinction is knowable from the error, because the failures fall into clear classes.

The decisions

Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.

• Retry transient failures only: a timeout, a renderer that was unavailable, a storage backend that was briefly unreachable, a rate limit. Those recover on their own.
• Never retry a request rejection. An invalid request, a payload that was too large, an unauthorized call and an expired key all fail identically on the second attempt, and retrying them delays the error somebody needs to see.
• Retry with exponential backoff and jitter, because a fleet retrying in lockstep after a shared failure produces a second outage on recovery.
• Honour Retry-After when it is present rather than using your own interval, since the server is telling you when it will be ready.
• Cap the total attempts and the total elapsed time, and make the give-up path do something visible rather than swallowing the failure.
• Only retry at all if the operation is idempotent by the previous page's definition. If it is not, fix that first, because retry without idempotency is a duplicate generator with a delay built in.

In practice

A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.

// The classes are knowable from the error, so the decision is a lookup
// rather than a judgement call at the call site.
const TRANSIENT = new Set([
  "timeout",
  "renderer_unavailable",
  "storage_unavailable",
  "db_error",
]);

const PERMANENT = new Set([
  "invalid_request",       // fails identically every time
  "payload_too_large",
  "unauthorized",
  "key_expired",
  "plan_required",
  "ssrf_blocked",
  "resource_load_failed",  // the asset URL is wrong; retrying will not fix it
]);

async function renderWithRetry(body, { attempts = 4 } = {}) {
  for (let attempt = 1; ; attempt += 1) {
    try {
      return await pdf.post("/v1/pdf", body);
    } catch (err) {
      if (PERMANENT.has(err.code)) throw err;          // surface it now
      if (!TRANSIENT.has(err.code) && err.status !== 429) throw err;
      if (attempt >= attempts) throw err;

      // The server's own answer beats our guess.
      const retryAfter = Number(err.headers?.["retry-after"]);
      const wait = Number.isFinite(retryAfter)
        ? retryAfter * 1000
        : Math.min(2 ** attempt * 250, 8000) * (0.5 + Math.random());

      await sleep(wait);   // jitter, so a fleet does not retry in lockstep
    }
  }
}

What people do instead

A generic HTTP client retry policy applied to the document endpoint. It retries everything including the permanent failures, so a malformed template produces four identical rejections and an error message four times slower than it needed to be, and a timeout produces two documents.

What the symptom looks like

A rising retry count with a stable success rate means something upstream is slow rather than broken. A rising retry count on permanent codes means the retry policy is misconfigured, and it is worth counting retries by error code for exactly that reason.

Frequently asked

Related operational topics

The decisions that depend on each other, then the rest of the same group.