Operations and scale for document generation

Making a repeated request produce the same document rather than a second one, which matters more here than in most APIs because the artefact is often numbered.

Deciding which failures should be retried and which should not, because a blanket retry on a document endpoint is a duplication mechanism.

Collapsing concurrent identical requests, which is a different problem from idempotency because both arrive before either has finished.

Serving a previously rendered document instead of making a new one, which is nearly free when the inputs are unchanged and wrong when they are not.

The full list of inputs a rendered document depends on, which is longer than the data and is the reason document caches serve stale output.

Structuring a job that produces thousands of documents at once, where the interesting decisions are about failure and progress rather than throughput.

Choosing how much work goes in a single call, which trades round trips against blast radius and against a real per-plan limit.

The normal outcome of a large batch, which is neither success nor failure, and which most client code has no branch for.

The handling for work that has exhausted its retries, which for documents means somebody is missing something they were promised.

Whether the user waits for the document, which is a decision about the document's size and the user's expectation rather than about performance.

Being told when work finished instead of asking repeatedly, which removes polling and adds an endpoint that has to be correct about who is calling it.

Running a document job on a calendar, where the interesting failures are a run that did not happen and a run that happened twice.

What the system does when demand exceeds capacity, which is a decision to make rather than a behaviour to discover.

Building the caller so it stays within a limit rather than discovering it, which is a different question from what to do when a 429 arrives.

The fields that make a render diagnosable weeks later, and the one thing that must not be in the log at any level.

Following a single document across the queue, the render and the delivery, which is the only way to answer where the time went or where it stopped.

Recording the distribution rather than the mean, because document render times are skewed by document size and the tail is what people experience.

Making the timeouts at each hop consistent with each other, because a path whose timeouts disagree fails in the least useful way available.

Choosing signals that catch the failures that matter, which for documents includes several that produce no error at all.

Keeping enough operational history that a specific document's story can be reconstructed, which log retention alone usually cannot do.

Re-running a set of documents that were produced wrongly, which is a recovery operation with its own risks and needs planning before it is needed.

The decisions to make when rendering is unavailable, most of which are about what to tell people and what to preserve rather than about restoring service.

Adding up the real per-document cost, which is more than the render and is the number every capacity and caching decision depends on.

Sizing for a workload that is idle most of the time and enormous occasionally, which is the normal shape for documents and the wrong shape for averages.

The latency difference between a pipeline that has been busy and one that has been idle, which for a monthly workload means every run starts cold.

Picking a concurrency that uses the available capacity without exceeding it, which is a measured number rather than a guessed one.

Deciding what happens to a document after it is delivered, which is a policy question with a cost attached and a legal floor underneath it.

Tracking consumption against the allowance so a limit is a planned event rather than a failed monthly run.

Producing something useful when the full document cannot be made, which is sometimes right and frequently worse than failing.