How many documents to put in one batch request

Why the default answer is wrong here

Batching reduces round trips, which is the main cost in a large run, and it enlarges the unit of failure, which is the main risk. One request carrying four hundred documents is efficient until it fails, at which point four hundred documents failed together and the retry decision covers all of them. There is also a hard constraint: the batch endpoint caps the number of requests per call, and the cap depends on the plan, so the size is not purely a tuning choice.

The decisions

Reasons rather than a description of the code. Each has a default that is defensible in general and wrong for documents specifically.

• Respect the endpoint's cap first. POST /v1/pdf/batch rejects a requests array above the per-plan limit with a 400 that names the limit, so read it from the error rather than hard-coding a guess.
• Size below the cap for failure reasons rather than at it. A batch is the unit of retry, so the batch size is the amount of work you are willing to repeat.
• Keep batches small enough that a single failure does not delay everything behind it, which matters more when the run has a deadline than when it does not.
• Send shared options once at the top level rather than repeating them per request, which the endpoint supports and which keeps the payload smaller.
• Watch the payload size as well as the count. A batch of large documents can hit a size limit before it hits the count limit, and that failure is a different error.
• Measure rather than reason. The right size depends on document size and on your own downstream, and it is a ten-minute experiment.

In practice

A fragment, with the thing that goes wrong kept in a comment where it is the more instructive half.

// The cap is per plan and the error names it, so discover it rather
// than hard-coding a guess.
const BATCH = Number(process.env.BATCH_SIZE ?? 25);

async function renderBatch(items) {
  const res = await pdf.post("/v1/pdf/batch", {
    // Shared options once, not repeated per request.
    options: { format: "A4" },
    requests: items.map((i) => ({
      html: renderPayslip(i),
      filename: `payslip-${i.period}-${i.employeeId}.pdf`,
    })),
  });
  return res.results;
}

for (const chunk of chunks(items, BATCH)) {
  try {
    const results = await renderBatch(chunk);
    await recordResults(chunk, results);
  } catch (err) {
    if (err.status === 400 && /limited to (\d+)/.test(err.message)) {
      // The endpoint told us the cap. Use it rather than guessing again.
      const cap = Number(RegExp.$1);
      log.warn("batch cap is %d, resizing", cap);
      return renderInChunks(items, cap);
    }
    throw err;
  }
}

/* The size is the amount of work you are willing to repeat, because the
   batch is the unit of retry. Twenty-five failing is an inconvenience;
   four hundred failing is an incident.                               */

What people do instead

Maximising the batch size for throughput. It optimises the cheap axis, round trips, at the cost of the expensive one, which is how much has to be redone when something fails partway through a run with a deadline.

What the symptom looks like

A 400 naming the limit is the endpoint telling you the cap for your plan. Treat it as configuration discovered at runtime rather than as an error to log and move past.

Frequently asked

Related operational topics

The decisions that depend on each other, then the rest of the same group.