Puppeteer on Google Cloud Run: 64 MB of shared memory

What is actually wrong

Chromium puts a lot of inter-process traffic through /dev/shm, and in a container that mount defaults to 64 MB. One render at a time stays under it. Several concurrent renders of a document with images do not, and the tab process dies. The container's memory limit is a different number and it is not the one being exceeded, which is why the metric you are watching looks healthy while the failures happen.

What Google Cloud Run gives you to work with

your container image, started on demand and scaled to zero, with requests routed into it.

• Packaging: no meaningful image size limit. Image size shows up as cold start latency instead, because the image has to be pulled before the first request is served.
• Filesystem: the container filesystem is writable, but writes go to memory and count against the instance memory limit. A process that writes a large temporary file is really allocating memory, and it will be killed for it.
• Processes: full process control inside the container. The default request concurrency is high, which means many simultaneous renders inside one instance rather than one per instance.

The fix, if you are keeping Puppeteer

Launch the browser with shared memory writes redirected to the ordinary filesystem, which is the documented flag for exactly this situation. Then look at request concurrency, because the default is high and it means many simultaneous renders inside one instance rather than one instance each.

const browser = await puppeteer.launch({
  args: [
    // Writes to /tmp instead of the 64 MB /dev/shm mount.
    "--disable-dev-shm-usage",
    "--no-sandbox",
  ],
});

// And in the service config, because the default concurrency means
// dozens of simultaneous renders share one instance's memory.
// gcloud run deploy --concurrency=4 --memory=2Gi

What still hurts after that

Redirecting shared memory to disk moves the pressure onto the container filesystem, and on Cloud Run the container filesystem is memory, so the same bytes still count against the same limit by a longer route. Lowering concurrency fixes the crashes and multiplies the instance count, which is where the cost of this approach actually shows up. The image is also large, and image size is cold start latency on a service that scales to zero.

What Puppeteer is genuinely good at

it renders exactly what a browser renders, which means modern CSS, web fonts and client-side JavaScript all behave the way they do in a tab. That is worth keeping in view: the problem on this page is where it runs, not what it produces. Puppeteer is a Node library that drives a real Chromium, and downloads that Chromium into node_modules when it is installed. It needs the browser binary, a long list of shared libraries the browser links against, fonts, and enough memory for a browser process that is not accounted for by your runtime's heap settings.

The other shape of the answer

Every fix above keeps the renderer inside your deployment, which means the constraint stays yours: the package size, the shared libraries, the fonts, the memory, the patching. The alternative is an HTTP call from Google Cloud Run to something that already has a browser in it, which turns all of the above into a request and a response. That is what this API is, and it is the reason these pages exist. Whether that trade is right depends on whether rendering is a thing you want to operate.

Frequently asked

Related failures

The same platform with a different renderer, and the same renderer on other platforms.