PDFPipe

Container orchestration / Kubernetes

wkhtmltopdf on Kubernetes: processes that never exit

Over hours or days the pod slows down and eventually stops responding. Process listings show a growing number of wkhtmltopdf processes in a defunct state.

What is actually wrong

Each render is a subprocess. If the parent does not wait on it, or if the render hangs on a resource that never loads and the parent gives up without killing the child, the process becomes a zombie. In a container, PID 1 is your application rather than an init system, and an application that was not written to reap children does not reap them. Zombies consume process table entries until the pod hits its limit.

What Kubernetes gives you to work with

the same container as Docker, scheduled onto a node with a memory limit the kernel enforces by killing things.

  • Packaging: no image size limit, but a large image slows every scale-up, because a new pod on a cold node waits for the pull.
  • Filesystem: as configured. /dev/shm defaults to 64 MB and is the single most common cause of browser crashes in a cluster.
  • Processes: full control inside the pod. Memory limits are enforced by the kernel out-of-memory killer, which terminates the process without a stack trace and without an application-level error.

The fix, if you are keeping wkhtmltopdf

Run an init as PID 1 so orphans are reaped, and give every render a timeout that kills the process rather than only abandoning it.

yaml
spec:
  containers:
    - name: renderer
      # tini as PID 1: reaps orphans, forwards signals.
      command: ["/usr/bin/tini", "--", "node", "server.js"]
      env:
        # A render that hangs on a remote asset must be killed, not
        # merely abandoned by the caller.
        - name: RENDER_TIMEOUT_SECONDS
          value: "30"

What still hurts after that

A timeout turns a hang into a failed render, which is better but is still a failed render, and the common cause is a remote image or stylesheet the document references that is slow or gone. The engine has no concept of waiting for the network to settle, so a document that depends on external assets is a source of intermittent failures that reproduce only under real network conditions.

What wkhtmltopdf is genuinely good at

it is a single binary with no runtime, so once it is present it does not care what language your application is written in. That is worth keeping in view: the problem on this page is where it runs, not what it produces. wkhtmltopdf is a command line tool built on an old fork of the Qt WebKit engine, invoked as a subprocess with an HTML file and an output path. It needs the binary itself, the X11 and font shared libraries it links against even in headless mode, and a font configuration, because it ships with no fonts of its own.

The other shape of the answer

Every fix above keeps the renderer inside your deployment, which means the constraint stays yours: the package size, the shared libraries, the fonts, the memory, the patching. The alternative is an HTTP call from Kubernetes to something that already has a browser in it, which turns all of the above into a request and a response. That is what this API is, and it is the reason these pages exist. Whether that trade is right depends on whether rendering is a thing you want to operate.

Frequently asked

Can wkhtmltopdf run on Kubernetes at all?

Run an init as PID 1 so orphans are reaped, and give every render a timeout that kills the process rather than only abandoning it. Whether that is worth doing is a separate question from whether it is possible, and the section above on what still hurts is the honest answer to it.

Why does it work locally?

Because a development machine has root, a package manager, several gigabytes of memory, a full font set and no packaging limit. Every one of those is a thing Kubernetes constrains. A renderer that works locally and fails on deploy has not regressed: it has met the first environment that does not give it everything.

Related failures

The same platform with a different renderer, and the same renderer on other platforms.

If the render moves off this platform, none of the above is your problem to keep solving. 100 free documents a month, and a playground that needs no signup.