Missing fonts in headless Chrome on Docker

Chromium asks fontconfig which face to draw a run of text in. fontconfig has no way to answer nothing matches: it returns the closest thing it has, always, and it is the caller's job to notice that the closest thing has no glyph for the character. Chromium does not treat that as an error either. A codepoint with no glyph is drawn as .notdef, which most fonts draw as an empty box.

Here is the same four-line document rendered twice in one container, once with the fonts the image ships and once with a fontconfig that can only see DejaVu and Liberation. The noise field collects page errors, console errors and failed requests.

--- image default ---
{"widths":[{"id":"chinese","width":86.3},{"id":"arabic","width":82.9},
           {"id":"hebrew","width":81.3},{"id":"cyrillic","width":79.8}],
 "pdfBytes":24667,"noise":[]}

--- dejavu + liberation only ---
{"widths":[{"id":"chinese","width":65},{"id":"arabic","width":96},
           {"id":"hebrew","width":80},{"id":"cyrillic","width":86}],
 "pdfBytes":8717,"noise":[]}

exit=0

Empty both times. Exit status zero both times. A valid PDF both times. There is no failure anywhere in the pipeline to catch, which is why this reaches customers instead of CI.

Look at the two byte counts above. Same markup, same four lines of text: 24,667 bytes when the glyphs were found and 8,717 bytes when they were not.

The broken PDF is a third of the size because there are no outlines to embed. A box is not a glyph, and a page of .notdef costs almost nothing to store. You can run this check on a PDF somebody emailed you, with nothing installed and no access to the machine that produced it: if a document full of Chinese weighs a few kilobytes, the Chinese is not in it.

# Inside the image, not on your laptop. Your laptop has fonts.
docker run --rm your-image:tag fc-match :lang=ar family

# Which families claim the script at all
docker run --rm your-image:tag sh -c 'fc-list :lang=ar family | sort -u'

fc-match answers the same question Chromium asks, so its answer is the decision that produced your document. fc-list answers a different one: which faces claim the script at all. You need both, because a script can be covered by a font that is never chosen, and that turns out to be the common case.

If the command is not in the image, it comes from the fontconfig package. Chromium already links the library; only the command line tools are missing.

One trap costs an afternoon if you meet it cold. fontconfig indexes coverage by language and territory, not by script subtag, so the tag from your HTML may match nothing:

# Prints nothing, even with every CJK font installed
fc-list :lang=zh-Hans family

# Prints twenty families
fc-list :lang=zh-cn family

There is an orthography file for zh-cn and none for zh-Hans, so the second query matches nothing and prints nothing. The empty output reads exactly like no CJK fonts are installed while the fonts sit on disk. Query with zh-cn, zh-tw and zh-hk.

This is where most write-ups stop, and it is where the interesting half starts. Install the Noto packages on a clean Debian and ask again:

docker run --rm debian:bookworm-slim sh -c '
  apt-get update -qq && apt-get install -y -qq fontconfig fonts-noto-core fonts-noto-cjk > /dev/null 2>&1
  for l in en ru ar he zh-cn zh-tw ja ko; do echo "$l => $(fc-match :lang=$l family)"; done
  echo "=== who claims arabic ==="
  fc-list :lang=ar family | sort -u
'

en => Noto Sans
ru => Noto Sans
ar => DejaVu Sans
he => DejaVu Sans
zh-cn => Noto Sans CJK SC
zh-tw => Noto Sans CJK TC
ja => Noto Sans CJK JP
ko => Noto Sans CJK KR

=== who claims arabic ===
DejaVu Sans
DejaVu Sans Mono
Noto Kufi Arabic
Noto Naskh Arabic
Noto Nastaliq Urdu
Noto Sans Arabic

Noto Sans Arabic is installed. It is in the list of families that claim Arabic, four lines below. fontconfig picks DejaVu Sans anyway, and it does the same for Hebrew.

Coverage and preference are two different things. Which face wins is decided by the ordering rules in /etc/fonts/conf.d, and the rules a distribution ships do not put Noto in front for every script. Nothing you install changes that, because the packages deliver files and the ordering lives somewhere else.

The second surprise is that adding more fonts can make the answer worse. An image that installs Chromium with its full dependency set picks up WenQuanYi Zen Hei and IPAGothic on the way. With those present and no ordering rule:

ja => WenQuanYi Zen Hei,文泉驛正黑,文泉驿正黑
ko => WenQuanYi Zen Hei,文泉驛正黑,文泉驿正黑

Japanese and Korean drawn by a Chinese face. Han unification means the characters exist and the page stays readable, so this survives every check that looks for boxes, while the characters whose standard forms differ between the regions come out wrong. A reader in Tokyo or Seoul sees it immediately.

Alpine is wrong in the opposite places. The same question, on alpine:3.22 with the Noto packages and no configuration:

en => Noto Sans
ru => Noto Sans
ar => Noto Sans Arabic
he => Noto Sans Hebrew
zh-cn => Noto Sans CJK JP
zh-tw => Noto Sans CJK JP
ja => Noto Sans CJK JP
ko => Noto Sans CJK JP

Arabic and Hebrew resolve correctly with no help at all, which is more than Debian manages. Every CJK language then resolves to the Japanese face, Chinese and Korean included. Whichever base image you are on, the packages are half of it.

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        fontconfig \
        fonts-dejavu-core \
        fonts-liberation \
        fonts-noto-cjk \
        fonts-noto-color-emoji \
        fonts-noto-core \
        fonts-noto-mono \
    && rm -rf /var/lib/apt/lists/*

COPY 60-fonts.conf /etc/fonts/conf.d/60-fonts.conf
RUN fc-cache -f

RUN apk add --no-cache \
        fontconfig \
        font-noto \
        font-noto-arabic \
        font-noto-cjk \
        font-noto-emoji \
        font-noto-hebrew \
        ttf-dejavu

COPY 60-fonts.conf /etc/fonts/conf.d/60-fonts.conf
RUN fc-cache -f

Two details in the ordering. Install the fonts before anything that installs Chromium, so the browser's own dependency resolution does not decide your font set for you. Run fc-cache -f after the last font lands, in the same layer if you can, because a stale cache is indistinguishable from a missing font at render time.

Then the half that the packages do not cover. This is the file that decides what gets chosen:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <alias>
    <family>sans-serif</family>
    <prefer>
      <family>Noto Sans</family>
      <family>Noto Sans Arabic</family>
      <family>Noto Sans Hebrew</family>
      <family>Noto Sans CJK SC</family>
      <family>Noto Sans CJK JP</family>
      <family>Noto Sans CJK KR</family>
      <family>Noto Color Emoji</family>
      <family>DejaVu Sans</family>
    </prefer>
  </alias>

  <match target="pattern">
    <test name="lang" compare="contains"><string>ar</string></test>
    <edit name="family" mode="prepend" binding="strong"><string>Noto Sans Arabic</string></edit>
  </match>

  <match target="pattern">
    <test name="lang" compare="contains"><string>he</string></test>
    <edit name="family" mode="prepend" binding="strong"><string>Noto Sans Hebrew</string></edit>
  </match>

  <match target="pattern">
    <test name="lang" compare="contains"><string>ja</string></test>
    <edit name="family" mode="prepend" binding="strong"><string>Noto Sans CJK JP</string></edit>
  </match>

  <match target="pattern">
    <test name="lang" compare="contains"><string>ko</string></test>
    <edit name="family" mode="prepend" binding="strong"><string>Noto Sans CJK KR</string></edit>
  </match>

  <match target="pattern">
    <test name="lang" compare="contains"><string>zh</string></test>
    <edit name="family" mode="prepend" binding="strong"><string>Noto Sans CJK SC</string></edit>
  </match>

  <!-- After the zh rule, never before it. See below. -->
  <match target="pattern">
    <test name="lang" compare="contains"><string>zh-tw</string></test>
    <edit name="family" mode="prepend" binding="strong"><string>Noto Sans CJK TC</string></edit>
  </match>

  <match target="pattern">
    <test name="lang" compare="contains"><string>zh-hant</string></test>
    <edit name="family" mode="prepend" binding="strong"><string>Noto Sans CJK TC</string></edit>
  </match>

  <match target="pattern">
    <test name="lang" compare="contains"><string>zh-hk</string></test>
    <edit name="family" mode="prepend" binding="strong"><string>Noto Sans CJK HK</string></edit>
  </match>
</fontconfig>

The alias block and the match blocks answer two different questions and you need both. The alias decides what sans-serif means. It does not decide what happens when the chosen family has no glyph for a character and fontconfig is asked again with a language hint, and that second question is the one that decides an Arabic or Hebrew document. Left alone it answers DejaVu Sans, which is what the Debian output above shows.

The rule order at the bottom is worth reading twice, because it is easy to get backwards and it fails quietly when you do. A lang test of zh matches zh-TW and zh-HK as well, so a single rule preferring the Simplified face takes the Traditional documents too. prepend puts a family at the front of the list, so the rule that runs last is the rule that wins: the specific rules go after the general one. Both spellings are handled because both reach fontconfig, and neither covers the other. A document may say zh-TW, which fontconfig knows as an orthography, or zh-Hant, which it does not know at all and can only compare as a string.

docker run --rm your-image:tag sh -c '
  for l in en ru el ar he zh-Hans zh-Hant zh-TW ja ko hi th; do
    echo "$l => $(fc-match :lang=$l family)"
  done
'

A container with the fonts and the rules above answers:

en => Noto Sans
ru => Noto Sans
el => Noto Sans
ar => Noto Sans Arabic
he => Noto Sans Hebrew
zh-Hans => Noto Sans CJK SC
zh-Hant => Noto Sans CJK TC
zh-TW => Noto Sans CJK TC
ja => Noto Sans CJK JP
ko => Noto Sans CJK KR
hi => FreeSans
th => Loma

A check that has never failed has not been shown to check anything, and this one is easy to point at a broken state without touching your image. FONTCONFIG_FILE replaces the whole configuration for one process, so you can ask what the answer would be if DejaVu were the only font on disk:

docker run --rm your-image:tag sh -c '
  printf %s "<fontconfig><dir>/usr/share/fonts/truetype/dejavu</dir><cachedir>/tmp/fc</cachedir></fontconfig>" > /tmp/dejavu-only.conf
  echo "zh-cn => $(FONTCONFIG_FILE=/tmp/dejavu-only.conf fc-match :lang=zh-cn family)"
  echo "ar    => $(FONTCONFIG_FILE=/tmp/dejavu-only.conf fc-match :lang=ar family)"
'

zh-cn => DejaVu Sans
ar    => DejaVu Sans

Run your check against that and watch it say so. If it still passes, it was measuring something else.

One limit worth knowing: fc-match tells you what fontconfig decided, not what Chromium drew. A family name misspelled in CSS falls back without a word and leaves fontconfig's answer looking perfect. The check that catches both is to render the string twice, once under sans-serif and once under the family by name, and compare the widths. A few percent apart means one face drew both. Far apart means one of them fell back, because .notdef boxes are all one width: the Chinese line in the first output on this page measures 86.3 pixels when the glyphs are found and 65.0 when they are not.

Direction is a separate problem with a separate fix. A face with complete Arabic or Hebrew coverage still lays the line out left to right unless the markup says otherwise, and the result looks nothing like a missing glyph: the characters are all there and the order is wrong. Fixing the font does not touch it.

Arabic is worth reading on its own, because it fails in two ways that look nothing alike and the better known fix only addresses one of them. Arabic not showing in PDF has the coverage measurements for both fonts, and settles whether DejaVu can join letters.

Colour emoji need a family of their own, which is a separate package on both distributions and is in both samples above. The CJK variant question outlives the missing-glyph one: once a face is chosen, a document that does not say which language it is in gets whichever variant the rules list first, so CJK documents should carry a lang attribute rather than rely on the ordering.

If the script that fails is one your base image already covers, the cause is usually somewhere else entirely. DejaVu carries Cyrillic, Greek and Hebrew, so a Debian image renders those with no font work at all, and a document that comes out empty anyway is normally pointing at a web font: an @font-face subset built for Latin has no Cyrillic in it, and the browser will use it and fall back per character. Check what the page loads before you change the image.