HarfBuzz brings professional typography to the desktop

By their nature, low-level libraries go mostly unnoticed by users and even some programmers. Usually, they are only noticed when something goes wrong. However, HarfBuzz deserves to be an exception. Not only does the adoption of HarfBuzz mean that free software's ability to convert Unicode characters to a font's specific glyphs is as advanced as any proprietary equivalent, but its increasing use means that professional typography can now be done from the Linux desktop as easily as at a print shop.

"HarfBuzz" is a transliteration of the Persian for "open type." Partly, the name reflects that it is designed for use with OpenType, the dominant format for font files. Equally, though, it reflects the fact that the library's beginnings lie in the wish of Behdad Esfahbod, HarfBuzz's lead developer, to render Persian texts correctly on a computer.

"I grew up in a print shop," Esfahbod explained during a telephone interview. "My father was a printer, and his father was a printer. When I was nine, they got a PC, so my brother and I started learning programming on it." In university, Esfahbod tried to add support for Unicode, the industry standard for encoding text, to Microsoft Explorer 5. "We wanted to support Persian on the web," he said. "But the rendering was so bad, and we couldn't fix that, so we started hacking on Mozilla, which back then was Netscape."

Esfahbod's early interest in rendering Persian was the start of a fifteen-year effort to bring professional typography to every Unicode-supported script (writing system). It was an effort that led through working on the GNOME desktop for Red Hat to working on Firefox development at Mozilla and Chrome development at Google, with Esfahbod always moving on amiably to wherever he could devote the most time to perfecting HarfBuzz. The first general release was reached in 2015, and Esfahbod continues to work on related font technologies to this day.

At the beginning of the project, three text renderers were in use on the Linux desktop. The original HarfBuzz code evolved from FreeType, with some borrowing from the Pango and Qt font-rendering systems. The problem with these strands of development was that none of the three was consistent with either of the others. The same characters in the same fonts could be displayed differently in different applications and might look different when printed than on the screen. Esfahbod recalls reading bug reports on Red Hat's internationalization efforts in which it was often impossible to know whether the source of the problems was the fonts or the different implementations.

Eventually, the code for the three implementations was merged under an MIT license, producing what is known as Old HarfBuzz. In the spirit of openness, Old HarfBuzz is still available from the project web site, but for historical reference only. The HarfBuzz code in use today is sometimes known as New HarfBuzz, or (formerly) as harfbuzz-ng. Today, it is available for all major operating systems, and incorporated in Firefox, GNOME, ChromeOS, Android, and KDE, reflecting the companies where Esfahbod has worked and the projects to which he has contributed over the years. In 2017, LibreOffice 5.3 joined the list of HarfBuzz users, and most other free-software projects are following HarfBuzz closely, especially those dealing with graphics and layout. For example, Inkscape has switched to HarfBuzz, and GIMP, Krita, and Scribus are planning to switch in upcoming releases. Yet while HarfBuzz is a significant development, its adoption is happening mostly unnoticed.

The importance of shaping engines

Esfahbod defines HarfBuzz as a shaping engine rather than a layout engine. That is, HarfBuzz is concerned with the consistent graphical representation of Unicode characters in fonts. By contrast, a layout engine is concerned with bodies of text and such issues as the co-existence of different scripts or writing systems, how text breaks at the end of the line, and whether text flows from left to right or right to left.

Because different writing systems display text differently, each script supported by Unicode requires its own shaping engine. In some languages, the shaping engine is straightforward. In Latin-based scripts, such as English, French, or German, and in Greek, Chinese, Japanese, and Korean, one glyph follows another in a set order, and the placement of a new glyph does not affect the positioning of the previous glyphs. Even the addition of accents and diacritical marks, or the use of ligatures (glyphs that are redrawings of letter combinations that would otherwise be poorly spaced) do not greatly complicate rendering. For this reason, Esfahbod describes the shaping engines of these scripts as "simple."

Esfahbod contrasts simple scripts with what he calls "complex" scripts. Examples of complex scripts include Arabic and Persian, in which the shape and position of each glyph is determined by those around it, and in which some glyphs are connected while others are written separately. Even more complicated are Indic scripts, Javanese, and Southeast Asian scripts. In some complex scripts, Esfahbod explains, ten glyphs can form a single syllable, and in some instances, the last glyph can be placed above the others, so that it is viewed first. Necessarily, the shaping engines of complex scripts must be much more detailed than those of simple scripts. In fact, to the users of simple scripts such as English, complex scripts can initially seem overwhelmingly bewildering. Yet with the internationalization of computing, the need to render complex scripts correctly is greater than ever before.

What HarfBuzz does is provide the shaping engines for all 139 writing systems included in Unicode. Esfahbod has joked in presentations that he asked his supervisors at Google to download the entire Internet for him so that the project could obtain enough samples to derive the rules required for the shaping engine for each script. In the end, he settled for the more manageable sample provided by different localizations of Wikipedia.

What HarfBuzz does

The greatest changes brought about by HarfBuzz is that it places all Unicode-supported scripts on an equal footing on all computing platforms. With consistent rendering, minority and non-European languages are now much easier to use in computing so long as a font supporting them is installed. In addition, when necessary, scripts can be mixed more easily in a single passage. However, the new ease of incorporating Unicode scripts also means some changes to Western European languages like English, which have long had a privileged position in computing.

To start with, the growing use of HarfBuzz means that free software matches the Microsoft Universal Shaping Engine, which Esfahbod suspects drew at least inspiration from HarfBuzz. Together, the two systems have also de-emphasized Graphite, which SIL International, a Christian missionary organization, developed to produce texts in minority languages. Graphite still exists in open source, but the fact that it required each font to support it means that only a handful of fonts, such as Gentium and Linux Biolinium, ever supported it. By contrast, HarfBuzz works with all OpenType fonts, which may mean that Graphite will soon become obsolete.

For long-time designers, who have collected fonts over a couple of decades, another change caused by HarfBuzz is the dropping of Type 1 or Postscript fonts. Type 1 fonts were a format popular in the 1990s, when they competed with Microsoft's TrueType format. Both have been largely superseded by the OpenType format in the last fifteen years, but for some, a consequence of applications incorporating HarfBuzz is the unannounced loss of the use of their Type 1 fonts. However, those still using Type 1 fonts can install FontForge and run a script to batch convert them to OpenType quickly and without any loss of detail in the glyphs – a handy trick to know, since Esfahbod has stated that "I don't think we will ever support Type 1 in HarfBuzz" – although he adds that "I'd be happy to work on a converter."

Any drawbacks caused by these changes, though, are outweighed by HarfBuzz's advantages. To start with, according to Esfahbod, HarfBuzz renders fonts much more quickly than previous technologies. While HarfBuzz's finishing touches were being added, Esfahbod was also working on making HarfBuzz work well with Google's Chrome web browser. As Esfahbod tells it, a basic policy in Chrome development was that enhancements should never slow the application. As a result, porting HarfBuzz to Chrome required numerous optimizations. In Chrome's Windows version, the speed of rendering increased by four hundred percent, he claims. As a result, loading and scrolling through long documents in other applications should also be enhanced, although hard figures for any improvement are not available.

At first, Latin-based scripts like English might seem to be affected least by HarfBuzz. Yet, on closer examination, they, too, can benefit. For centuries, printing has been in the hands of professionals. The typewriter reduced that monopoly of expertise, but was unable to reproduce many of the features of professional printing. Full justification, for instance, was impossible on most typewriters and, for the most part, italics could only be indicated by underlining or, on an IBM Selectric, stopping to change the type ball.

In the 1980s, the word processor reduced the monopoly of expertise still further, bringing users many of the features that the typewriter either lacked or could only produce with difficulty. Yet advanced features were still lacking, or could only be added with makeshifts such as special character dialogs or LibreOffice's Typography Toolbar extension – which many users could not be bothered with. In English in particular, communication was mostly in ASCII. The increasing availability of diverse keyboard layouts helped to make computing more versatile, especially for Western European languages, yet the finishing touches of professional print shops remained unavailable.

Now, with HarfBuzz, those finishing touches are used automatically. Using ligatures no longer requires extra effort – when available, they are inserted automatically, eliminating unsightly letter combinations. True small capitals, which are smaller and different in design from the standard capital letters of the font so that they fit into other text more elegantly, can be used without switching to a special font or being indicated by clumsy small capitals manufactured – inevitably, incorrectly – by word processors. Similarly, instead of arranging numbers with a common base line, HarfBuzz inserts old-style figures, in which each numeral has its own baseline, which fits them more aesthetically into blocks of texts. With HarfBuzz, the only limitation is whether the font actually includes such features – which, increasingly, they do. At last, the revolution that began over three decades ago with the introduction of word processors is completed, and applications like LibreOffice can produce publication-ready copy as easily as a print shop.

Moreover, HarfBuzz also gives users control over which advanced features to use, rather than being dependent on what the font designers choose to give them. To customize any feature, users can add a small code snippet in a field that defines what font to use. For example, typing directly after the font name :smcp or :smcp=on converts regular capital letters to small capitals. If you want to turn off this feature in a font that enables them automatically, type :-smcp or :smcp=off. If you want to enter more than one snippet, add the second one directly after the first, with no space between them, so that :liga:onum::cpsp after the font name would force the use of standard ligatures, old style figures, and kerning for upper case capitals, always assuming that the font supports these features.

Make these changes in a paragraph style in a word processor like LibreOffice Writer, and you have a custom font without having to define it each time. You might also create a character style for a feature like kerning (the adjustment of the space between letters), and apply it only when you think it necessary.

Which of these features are available in a font depends on the designer. Some features may be included in a font file, but turned off by default. You may also need to experiment with some of the options to see if they make a visible or significant difference with a given font. The features are based on those used with OpenType in LaTeX, and a complete list is available on Wikipedia.

Further enhancements

Since HarfBuzz reached general release in 2015, Esfahbod has concentrated on related projects. One such project is Noto, a font whose goal is to include all of the approximately 137,000 Unicode characters, enabling the use of all the supported scripts. Currently, Noto includes all the characters in Unicode 6.1 or so but, since Unicode is currently at version 10.0, completion of Noto is likely to remain a continuously moving target. In addition, Esfahbod is involved in the cross-platform development of OpenType Font Variations, which are an open-source implementation of Adobe's proprietary multiple master font technology. Font Variations gives users the ability to change the weight of a font with a slider.

Font designers are already in the habit of designing fonts in multiple weights, ranging from Thin to Regular to Bold, or even Extra Bold, because changing the weight often requires redesigning the font slightly. The free-licensed Lato, for example, is available in no less than seven weights. Now, with Font Variations, users can select a font weight that automatically changes designs as necessary. Where multiple master fonts support only a handful of design changes with weight, a variations font can incorporate almost a hundred, allowing for more subtle and elaborate designs. The change to the toolchain for both font designers and users is creating a lot of excitement at gatherings of typographers. At one recent conference, according to Esfahbod, over half the papers were about Font Variations.

Ever since the SIL Free Font License and the GPL Font Exception were released just over a decade ago, providing licenses acceptable to font designers, high-quality open-source fonts have been released by the hundreds, and made available for both print and online use on sites like Google Fonts and The League of Movable Type. Some are near-replicas of popular proprietary fonts. Others are revivals of forgotten fonts. Still others are original designs. Together, these free fonts mean that modern graphic designers can work entirely with free software to an extent that was never previously possible.

Today HarfBuzz and its related technologies are promising users control over free-licensed OpenType fonts as never before. Moreover, it is a sign of how far free software has come that it is no longer playing catch up with its proprietary counterparts, but keeping pace with them, collaborating with them, and even sometimes leading development.