Mandoc in OpenBSD

Training a foal to replace a venerable workhorse

Ingo Schwarze
<schwarze@openbsd.org>

BSDCan 2011
Ottawa, May 13

Csikó - Foal. - Photo: Adam Tomkó @flickr (CC)

Ingo Schwarze: Mandoc in OpenBSD - page 2: INTRO I - BSDCan 2011, May 13, Ottawa

What for?

Formatting man(1) pages.

→ See also the companion talk by Kristaps Dzonsons, "The roff tradition and mandoc".

$ mandoc /usr/src/share/man/man7/mdoc.7
$ man mdoc

Both commands produce:

$ man mdoc

MDOC(7)                    OpenBSD Reference Manual                    MDOC(7)

NAME
     mdoc - mdoc language reference

DESCRIPTION
     The mdoc language is used to format BSD UNIX manuals.  This reference
     document describes its syntax, structure, and usage.  The reference
     implementation is mandoc(1); the COMPATIBILITY section describes
     compatibility with other troff -mdoc implementations.

Make documentation easily accessible:

• Have it at as few places as possible, and as much as possible at one single place.
• In OpenBSD, almost all documentation is manuals, and the little that doesn't fit there is all in the FAQ.

Thus, manuals are very important in OpenBSD.

Ingo Schwarze: Mandoc in OpenBSD - page 3: INTRO II - BSDCan 2011, May 13, Ottawa

What from?

→ See also Kristaps' talk.

mdoc(7) or man(7) source files - plus roff(7), tbl(7), ...

$ less /usr/src/share/man/man7/mdoc.7
.Dd $Mdocdate: April 17 2011 $          \" rcs(1) style Mdocdate ID
.Dt MDOC 7                              \" prologue: meta information
.Os
.Sh NAME
.Nm mdoc                                \" one word document title
.Nd mdoc language reference             \" one line document description
.Sh DESCRIPTION                         \" section structure
The                                     \" free text
.Nm mdoc                                \" logical markup
language is used to format
.Bx
.Ux
manuals.
This reference document describes its syntax, structure, and
usage.
The reference implementation is
.Xr mandoc 1 ;                          \" cross references to other manuals
the
.Sx COMPATIBILITY                       \" cross references inside one manual
section describes compatibility with other troff \-mdoc implementations.
.Pp
An
.Nm
document follows simple rules: lines beginning with the control
character
.Sq \.                                  \" block enclosures
are parsed for macros.
Other lines are interpreted within the scope of
prior macros:
.Bd -literal -offset indent             \" display blocks
\&.Sh Macro lines change control state. \" escaping
Other lines are interpreted within the current state.
.Ed

Ingo Schwarze: Mandoc in OpenBSD - page 4: INTRO III - BSDCan 2011, May 13, Ottawa

Table of contents

• OpenBSD 4.6 - the foal's pasture
• OpenBSD 4.7 - serious training
• OpenBSD 4.8 - drawing the cart
• OpenBSD 4.9 - removing groff
• Recurring topics
• Conclusions
• Future directions

Ingo Schwarze: Mandoc in OpenBSD - page 5: Apr to Dec 2009 - BSDCan 2011, May 13, Ottawa

Grazing the foal's pasture

Half a year free of serious responsibilities.

• Imported mandoc into the OpenBSD tree.
• Collected initial experience using mandoc.
• Main occupation during this time: Fixing bugs.
• Slowly started setting up a regression suite.
• Established the contact to everybody involved.

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 6: TRAIN I - Jan and Feb 2010 - BSDCan 2011, May 13, Ottawa

See the world!

Let mandoc build the whole tree.

Systematically investigate all fatal issues.

• Ignore all non-fatal issues for now.
• Fix the easy problems in mandoc, right away.
• Work around hard ones by changing manuals in the tree.

Most workarounds got fixed later:

• block opened, but not closed, and vice versa
• bad nesting of (full) blocks
• free-form text in context requiring macros
• blank lines outside literal context
• low-level requests not (yet) implemented in mandoc
• wrong number of arguments
• ...

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 7: TRAIN II - Feb 2010 - BSDCan 2011, May 13, Ottawa

Block nesting

An example of nice mandoc design

→ See also Kristaps' talk.

Example manual snippet using nested blocks:

$ mandoc chgrp.1
SYNOPSIS
     chgrp [-fh] [-R [-H | -L | -P]] group file ...

The corresponding mdoc(7) source code:

$ less chgrp.1
[...]
.Sh SYNOPSIS
.Nm chgrp
.Op Fl fh
.Oo
.Fl R
.Op Fl H | L | P
.Oc
.Ar group
.Ar
[...]

Mandoc respresents this by this syntax tree:

$ mandoc -Ttree chgrp.1   # much simplified
    Sh (block) SYNOPSIS
        Nm (block) chgrp
            Op (block)
                Fl (elem) fh
            Oo (block)
                Fl (elem) R
                Op (block)
                    Fl (elem) H
                    (text) |
                    Fl (elem) L
                    (text) |
                    Fl (elem) P
            Ar (elem) group
            Ar (elem) file ...

Traditional roff/mdoc design: no block structure!

low level: roff requests provide

• physical formatting
• registers to store data
• macro expansion

high level: mdoc macros

• call physical formatting
• set registers to keep state
• have no syntax tree

Ingo Schwarze: Mandoc in OpenBSD - page 8: TRAIN III - Feb to Jun 2010 - BSDCan 2011, May 13, Ottawa

Blood, sweat, and tears

Badly nested blocks and the Xo macro

Explicit blocks, badly nested:

.Ao ao
.Bo bo
.No ac Ac
.No bc Bc

<ao [bo ac> bc]

Implicit block broken by explicit block:

.Aq aq Bo bo eol
.No bc Bc

<aq [bo eol> bc]

The most important case in practice: .It Xo

$ less find.1
[...]
.It Xo
.Ic -exec Ar utility
.Op argument ...
.No ;
.Xc

-exec utility [argument ...] ;

Our first thought: deprecate this abomination!

• use: .It Ic -exec Ar utility Oo argument ... Oc No ;
• but lots of manuals use ".It Xo" - historical argument limit
• no way to change all manuals, everywhere
• or even the habits of people writing manuals...
• reluctant implementation:
  • Ao block contains the *whole* Bo block
  • Bo block contains an Ao body-end element

All mdoc(7) manuals build at this point!

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 9: TRAIN IV - Mar 2010 - BSDCan 2011, May 13, Ottawa

Hey, aren't you cheating?

Explicit pod2man(1) preamble support.

Example code from the standard pod2man preamble:

$ less perl.1
[...]
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'

This uses low-level roff requests:

• .ds user defined strings
• .de macro definitions
• .if conditionals
• and some more...

No chance to get all that implemented quickly.

Quick and dirty solution:

• Hardcode the strings (always the same - it's a preamble).
• Implement the macros as if they were man(7) macros.
• Parse and ignore the rest.

Now all man(7) manuals build as well!

No fixes in man(7) manuals, just workarounds in mandoc.

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 10: TRAIN V - conclusions - BSDCan 2011, May 13, Ottawa

Don't be lazy.

What went wrong while getting the tree to build?

• We should have started several months earlier.
• At first, mandoc reported way too many fatal errors.
• Too few mandoc bugs were fixed in this phase.
• Too many workarounds committed to manuals instead.

Lessons learnt:

• From the start, avoid throwing fatal errors when possible.
• By all means, report errors, but try hard to keep going.
• Try real-world usage as early as possible and use the feedback.
• Move quickly into production, don't delay for polishing.
• Code will only really mature when used.

New Forest Foal. - Photo: Krista van der Voorden <Kris*V*@flickr> (CC)

Ingo Schwarze: Mandoc in OpenBSD - page 11: DRAW I - Apr 2010 - BSDCan 2011, May 13, Ottawa

What are you waiting for?

Switch over the tree as soon as possible.

• Ready for production just before the 4.7 release.
• Switch over the build right after the release, to have as much time as possible to fix fallout.
• Bug reports from real-world users come in now.
  • Give priority to these bug reports.
  • All serious fallout fixed very quickly - within a few days.
• Lots of time for polishing before the first release.

Related timeline:

Photo: tiny_package @flickr (CC)

Ingo Schwarze: Mandoc in OpenBSD - page 12: DRAW II - Apr and May 2010 - BSDCan 2011, May 13, Ottawa

If ifs & ands were docs & mans...

No way around some low-level roff requests.

• Original design:
  • Forget about the complete bottom layer.
  • Only implement high-level mdoc(7) & man(7) macros.
• Didn't work out.
  • Real-world manuals use some low-level requests.
  • Realizing that was a slow, painful process.
  • We reluctantly edged in some low-level roff support.
  • Complexity was kept to the bare minimum.
• Both steps have proven to be just right:
  • Starting from the high level gave a clean design.
  • Roff reluctance prevented getting off track.
  • Surprise: Rather little impact in the end.

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 13: DRAW III - May 2010 - BSDCan 2011, May 13, Ottawa

Desperation lead to success:

How an afterthought improved the design.

Original mandoc main program:

• main loop to read input files
• push line after line into the parser backends
• parsers look for high-level macros, e.g. mdoc(7)
• call the formatting frontend on the resulting syntax tree

Modified mandoc main program:

• main loop to read input files
• call the roff preprocessor on each line
• push line after line into the parser backends
• parsers look for high-level macros, e.g. mdoc(7)
• call the formatting frontend on the resulting syntax tree

Paradigmatic switch:

roff: expand high-level macros into low-level requests

• then pass the low-level requests into the formatters
• all structural info lost long before the main parser

mandoc: first handle low-level requests (preprocessor)

• then pass the high-level code to the parsers
• structural info kept even when intermixed with low-level roff

→ See Kristaps' talk for applications

Ingo Schwarze: Mandoc in OpenBSD - page 14: DRAW IV - May 2010 - BSDCan 2011, May 13, Ottawa

.if .ds .de

Examples of roff macros we need.

• ignore:
  • adjustment: .ad .ta
  • spacing: .ne .ns
  • hyphenation: .hy .nh
  • character formatting: .ps .tr
• error out:
  • input line traps: .it
• partial implementation:
  • conditionals: .if .ie .el
  • register definitions: .nr
• complete implementation:
  • string definitions: .ds
  • macro definitions: .de .rm
  • source file inclusion: .so

Very young zebra. - Photo: Tambako @flickr (CC)

Ingo Schwarze: Mandoc in OpenBSD - page 15: DRAW V - Jun and Jul 2010 - BSDCan 2011, May 13, Ottawa

Invented here.

Home-grown non-standard features even in OpenBSD.

Almost all related to SYNOPSIS formatting:

• Roff register to control SYNOPSIS mode.
• Nm indentation in SYNOPSIS mode.
• Word keeps in SYNOPSIS mode.

Quite some work to get this right, for features of no theoretical importance that are not even documented.

Similar stuff would probably come up in other systems, when switching them over.

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 16: DRAW VI - Summer 2010 - BSDCan 2011, May 13, Ottawa

Count your beans.

Automatic comparisons as part of the build system.

Iterative process, several cycles:

• Run comparisons on a small part of the tree.
• Identify parsing and formatting issues.
• Prioritize by frequency and severity.
• Fix sufficiently few that you don't lose too much time.
• Fix sufficiently many to significantly reduce the noise.
• Start over with a larger part of the tree.

In parallel, tweak groff to reduce noise in comparisons:

• no adjustment
• no hyphenation
• OpenBSD headers
• no special handling of Pa in FILES

A text file in CVS was the ideal bug tracker:

• Keep the number of known issues low.
• Don't let the complexity of the backlog outgrow the complexity of the code.

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 17: DIGRESSIONS - BSDCan 2011, May 13, Ottawa

All work and no play...

Nice features along the way

Critical points:

• Reliable parsing.
• Correct ASCII rendering.

However, fancy output is fun.

• We took the time now and again.
• Almost all projects went well: e.g. HTML, XHTML, PostScript, PDF
• One was overdone and a dead end: hyphenation

Related timeline:

Foals Just Wanna Have Fun. - Photo: Gary Tanner <gazzat@flickr> (CC)

Ingo Schwarze: Mandoc in OpenBSD - page 18: REMOVE I - early Oct 2010 - BSDCan 2011, May 13, Ottawa

Only a dozen pages...

... are using tbl, the last obstacle to groff removal.

• We long shunned the task:
  • Even though basic code was long available,
  • these were very few pages,
  • and it was unclear how to do it best;
  • we nearly waited too long for the release.
• Finally, i deliberately chose the minimal route:
  • hook the code directly into the high-level parsers
  • working integration within a single weekend
  • immediately rely on it for production
• Kristaps implemented a better way later:
  • parse tbl block macros on the roff level
  • call the parser for the content from the main program just like in the case of mdoc(7) and man(7)

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 19: REMOVE II - late Oct 2010 - BSDCan 2011, May 13, Ottawa

Not scared by one-way streets!

Groff emigrating to ports land.

• Groff and ports tree already prepared by espie@.
  • Use a groff build dependency and install formatted manuals.
• X11 manuals got briefly broken:
  • Support for file inclusion (.so) arrived one week late.
• Set up instructions for checking ports. LI>Start moving over ports to mandoc case by case.
  • Installing source manuals.
• Bug reports from porters start flowing in.
  • Again, give priority to fixing real-world issues.

Related timeline:

Cart. - Photo: garycycles7 @flickr (CC)

Ingo Schwarze: Mandoc in OpenBSD - page 20: CONCLUSIONS I - BSDCan 2011, May 13, Ottawa

Done!

Reached the stage: It just works.

To get there:

• Re-implemented a small family of languages.
• Turned the whole paradigm upside down.
• Remained compatible where it matters.
• Flouted compatibility that would just hinder.

Other systems can now do the same, if they want.

• ... without much risk.
• ... without having to fear major surprises.
• ... getting support from us in case of need.

Just mail us!

Ingo Schwarze: Mandoc in OpenBSD - page 21: RECURRING I - BSDCan 2011, May 13, Ottawa

Show what's relevant, not more.

Why it is critical to get errors and warnings right.

• An error message is missing or too generic:
  • Users have a hard time to fix their errors.
• A warning message is missing:
  • Users don't even notice their dangerous idioms.
• A few warnings too many, or too prominent:
  • Users get annoyed and switch off all warnings.
• More than one or two knobs:
  • Users don't remember and don't use them.
• Too few and too many can happen all at once!
  • (It did with mandoc, and it had too many knobs - at first.)

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 22: RECURRING II - BSDCan 2011, May 13, Ottawa

Bogue déjà vue:

Collecting regression tests.

• Slow start in 2009:
  • Many new bugs, few regressions.
  • Automatic tests not a priority.
  • However, start collecting anyway!
• The code matures...
  • More tests show up naturally.
  • They slowly become more important.
  • Periodic sorting and checking in.
• Current state:
  • When doing major merges,
  • the suite regularly catches issues.
  • The continuous effort starts to pay off.

Related timeline:

Horse Fly Portrait. - Photo: Jeff Burcher <VonShawn@flickr> (CC)

Ingo Schwarze: Mandoc in OpenBSD - page 23: RECURRING III - BSDCan 2011, May 13, Ottawa

Bad patches triggering good ones

Preliminary code put in and ripped out again.

That may seem inefficient, but actually it's a perfectly sane approach:

• The first implementation explores the feature.
• The second implementation gets it right.
• Just don't let the first one sprawl until it can't be ripped out any more.

Related timeline:

This approach got used in at least five cases:

Ingo Schwarze: Mandoc in OpenBSD - page 24: CONCLUSIONS II - BSDCan 2011, May 13, Ottawa

Newly designed.

Clean implementation of dirty languages.

• Started coding with the nice high-level stuff.
  • That gave us a very clean overall design.
• Edged in low-level ugliness later, where required.
  • Even while the tool was already in production!
• Only possible since we kept it small and simple.
  • Otherwise, such stunts would break the design.

Shun complexity!

This is a third reason why simplicity is key:
1. correctness and 2. security are well known.

But flexibility is important as well!

Ingo Schwarze: Mandoc in OpenBSD - page 25: CONCLUSIONS III - BSDCan 2011, May 13, Ottawa

Move fast!

How a replacement project can succeed.

• Quickly put your work to real-world use.
  • Do not let it rot in a corner of the OS.
• Keep moving fast, do not fear change.
  • But don't trap your users with incompatible changes.

And keep the balance:

• No need to do it when it makes no difference.
• You won't find users when you break behaviour.
• You won't find bugs when you don't have users.

Ingo Schwarze: Mandoc in OpenBSD - page 26: FUTURE I - Mar and Apr 2011 - BSDCan 2011, May 13, Ottawa

The future is already past.

When i proposed this talk, groff-1.20 was a plan.

We now provide a groff-1.21 port.

• Advantage of the move to ports even for groff lovers:
• It is much easier to update ports than base.
• And the GPL v3 doesn't hurt so much in the ports tree.
• NetBSD and FreeBSD are stuck with groff-1.19.2 which was released in 2005.

According to the ChangeLog, groff-1.21 has:

• Unicode, wide characters, Japanese localization.
• XHTML output (crappy, of course, but at least something).
• Many tbl(1) improvements, including MathML output.
• Better performance, bugfixes, features, and much more...

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 27: FUTURE II - BSDCan 2011, May 13, Ottawa

Not done...

Possible future directions.

• Infrastructure:
  • Install manual sources, not preformatted manuals.
  • Rewrite apropos(1) and makewhatis(8) to use mandoc.
  • Use that to replace man.cgi on the websites.
• Functionality:
  • Write pod2mdoc(1) in Perl.
  • Implement -mdoc -Tman:
    • millert@ wants to convert sudo(1) manuals to mdoc(7)
    • but some systems neither want mdoc(7) nor mandoc(1)
  • Implement -man -Tmdoc:
    • to help with migration to modern tools.
    • much more difficult to implement this direction.
• And of course:
  • bugfixing, missing functionality, documentation, ...

Related timeline:

Ingo Schwarze: Mandoc in OpenBSD - page 28 - BSDCan 2011, May 13, Ottawa

Thanks!

• Kristaps@ Dzonsons (bsd.lv)
  • for writing mandoc.
• Joerg@ Sonnenberger (NetBSD)
  • for important code contributions.
  • for hosting an excellent mandoc hackathon at BEC.de.
  • for lots of testing in the NetBSD tree.
• Jason McIntyre (jmc@)
  • for excellently and tirelessly maintaining our manuals.
  • for helping with countless bug reports.
  • for discussing countless questions regarding mdoc(7).
• Theo de Raadt (deraadt@)
  • for inviting Kristaps and getting mandoc imported. (Otherwise, i might have missed the project.)
  • for ongoing encouragement, in particular to make OpenBSD developers and users our guinea pigs. (None complained, they seemed to enjoy it.)
• Marc Espie (espie@)
  • for getting ports integration done.
  • for quite some help in various respects.
• Ulrich Spoerlein (uqs@ FreeBSD)
  • for lots of bug reports.
  • for a couple of patches and discussions.
  • for lots of testing in the FreeBSD tree.
• Matthieu@ Herrb, Todd@ Fries, Miod@ Vallat
  • for help with Xenocara integration.
• Peter Hessler (phessler@)
  • for help with the regression infrastructure.
  • for portability testing.
• Igor Sobrado (sobrado@)
  • for lots of bug reports and discussions.
• Christian Weisgerber (naddy@)
  • for very thorough testing.

For important bug reports and discussions:

• Stuart Henderson (sthen@)
• Landry Breuil (landry@)
• Todd Miller (millert@)
• Brad@ Smith (comstyle.com)
• Will Backman
• Thomas Klausner (wiz@ NetBSD)
• Sascha Wildner (DragonFly BSD)
• Werner Lemberg (GNU)

David Coppa - Edd Barrett - Giovanni Bechis - Gleydson Soares - Ian Darwin - Janne Johansson - Jasper Lievisse Adriaanse - Kurt Miller - Matthew Dempsky - Matthias Kilian - Nicholas Marriott - Paul Irofti - Philipp Guenther - Remi Pointel - Ted Unangst - Thordur Bjornsson (all OpenBSD)

Anthony J. Bentley - Chris Bennett - Frantisek Holop - Igor Zinovik - James Jerkins - Maxim Belooussov - Mikolaj Kucharski - Nicolas Joly - Pascal Stumpf - Ryan Flannery - Tim van der Molen - Tristan Le Guern - Yuri Pankov

... and probably some i have forgotten (sorry for that)

For images under CC licenses:

A Youngster on the Quantocks. - Photo: Mark Robinson <me'nthedogs'@flickr> (CC)