Let’s play with make (part 1)

Context

The idea is to produce pdf outputs of short problems (1-3 pages usually). I use those in class with my students. Note the plural: for one problem, we might have several different outputs (like a standard version, and a version with answers). In addition, the same output might be published for different classes or courses: this only changes the title headers and is called a “flavor” [1].

The source directory is versionned and organized as follows:

• The sources for the different types of outputs are named after the problem: the file name is thus the problem name, followed by a dash and the output type (such as ‘text’, ‘answers’ or ‘exam’), followed by a dot and the usual ‘tex’ suffix. Thus is the problem if called ‘ABC’, we might have abc-text.tex, abc-answers.tex and so on.
• As the different outputs have some parts in common (e.g. the main text of the problem or some parts of the latex preamble), these are maintained as external files which are included in the version upon compilation. These dependencies filenames do not include the problem name, so we can easily distinguish between a dependency and a source which should produce output. The dependencies are also ‘tex’ files, and have simple names such as src.tex or questions.tex.
• One dependency is compulsory, it contains the problem metadata, namely author, title and version. Its filename is meta.tex.
• The flavored header is included upon compilation, from the user ~/texmf/tex/latex/helpers/ directory: these files are external to the directory and will not be managed here: if they change, this will not trigger a new compilation of the outputs. In the source output file, the file name of the included header contains the variable FLAVOR which should be replaced by the correct flavor at compilation time.

Let’s take a fictitious ‘ABC’ problem as an example. The contents of the source directory are:

$ ls -l
total 20
-rw-r--r-- 1 jcb jcb 7325 Jan 15 11:14 abc-answers.tex
-rw-r--r-- 1 jcb jcb 1288 Jan 15 11:11 abc-text.tex
-rw-r--r-- 1 jcb jcb  243 Jan 15 11:09 meta.tex
-rw-r--r-- 1 jcb jcb 3571 Jan 11 11:20 src.tex

We thus have two possible outputs, text and answers, and two dependencies: the required meta.tex and another one called src.tex.

The meta.tex file contains minimal metadata as LaTeX commands:

\providecommand{\versiondate}{v2018.01.1}
% Changelog
% v2018.01.1    Initial version

\author{Jean-Charles~Bagneris}
\title{ABC_Problem}
\date{}

The abc-*.tex files include meta.tex, src.tex, and a header file depending on the chosen flavor:

$ egrep '{meta|src|FLAVOR}' abc-*.tex
abc-answers.tex:\input{meta}
abc-answers.tex:\input{helpers/title-FLAVOR}
abc-answers.tex:\input{src}
abc-text.tex:\input{meta}
abc-text.tex:\input{helpers/title-FLAVOR}
abc-text.tex:\input{src}

The produced outputs should go in a special destination subdirectory, and if any intermediary files are produced, they should go in a special build subdirectory as well. All this of course to avoid cluttering the main versionned directory with the results of compilation.

The compilation from the source files to the pdf outputs will be performed with the rubber utility.

Makefile: a first naive version

If we had to compile a pdf output of the abc-text.tex file using the ‘std’ (standard) flavor, we would have to:

• create the destination directory
• create a build directory
• copy meta.tex and src.tex in the build directory
• replace FLAVOR in the abc-text.tex source file with std, using sed, for example, and put the resulting source in the build directory
• invoke rubber on the final source located in the build directory

Let’s build a first and simple version of a Makefile to accomplish all this for us. The main parts are shown below:

# -------------------------------------
# vars
# -------------------------------------
DEST = _dest
BUILD = _build
FLAVOR = std

# -------------------------------------
# commands and options
# -------------------------------------
RUBBER = rubber --pdf --into $(DEST)

# -------------------------------------
# rules
# -------------------------------------
.SECONDARY: # Do not delete intermediary targets

.PHONY: all text cleandest

all: text ## Make pdf of the latest text version

cleandest: ## Delete intermediary latex files from $(DEST) directory
    -rm -f $(DEST)/*log $(DEST)/*aux $(DEST)/*toc $(DEST)/*snm $(DEST)/*nav $(DEST)/*out $(DEST)/*bbl $(DEST)/*blg

text: $(DEST)/abc-text.pdf

$(DEST)/%.pdf: $(BUILD)/%.tex $(BUILD)/meta.tex $(BUILD)/src.tex | $(DEST)
    $(RUBBER) $<

$(BUILD)/abc-%.tex: abc-%.tex | $(BUILD)
    -cp $< $@
    sed -i 's;FLAVOR;$(FLAVOR);' $@

$(BUILD)/%.tex: %.tex | $(BUILD)
    -cp $*.tex $(BUILD)/

$(DEST) $(BUILD):
    mkdir -p $@

(There is a refresher about the various automatic variables such as $@ in the documentation.)

So we start by creating useful variables for the destination and build directories. The default FLAVOR value will be std, any other value should thus be passed on the command line as an environment variable. The rubber command with its parameters is also conveniently stored in its own variable.

Then come the targets and rules. Here .SECONDARY ensures that we keep intermediary targets (the files in the build directory), and .PHONY lists the three targets which are not files: the usual all, then text to make the text output and finally cleandest to clean the destination directory.

The default target (all) depends on text: build the ‘text’ output of the problem, named after it (abc-text.pdf).

The actual building rules for the file targets take advantage of pattern rules, which are then used implicitely by make to produce targets which are not explicitely defined. Here we use three pattern rules:

• The first one produces any pdf output using rubber. You might notice that it depends on the existence of a source file with the same name in the build directory, as well as on the meta.tex and src.tex in the build directory too. The destination directory should exist and is thus an order-only prerequisite.
• The second pattern rule ensures that an abc-*.tex source file (here abc-text.tex) is copied in the build directory, and that sed is then invoked on it to replace the variable FLAVOR with its value. Thus, this rule produces the final source.
• Finally, the last pattern rule simply describes how to copy any ‘tex’ source, such as meta.tex and src.tex, in the build directory, where it will be used by the main source file.

Note that, because more specific rules are preferred over more generic ones, make will know that to create the final source file _build/abc-text.tex, it should use the second pattern rule, not the third one.

Lastly comes a simple rule to create the destination and build directories if needed.

This makefile works as expected:

$ make
mkdir -p _build
cp abc-text.tex _build/abc-text.tex
sed -i 's;FLAVOR;std;' _build/abc-text.tex
cp meta.tex _build/
cp src.tex _build/
mkdir -p _dest
rubber --pdf --into _dest _build/abc-text.tex
compiling _build/abc-text.tex...
compiling _build/abc-text.tex...
$ ls -l
total 36
-rw-r--r-- 1 jcb jcb 7325 Jan 15 11:14 abc-answers.tex
-rw-r--r-- 1 jcb jcb 1288 Jan 15 11:11 abc-text.tex
drwxr-xr-x 2 jcb jcb 4096 Jan 16 14:55 _build
drwxr-xr-x 2 jcb jcb 4096 Jan 16 14:55 _dest
-rw-r--r-- 1 jcb jcb 1565 Jan 16 14:54 Makefile
-rw-r--r-- 1 jcb jcb  243 Jan 15 11:09 meta.tex
-rw-r--r-- 1 jcb jcb 3571 Jan 11 11:20 src.tex
$ ls -l _dest
total 84
-rw-r--r-- 1 jcb jcb     8 Jan 16 14:55 abc-text.aux
-rw-r--r-- 1 jcb jcb 34589 Jan 16 14:55 abc-text.log
-rw-r--r-- 1 jcb jcb 43541 Jan 16 14:55 abc-text.pdf

But it is not very satisfying (yet):

1. The ‘text’ target depends on an explicit filename (abc-text.tex), including the problem name: we cannot re-use this makefile for another problem without manually changing this line.
2. The same happens with the second pattern rule, in which we find again the prefix abc related to the name of this problem.
3. The meta.tex and src.tex dependencies are explicitely listed in the first pattern rule: what if we had another one? What if src.tex did not exist? How can we manage this, while ensuring that meta.tex does exist, as this one is compulsory?
4. It would be convenient for the all target to produce all possible outputs for a given flavor.
5. We had to explicitely define a target to make the ‘text’ output, which means that the ‘answers’ target is missing here. And what if there were an ‘exam’ output? Or if the ‘text’ output did not exist? How could we automatically define all “individual” targets?

Second version: functions on strings and filenames

print-%:
    @echo '$*=$($*)'

NAME = $(subst -text,,$(basename $(wildcard *-text.tex)))

SOURCE = $(wildcard *-text.tex)
BASENAME = $(basename $(SOURCE))
NAME = $(subst -text,,$(BASENAME))

$ make print-SOURCE
SOURCE=abc-text.tex
$ make print-BASENAME
BASENAME=abc-text
$ make print-NAME
NAME=abc

SOURCE = $(firstword $(wildcard *-*.tex))

text: $(NAME)-text.pdf

$(BUILD)/$(NAME)-%.tex: $(NAME)-%.tex | $(BUILD)
    -cp $< $@
    sed -i 's;FLAVOR;$(FLAVOR);' $@

SOURCES = $(wildcard *-*.tex)
NAME = $(subst -text,,$(basename $(firstword $(SOURCES))))
DEPS = $(filter-out $(SOURCES),$(wildcard *.tex))
META = meta.tex

BUILDDEPS = $(foreach dep,$(DEPS),$(BUILD)/$(dep))

$ make print-DEPS
DEPS=src.tex meta.tex
$ make print-BUILDDEPS
BUILDDEPS=_build/src.tex _build/meta.tex

$(DEST)/%.pdf: $(BUILD)/%.tex $(BUILDDEPS) $(META) | $(DEST)
    $(RUBBER) $<

PDFS = $(patsubst $(NAME)-%.tex,$(DEST)/$(NAME)-%.pdf,$(SOURCES))

$ make print-PDFS
PDFS=_dest/abc-text.pdf _dest/abc-answers.pdf

all: $(PDFS) ## Make all possible pdf outputs

$ rm -rf {_build,_dest}
$ make
mkdir -p _build
cp abc-text.tex _build/abc-text.tex
sed -i 's;FLAVOR;std;' _build/abc-text.tex
cp src.tex _build/
cp meta.tex _build/
mkdir -p _dest
rubber --pdf --into _dest _build/abc-text.tex
compiling _build/abc-text.tex...
compiling _build/abc-text.tex...
cp abc-answers.tex _build/abc-answers.tex
sed -i 's;FLAVOR;std;' _build/abc-answers.tex
rubber --pdf --into _dest _build/abc-answers.tex
compiling _build/abc-answers.tex...
compiling _build/abc-answers.tex...

Final version: create the targets for me, please!

Let us think now about problem 5: we would like “individual” targets to be still available, allowing for example to make answers if a NAME-answers.tex file existed, and throw an error if it did not.

We already have a PDFS variable, and its definition can easily be changed to produce a list of targets instead:

TARGETS = $(patsubst $(NAME)-%.tex,%,$(SOURCES))

Then we get some targets, not pdf file names:

$ make print-TARGETS
TARGETS=text answers

That was easy, but now we have to define the rules to make those targets. What if we wrote something like:

$(TARGETS): $(PDFS)

It would not work as expected as once the variables are substituted for their value, what we get is:

text answers: _dest/abc-text.pdf _dest/abc-answers.pdf

That is, each individual target depends on all the outputs. In addition, it seems a bit stupid to have to define two variables so close in their contents as TARGET and PDFS.

Part of the problem here comes from the fact that TARGETS is a list of phony targets, not file names. Remember, make primary intent is to build files from other files. The more you deviate from this, the more voodoo you need, and that’s probably not a good thing. There are many ways to do black magic with make, but if you want to understand your own makefiles a few months after you wrote those, the less magic, the better.

What we need here to define our multiple targets is, fortunately, not too deep magic, and is called Static pattern rules. As the documentation teaches us, these are rules “…which specify multiple targets and construct the prerequisite names for each target base on the target name”. Exactly what we want, if I may.

So, here is how to write the rule to make our TARGETS:

$(TARGETS): %: $(DEST)/$(NAME)-%.pdf

The key here is the additional %: in the middle, which defines what is called the target-pattern. This is what allows to define the prerequisite for each target, and not for all of them.

Note that with these new targets available, we no longer need to define and use PDFS and we substitute TARGETS for it in all prerequisites, saving one variable.

all: $(TARGETS) ## Make all possible pdf outputs

A final check to make sure that everything works as expected. Notice that make only makes what is required and no more on each invocation.

$ rm -rf {_build,_dest}
$ make text
mkdir -p _build
cp abc-text.tex _build/abc-text.tex
sed -i 's;FLAVOR;std;' _build/abc-text.tex
cp src.tex _build/
cp meta.tex _build/
mkdir -p _dest
rubber --pdf --into _dest _build/abc-text.tex
compiling _build/abc-text.tex...
compiling _build/abc-text.tex...
$ make text
make: Nothing to be done for 'text'.
$ make
cp abc-answers.tex _build/abc-answers.tex
sed -i 's;FLAVOR;std;' _build/abc-answers.tex
rubber --pdf --into _dest _build/abc-answers.tex
compiling _build/abc-answers.tex...
compiling _build/abc-answers.tex...
$ make answers
make: Nothing to be done for 'answers'.
$ make
make: Nothing to be done for 'all'.
$ make foobar
make: *** No rule to make target 'foobar'.  Stop.

Yay!

Conclusion: a lot more is possible

As usual, this post is far longer than I expected when I started it, and if you made it thus far, congrats! I hope it whetted your appetite. make is extremely powerful, and the limit is your imagination. Remember, though, that with great power comes great responsibility, and that make is intended to build files from other files.

What kind of additional targets could we imagine for our makefile? Well, the version I use is a bit different: the final output filename includes the output type of course, but also the metadata (title, author and version), as well as the flavor. I can then keep multiple outputs in the destination directory. I did not include this part above, as it uses a bit of awk voodoo, and it was a bit off-topic.

Have fun!

Update (2018-01-22): there is a sequel to this post, introducing the use of the include directive in makefiles to factor some common parts, and to manage chains of prerequisites.