Introduction

Please note that this draft is slightly outdated! Basic concepts and the overall structure still apply, but details of the implementation and the naming of certain elements have changed in the stable version of logreq. See the the section on suggested improvements for some changes and have a look at the example files which ship with logreq.

In addition to user-readable messages like "Please rerun LaTeX", the biblatex package also writes maschine-readable hints to an external file. Such messages may request an additional LaTeX or BibTeX run, and they also provide the information required to run BibTeX on the right files, using the right command line options. This page presents a new format for such requests which has been adopted in biblatex 0.9b. The new format is based on XML. To facilitate adoption of this format in other packages, a LaTeX frontend is available as an independent package called logreq. You can get the package from CTAN. The package will do two things:

• Enable LaTeX package authors to use LaTeX commands to issue requests.
• Collect all requests from all packages and write them to an external file (jobname.run.xml) at the end of the document. The jobname.run.xml file is a valid XML file with an embedded document type definition.

In this draft, the term 'script' refers to the application processing the requests. This may in fact be an external Perl or other script, such as latexmk, which automates the typesetting process be running LaTeX, BibTeX, makeindex, etc. the required number of times. It may also be an integrated LaTeX editing environment, such as AUCTeX or Kile. The term 'tool' refers to any external application to be executed by the 'script', such as BibTeX, Biber, makeindex, xindy, etc. The term 'package' refers to any LaTeX package which depends on one or more external 'tools' and issues the requests to be processed by the 'script'.

The Basics

Let's start off with a stripped-down example of two requests. The examples further down will increase in complexity as more elements are added.

<request package="biblatex" priority="1" active="1">
 <generic>latex</generic>
</request>

<request package="biblatex" priority="2" active="1">
 <generic>bibtex</generic>
 <cmdline>
  <binary>bibtex</binary>
  <option>-min-crossrefs=2</option>
  <infile>example</infile>
 </cmdline>
</request>

Let's go over this example one line at a time:

<request package="biblatex" priority="1" active="1">
<request package="biblatex" priority="2" active="1">

The package attribute specifies the package issuing the request. The priority attribute indicates the order in which the requests need to be processed. The script needs to run BibTeX first and LaTeX afterwards, hence the BibTeX request gets a higher priority. The active attribute indicates whether or not the request is active. This will be discussed in more detail below.

<generic>latex</generic>
<generic>bibtex</generic>

The generic element gives the generic name of the tool, e.g., latex rather than pdflatex or xelatex, or bibtex rather than bibtex8. When requesting another LaTeX run, there is usually no point in requesting a specific binary. The script is expected to rerun LaTeX, using whatever binary and setup it has been configured to use. Therefore, the first request has a solitary generic element only. The request:

<request package="biblatex" priority="1" active="1">
 <generic>latex</generic>
</request>

is a formal way of saying "Please rerun LaTeX". This is the most simple type of request possible.

The corresponding LaTeX code used in the package issuing the request is similar in structure:

\logrequest[package=biblatex,priority=1,active=1]{
  \generic{latex}
}

\logrequest[package=biblatex,priority=2,active=1]{
  \generic{bibtex}
  \cmdline{
    \binary{bibtex}
    \option{-min-crossrefs=2}
    \infile{example}
  }
}

There is also a shorthand for simple "rerun LaTeX" requests:

\latexrequest{packagename}

Command Line

The cmdline block has all the information required to construct the command line. Its components are given as separate elements.

<cmdline>
 <binary>bibtex</binary>
 <option>-min-crossrefs=2</option>
 <infile>example</infile>
</cmdline>

There are obviously potential security issues when running an arbitrary command. There is no way to provide robust and effective security checks on the LaTeX end of the workflow as they would be trivial to undermine. Any check performed by a TeX macro may easily be redefined by any file read in by TeX. Security checks can only be implemented in the script parsing the requests. In this case, it is useful to get the individual components of the command line as separate elements. For example, the script may maintain a list of trusted binaries and check if the binary is on this list. It could also check the command line options for variables and suspicious escape sequences.

Input and Output

There are two additional elements related to the command line, input and output:

<request package="biblatex" priority="2" active="1">
 <generic>bibtex</generic>
 <cmdline>
  <binary>bibtex</binary>
  <option>-min-crossrefs=2</option>
  <infile>example</infile>
 </cmdline>
 <input>
  <file>example.aux</file>
 </input>
 <output>
  <file>example.bbl</file>
 </output>
</request>

The corresponding LaTeX code:

\logrequest[package=biblatex,priority=2,active=1]{
  \generic{bibtex}
  \cmdline{
    \binary{bibtex}
    \option{-min-crossrefs=2}
    \infile{example}
  }
  \input{
    \file{example.aux}
  }
  \output{
    \file{example.bbl}
  }
}

The input and output blocks indicate the main files the application reads and writes. These blocks are optional since the information provided is supplemental. They are neither required to contruct the command line (the cmdline information must be complete), nor are they required to resolve dependencies. They may be useful for scripts which provide hooks for pre and postprocessing. If applicable, it is generally a good idea to provide this information even though it is somewhat redundant in most cases.

BibTeX is not really a case in point because there is no need to preprocess the aux files or postprocess the bbl files when using Biblatex. However, pre and postprocessing hooks may be useful when working with indices and makeindex.

Dependencies

Now let's look at a fully fledged example as written by biblatex. Note that the input and output blocks are omitted in most of the remaining examples for the sake of readability only. Biblatex always provides this information. Let's assume that LaTeX and BibTeX have been run before, and that all requests are currently inactive:

<request package="biblatex" priority="1" active="0">
 <generic>latex</generic>
 <provides type="dynamic">
  <file>example.aux</file>
  <file>example-blx.bib</file>
 </provides>
 <requires type="dynamic">
  <file>example.bbl</file>
 </requires>
</request>

<request package="biblatex" priority="2" active="0">
 <generic>bibtex</generic>
 <cmdline>
  <binary>bibtex</binary>
  <option>-min-crossrefs=2</option>
  <infile>example</infile>
 </cmdline>
 <provides type="dynamic">
  <file>example.bbl</file>
 </provides>
 <requires type="dynamic">
  <file>example.aux</file>
  <file>example-blx.bib</file>
 </requires>
 <requires type="editable">
  <file>bibdata1.bib</file>
  <file>bibdata2.bib</file>
 </requires>
 <requires type="static">
  <file>biblatex.bst</file>
 </requires>
</request>

The corresponding LaTeX code:

\logrequest[package=biblatex,priority=1,active=0]{
  \generic{latex}
  \provides[type=dynamic]{
    \file{example.aux}
    \file{example-blx.bib}
  }
  \requires[type=dynamic]{
    \file{example.bbl}
  }
}

\logrequest[package=biblatex,priority=2,active=0]{
  \generic{bibtex}
  \cmdline{
    \binary{bibtex}
    \option{-min-crossrefs=2}
    \infile{example}
  }
  \provides[type=dynamic]{
    \file{example.bbl}
  }
  \requires[type=dynamic]{
    \file{example.aux}
    \file{example-blx.bib}
  }
  \requires[type=editable]{
    \file{bibdata1.aux}
    \file{bibdata2.aux}
  }
  \requires[type=static]{
    \file{biblatex.bst}
  }
}

There are two additional types of elements in this example. The provides blocks list all files provided by the package or tool. The requires blocks list files the package or tool tries to read. This information may be used by the script to 'trigger' an inactive request, i.e., to treat an inactive request as an active one.

Let me elaborate. After a latex/bibtex/latex processing cycle, biblatex normally has all the data it needs. Unless new citations are added to the source file, no further BibTeX runs are required. That's why the requests are inactive (

active="0"

), but they still provide a list of required resources and specify all dependencies. If a new citation is added, biblatex knows that it needs more data from one of the bib files and issues an active request (

active="1"

). However, it has no way of detecting whether one of the bib has been modified since the last BibTeX run. If an entry in one of the bib files is modified, the bbl file will hold outdated data.

Monitoring external files for changes is one of the tasks that only the script can perform. To do that, it obviously needs to know which resources are used. It also needs some information about the volatility of the resource. After all, it doesn't make sense to use timestamping to detect changes if the monitored file is rewritten on every LaTeX pass. This information is provided by the required blocks. The provides blocks indicate the resources which are provided by the package or tool. There are three types of resources:

static
Static files are not modified during normal operation, neither by the package nor by the user. A typical example is biblatex.bst, which is not intended for modification by users. All system files are static resources in this sense. Monitoring of static files is generally optional. It is not required for normal operation but useful for developers. Timestamping is sufficient to monitor changes to static resources.

dynamic
Dynamic files are (re)generated automatically on a regurlar basis, typically on every run, even if their contents did not change since their last update. Typical examples are aux and bbl files. Monitoring of dynamic resources is not required by biblatex, but it may very well be required by other packages. When monitoring dynamic files, some sort of checksum, such as an MD5 hash, is required.

editable
Editable files are not generated automatically by a package or a tool. They may be and usually are modified by a user, but they do not necessarily change in between runs. Some of them may be edited on a rather infrequent basis. A typical example are bib files. Timestamping is sufficient to monitor changes to editable resources.

With biblatex, it is generally sufficient to monitor editable resources because the package uses an internal checksum to track changes. It can detect whether it needs another LaTeX and/or BibTeX run in a reliable way. However, it can not detect changes to any of the external bib files. For this reason, it also outputs inactive requests. The active attribute reflects biblatex's knowledge about the current state of the workflow. Even if the package has no open requests, it still provides the information required to verify this state. If a request is active, it is safe to assume that an additional processing cycle is indeed required. If it is inactive, additional checks are required. In other words, inactive requests are a formal way of stating: "I don't need anything at this time but if you detect a change in any file listed in this block, this is what I'd like you to do."

In the example above, the first request states that LaTeX provides example.aux and requires example.bbl, the second request has it the other way around. Even without any built-in knowledge of the workflow, the script can infer from this information and the priorities that a latex/bibtex/latex processing cycle will pick up all changes. The bib files, however, come from a source which is not part of the workflow (they are listed in a requires block but there is no matching entry in any of the provides blocks), and therefore they need to be monitored for changes. There is no need to analyze these dependencies in detail since the

type="editable"

attribute reflects this fact.

Let's consider such a change. If a bib file is modified, the corresponding inactive request should be 'triggered', i.e., the script is expected to treat it as an active request. This means that a BibTeX run is required (as indicated by the requires block of the second request), based on the command line information in the corresponding request. This will update the example.bbl file (as indicated by the provides block of the second request). Since this file is listed in the requires block of the first request, it is clear that another LaTeX run is required, too.

This may all seem overly complicated. Note, however, that the point of this spec is to describe any arbitrary workflow which involves LaTeX plus one or more external tools which may need to be run one or more times on one or more files.

When running biblatex with

backend=bibtex

or

backend=bibtex8

, the package needs multiple aux/bbl files if there are any

refsection

environments in the document. Let's consider an example.tex file with two

refsection

environments. This file would require the following workflow:

latex example
bibtex example
bibtex example1-blx
bibtex example2-blx
latex example

In this case, biblatex will output four requests, one for LaTeX and three for BibTeX:

<request package="biblatex" priority="1" active="1">
 <generic>latex</generic>
 <provides type="dynamic">
  <file>example.aux</file>
  <file>example1-blx.aux</file>
  <file>example2-blx.aux</file>
  <file>example-blx.bib</file>
 </provides>
 <requires type="dynamic">
  <file>example.bbl</file>
  <file>example1-blx.bbl</file>
  <file>example2-blx.bbl</file>
 </requires>
</request>

<request package="biblatex" priority="2" active="1">
 <generic>bibtex</generic>
 <cmdline>
  <binary>bibtex</binary>
  <option>-min-crossrefs=2</option>
  <infile>example</infile>
 </cmdline>
 <input>
  <file>example.aux</file>
 </input>
 <output>
  <file>example.bbl</file>
 </output>
 <provides type="dynamic">
  <file>example.bbl</file>
 </provides>
 <requires type="dynamic">
  <file>example.aux</file>
  <file>example-blx.bib</file>
 </requires>
 <requires type="editable">
  <file>bibdata1.bib</file>
  <file>bibdata2.bib</file>
 </requires>
 <requires type="static">
  <file>biblatex.bst</file>
 </requires>
</request>

<request package="biblatex" priority="2" active="1">
 <generic>bibtex</generic>
 <cmdline>
  <binary>bibtex</binary>
  <option>-min-crossrefs=2</option>
  <infile>example1-blx</infile>
 </cmdline>
 <input>
  <file>example1-blx.aux</file>
 </input>
 <output>
  <file>example1-blx.bbl</file>
 </output>
 <provides type="dynamic">
  <file>example1-blx.bbl</file>
 </provides>
 <requires type="dynamic">
  <file>example1-blx.aux</file>
  <file>example-blx.bib</file>
 </requires>
 <requires type="editable">
  <file>bibdata3.bib</file>
  <file>bibdata4.bib</file>
 </requires>
 <requires type="static">
  <file>biblatex.bst</file>
 </requires>
</request>

<request package="biblatex" priority="2" active="1">
 <generic>bibtex</generic>
 <cmdline>
  <binary>bibtex</binary>
  <option>-min-crossrefs=2</option>
  <infile>example2-blx</infile>
 </cmdline>
 <input>
  <file>example2-blx.aux</file>
 </input>
 <output>
  <file>example2-blx.bbl</file>
 </output>
 <provides type="dynamic">
  <file>example2-blx.bbl</file>
 </provides>
 <requires type="dynamic">
  <file>example2-blx.aux</file>
  <file>example-blx.bib</file>
 </requires>
 <requires type="editable">
  <file>bibdata5.bib</file>
  <file>bibdata6.bib</file>
 </requires>
 <requires type="static">
  <file>biblatex.bst</file>
 </requires>
</request>

Further Examples

As a last example, let's consider a different package which depends on an external tool, the

index

package. The

index

package is capable of handling multiple indices. We'll use a file with three indices as an example, a subject index, a name index, and a title index:

\documentclass{article}
\usepackage{index}
\newindex{default}{idx}{ind}{Subject Index}
\newindex{names}{ndx}{nnd}{Name Index}
\newindex{titles}{tdx}{tnd}{Title Index}
\begin{document}
...
\end{document}

This example.tex file is processed as follows:

latex example
makeindex -o example.ind example.idx (= makeindex example)
makeindex -o example.nnd example.ndx
makeindex -o example.tnd example.tdx
latex example

Now let's try to express this workflow with our spec:

<request package="index" priority="1" active="0">
 <generic>latex</generic>
 <provides type="dynamic">
  <file>example.idx</file>
  <file>example.ndx</file>
  <file>example.tdx</file>
 </provides>
 <requires type="dynamic">
  <file>example.ind</file>
  <file>example.nnd</file>
  <file>example.tnd</file>
 </requires>
</request>

<request package="index" priority="2" active="0">
 <generic>makeindex</generic>
 <cmdline>
  <binary>makeindex</binary>
  <option>-o example.ind</option>
  <infile>example.idx</infile>
 </cmdline>
 <provides type="dynamic">
  <file>example.ind</file>
 </provides>
 <requires type="dynamic">
  <file>example.idx</file>
 </requires>
</request>

<request package="index" priority="2" active="0">
 <generic>makeindex</generic>
 <cmdline>
  <binary>makeindex</binary>
  <option>-o example.nnd</option>
  <infile>example.ndx</infile>
 </cmdline>
 <provides type="dynamic">
  <file>example.nnd</file>
 </provides>
 <requires type="dynamic">
  <file>example.ndx</file>
 </requires>
</request>

<request package="index" priority="2" active="0">
 <generic>makeindex</generic>
 <cmdline>
  <binary>makeindex</binary>
  <option>-o example.tnd</option>
  <infile>example.tdx</infile>
 </cmdline>
 <provides type="dynamic">
  <file>example.tnd</file>
 </provides>
 <requires type="dynamic">
  <file>example.tdx</file>
 </requires>
</request>

LaTeX code:

\logrequest[package=index,priority=1,active=0]{%
  \generic{latex}
  \provides[type=dynamic]{
    \file{example.idx}
    \file{example.ndx}
    \file{example.tdx}
  }
  \requires[type=dynamic]{
    \file{example.ind}
    \file{example.nnd}
    \file{example.tnd}
  }
}

\logrequest[package=index,priority=2,active=0]{%
  \generic{makeindex}
  \cmdline{
    \binary{makeindex}
    \option{-o example.ind}
    \infile{example.idx}
  }
  \provides[type=dynamic]{
    \file{example.ind}
  }
  \requires[type=dynamic]{
    \file{example.idx}
  }
}

\logrequest[package=index,priority=2,active=0]{%
  \generic{makeindex}
  \cmdline{
    \binary{makeindex}
    \option{-o example.nnd}
    \infile{example.ndx}
  }
  \provides[type=dynamic]{
    \file{example.nnd}
  }
  \requires[type=dynamic]{
    \file{example.ndx}
  }
}

\logrequest[package=index,priority=2,active=0]{%
  \generic{makeindex}
  \cmdline{
    \binary{makeindex}
    \option{-o example.tnd}
    \infile{example.tdx}
  }
  \provides[type=dynamic]{
    \file{example.tnd}
  }
  \requires[type=dynamic]{
    \file{example.tdx}
  }
}

Since the

index

package does not use a checksum to track changes, it can detect the initial state of the workflow (the auxiliary files have not been generated at this point), but it can't detect any subsequent changes. This means that it would only issue an active request on the very first pass. All subsequent requests would be inactive. In this case, the script needs to monitor the dynamic resources for changes in order to determine whether or not a makeindex run is required.

Suggested Improvements

• Perhaps it might be a good idea to rename or change the numbering semantics of the "priority" attribute in the XML as "priority" has lots of colloquial terms which have "1" as the highest priority, rather than the lowest. One hears "my number one priority" quite a lot, which automatic association threw me when I read the spec until I read it more carefully... I'd reverse the numbering so priority "1" is the highest. [Philip Kime]
• The 'priority' attribute seems to be backward. I'd expect '1' to be the most important thing, with higher numbers being executed later ('First do this, second do this, ...'). [Joseph Wright]