\documentclass[a4paper]{article}
\pdfinfoomitdate 1
\pdftrailerid{findent design}
\usepackage{color}
\usepackage{hyperref}
\hypersetup{
   colorlinks,
   citecolor=black,
   filecolor=black,
   linkcolor=black,
   urlcolor=black
}
\usepackage{etoolbox}
\makeatletter
\preto{\@verbatim}{\topsep=10pt \partopsep=10pt }
\makeatother

\usepackage{upquote} % straight quotes in verbatim

\renewcommand{\familydefault}{\sfdefault}   % font sans
\newcommand{\findent}{{\it{findent}}}
\newcommand{\Findent}{{\it{Findent}}}
\newcommand{\wfindent}{{\it{wfindent}}}
\newcommand{\wFindent}{{\it{wFindent}}}

\newcommand*{\thead}[1]{\multicolumn{1}{|c|}{\bfseries #1}}

\hyphenation{wfindent}
\hyphenation{Wfindent}
\hyphenation{findent}
\hyphenation{Findent}
\begin{document}
\frenchspacing
\title{Findent: design}
\author{Willem Vermin\thanks{Email: \href{mailto:contact@ratrabbit.nl}{contact@ratrabbit.nl}. 
Website: \href{https://ratrabbit.nl}{https://ratrabbit.nl}.}}
\date{\input{date.tex}}
\maketitle
\begin{abstract}
   \noindent{} \Findent{} is a computer program to indent Fortran sources. 
   \Findent{} can also be used to convert from fixed format to free format and
   {\it{vice versa}}.
   The program \findent{} became more complex than originally foreseen, that
   is why I wrote 
   this document covering the inner workings of \findent{}.
\end{abstract}

\tableofcontents

\section{\Findent{}: Design objectives}

\subsection{Reliability}
Assuming that \findent{} will be used in serious projects, involving large Fortran programs, 
the code of \findent{} should be as reliable as possible, therefore it is kept as simple as possible:
\begin{itemize}
   \item \Findent{} handles only two files: standard input and standard output.
      The input contains the Fortran program to be handled, the output contains 
      the modified program.
   \item The programming language is C++, a well maintained and documented language.
   \item No multithreading is used.
   \item Parsing the input is done with the aid of \verb|bison| and \verb|flex|: 
      well known and maintained tools.
   \item \Findent{} mostly works on a line-by-line basis. (Exceptions:
      labelled \verb|DO|-loops require a simple administration, and relabelling
      needs the complete source of a program unit.) 
   \item \Findent{} uses no configuration files: it is steered by command-line
      parameters and an environment variable containing command-line
      parameters.
   \item A comprehensive test-suite is part of the distribution. 
\end{itemize}

\subsection{Usability}
\Findent{} is easy to use, yet offers the possibility to tweak the indentation
to the user's taste:
\begin{itemize}
   \item All options have reasonable defaults, for example, 
      usage can be as simple as:
      \begin{verbatim}
      findent < program.f90 > newprogram.f90
      \end{verbatim}
      Furthermore, the \findent{} distribution comes with a wrapper script
      \wfindent{} that can be used like (to indent all .f90 files):
      \begin{verbatim}
      wfindent *.f90
      \end{verbatim}
      Normally, \findent{} detects if the input in fixed or free form.
   \item All types of indentation (\verb|DO|, \verb|SUBROUTINE|, ...) can be specified 
      on the command line, for example to use 5 spaces after \verb|DO|:
      \begin{verbatim}
      findent --indent-do=5 <program.f90 > newprogram.f90
      \end{verbatim}
   \item \Findent{} ignores white space outside strings and label fields.
   \item Fixed and free format Fortran are supported.
   \item Conversion from fixed to free form is implemented, as well as the
      other way around.
   \item All kinds of \verb|DO|-loops are recognized, even nested \verb|DO|-loops using
      the same label.
   \item \Findent{} has been tested on legacy Fortran sources, going back
      to Fortran IV. Hollerith's are parsed correctly.
   \item Unrecognized
      constructions are allowed and are written on the output as-is. 
      Incomplete Fortran sources are handled gracefully.
   \item \Findent{} can relabel the Fortran source. The man page contains a warning: 'use this
      only on correct programs'. If \findent{} detects a problem (missing
      label definition; incomplete program unit; ...), relabelling
      is abandoned.
   \item High speed: \findent{} indents about 100.000 lines per second.
\end{itemize}

\section{Building \findent{}}
\subsection{End users}
Building \findent{} is easy and is based on standard tools: 
\begin{itemize}
   \item The distribution tar ball is based on \verb|autoconf|, a mature program suite
      to distribute program sources.
   \item The distribution tar ball contains the output files of \verb|flex| and
      \verb|bison|, so the user doesn't need to install these programs. (If they
      are installed, they will be used).
   \item On Linux \findent{} is built by unpacking the tar-ball, and issue the commands:
      \begin{verbatim}
         cd findent-xx.yy.zz
         ./configure
         make
         make check  # to run the test-suite
         sudo make install
      \end{verbatim}
   \item For \verb|MacOS|, building \findent{} is the same as for Linux. 
   \item A \verb|Windows|
      version can be obtained by the following:
      \begin{verbatim}
         a=i686-w64-mingw32
         b=`gcc -dumpmachine`
         export CXX="$a-g++ -static"

         ./configure --build=$b --host=$a
         make clean
         make
      \end{verbatim}
      You need to have \verb|g++-mingw-w64-i686-win32| or something
      like that available. Probably, using \verb|WSL| or \verb|Cygwin| on \verb|Windows| should make 
      it possible to do the build on the Windows system.
   \item If building as presented above does not succeed, the script
      \verb|simplemake.sh|, containing usage instructions, can be used.
\end{itemize}
   \subsection{Program development and maintenance}
   The following is for developers and maintainers:
      \begin{itemize}
         \item The script \verb|bootstrap| 
            runs \verb|autoreconf|, replaces the copyright statements in
            nearly all sources and generates the output of the \verb|flex| and \verb|bison|.
            This output will be contained in the distribution tar ball, generated with
            \begin{verbatim}
   make distcheck
            \end{verbatim}
         \item An esoteric option is \verb|--with-esope|. This causes \findent{} to recognize
            \verb|esope| constructs, see \url{http://www-cast3m.cea.fr/html/esope/esope.html}.
         \item In the files \verb|src/debug.h| and \verb|src/debug.cpp| macros and functions
            are defined to be used when debugging. There is comment in those files how
            to use them. 
         \item \Findent{} comes with a comprehensive test suite, located in the directory
            \verb|test|. The tests will exercise every flag, and check if solved
            bugs are still solved. Testing is activated by running:
\begin{verbatim}
   make check
\end{verbatim}
      \end{itemize}


\section{Usage}
\subsection {Flags to influence the working of \findent{}}
Options to \findent{} can be given on the command line, like:
\begin{verbatim}
   findent -ifixed -ofree -i2 < prog.f > prog.f90
\end{verbatim}
and/or in the environment variable FINDENT\_FLAGS, like:
\begin{verbatim}
   export FINDENT_FLAGS='-i4 -I8'
   findent -ifixed -ofree < prog.f > prog.f90
\end{verbatim}
Most flags relate to the format of the input file and output file.
However, some options arrange that \findent{} does not
output an indented Fortran source, but other information.
These flags are marked in the man page with the string \verb|[NO_ENV]|
and are ignored when present in the environment variable \verb|FINDENT_FLAGS|.
Invalid flags, both on the command line and in the environment,
are silently ignored\footnote{Since \findent{} only writes 
to standard output, error messages would clutter the indented Fortran program.}.
Flags are read first from \verb|FINDENT_FLAGS| and secondly from the command
line. \newline
The flags are handled in the files \verb|src/flags.cpp| and \verb|src/flags.h|.
See the man page for a description of the flags.

\subsection{Generating documentation}
\Findent{} can generate the following documentation:
\begin{itemize}
   \item A text file ('help-file'), describing all flags.
   \item A man page, suitable for processing with the program \verb|man|.
   \item A text file, containing the ChangeLog.
   \item Text files, describing the usage in an editor.
   \item Text file describing how to use \findent{} in editor, for example \verb|vim|.
\end{itemize}

\subsection{Miscellaneous other functions}
\begin{itemize}
   \item Print version of \findent{}.
   \item Print 'free' or 'fixed', depending on what \findent{} deduces from
      the input.
   \item Print dependency information, based on:
      \begin{itemize}
         \item Usage and definitions of modules.
         \item Usage of \verb|include files|.
      \end{itemize}
   \item Print a shell script to be used in combination with the dependencies.
   \item Print the amount of indentation of the last line read.
   \item Print the line number of the last usable line as a start for indenting.
   \item Print a report of defined and used labels.
   \item Print scripts to incorporate \findent{} in the editors
      \verb|Vim|, \verb|Emacs| or \verb|Gedit|.
\end{itemize}


\section{Detailed overview of the internals of \findent{}}
\subsection{Starting the machinery}
The main program is in \verb|findent.cpp|. The flags are read (\verb|get_flags()|),
and if
some kind of documentation has to be produced (\verb|docs.print()|), 
the program prints the documentation and returns.
Otherwise, the class \verb|Findent| from \verb|findentclass.h| is instantiated as
\verb|findent| and \verb|findent.run()| from \verb|findentrun.cpp| is called.

\subsection{The main driver: fortran.run()}
\verb|fortran.run| executes the following tasks (trivia are omitted here):

\begin{itemize}
   \item If standard input is connected to a terminal, take appropriate actions.
   \item Read
      all of the input and store the lines in a buffer (\verb|input_buffer|).
   \item If the input format is not forced to be fixed or free, call
      \verb|determine_fix_or_free()| to determine the format.
   \item Instantiate \verb|class indent| to either \verb|Free| or \verb|Fixed| as
      appropriate. These are subclasses of class \verb|Fortran| in 
      \verb|fortran.h| to define the free or fixed alternatives of certain 
      functions. See \verb|free.cpp|, \verb|free.h|, \verb|fixed.cpp| and 
      \verb|fixed.h|.
   \item Take actions if the user wants to relabel the input.
   \item Enter the indenting loop (trivia omitted here):
      \begin{itemize}
         \item Call \verb|get_full_statement()| to create full Fortran line 
            \verb|full_statement|
            by collating the possible continuation lines to the first (see below).
         \item Call \verb|indent_and_output()| to determine the indentation
            and output the lines that define the full Fortran line.
      \end{itemize}
\end{itemize}

\subsection{Collecting a full Fortran statement}
Collecting a full Fortran statement from the first line and continuation lines
is done in \verb|src/fortran.cpp|: \verb|get_full_statement()|. This function
looks surprisingly complex at first sight.
This is because continuation lines can contain:
\begin{itemize}
   \item comment lines,
   \item blank lines,
   \item \verb|cpp| or \verb|coco| preprocessor statements,
   \item \verb|findentfix| lines (see the man page). 
\end{itemize}
Furthermore, attention must be paid if we are relabelling or not.
The full Fortran statement is stored in \verb|full_statement|.

\noindent{} In \verb|src/fortranline.cpp| and \verb|src/fortranline.h| functions are
defined to handle lines with Fortran code. Care has been taken that 
for fixed format, a tab at the start of a line is handled properly 
(see \verb|do_clean()|).

\noindent{} The call to \verb|build_statement()| has a different implementation for 
free and fixed format, see \verb|src/free.cpp| and \verb|src/fixed.cpp|,
respectively. This function performs the following tasks:

\begin{itemize}
   \item Add the input \verb|Fortranline| to \verb|c-lines| (a list of 
      \verb|Fortranline|'s).
   \item Add the line, stripped from all non-fortran stuff to 
      \verb|full_statement|.
   \item Signal if there are continuation lines to be expected.
      This is easy in the free-form case, but in the fixed-form case 
      a look-ahead is necessary. See \verb|wizard()|
      in \verb|fixed.cpp|.
\end{itemize}

\subsection{Preprocessing the full Fortran statement}
Once a \verb|full_statement| has been obtained, this line is preprocessed
to make it suitable for parsing using \verb|flex| and \verb|bison|.
This is done in \verb|Line_prep::set_line()|, in file \verb|src/line_prep.cpp|.
An example may clarify this.\newline

Below is:
\begin{itemize}
   \item \textbf{s} - The full statement.
   \item \textbf{sl} - Spaces removed, except in strings and Hollerith's, and after the
      statement label.
   \item \textbf{sv} - Strings, Hollerith's, operators and statement label replaced
      by a space. 
   \item \textbf{sc} - Strings etc. replaced by \verb|space number space|, the number 
      is the index in \verb|sv|. (\verb|sc| is used for parsing with \verb|bison|
      and \verb|flex|.)
   \item \textbf{wv} - A list, length = \verb|sv.size()|. Each entry consists of a 
      \verb|struct whats| (see \verb|line_prep.h|) which tells (\verb|type|) what this
      entry contains: \verb|invalid|, \verb|none|, \verb|string|, 
      \verb|statement label| or \verb|operator|. In case of \verb|string| there
      is \verb|stringtype| which discriminates between Hollerith (\verb|h|),
      single quoted string (\verb|'|) or double quoted string (\verb|"|).
      The value of the entry is contained in \verb|value|.
\end{itemize}

\hrule
\begin{verbatim}
s:  [123   call sub(5habcde  , 5,  'f oo', 'ab c' .concat. "def")]
sl: [123 callsub(5habcde,5,'f oo','ab c'.concat."def")]
sv: [ callsub( ,5, ,   )]
    [0123456789012345678]   ! these are index numbers in sv
sc: [ 0 callsub( 9 ,5, 13 , 15  16  17 )]
wv[0]:  statement label      [123]
wv[9]:  hollerith string     [abcde]
wv[13]: single quoted string [f oo]
wv[15]: single quoted string [ab c]
wv[16]: operator             [concat]
wv[17]: double quoted string [def]

The other entries have type=none.
\end{verbatim}
\hrule
\subsubsection{Parsing the preprocessed full statement}
Parsing the preprocessed full statement is done using \verb|bison|
and \verb|flex|. Things are arranged that one line at a time
is parsed, see \verb|lexer_set(Line_prep p, const int state)| in
\verb|lexer.l|. The string \verb|sc| (see above) is used for
parsing. Parsing is initiated in \verb|indent_and_output()|
in \verb|fortran.cpp| by a call to \verb|parseline()|.
This function, returning a \verb|propstruct| (see \verb|prop.h|) 
containing the results, 
parses the full statement in two passes:
\begin{itemize}
   \item The lexer is brought in a state that does not recognize
      Fortran keywords.\newline
      For example:\newline
      \verb|   subroutinesub(x)=10|\newline
      will return kind=\verb|ASSIGNMENT|.
   \item If parsing does is not successful (kind = \verb|UNCLASSIFIED|),
      full statement is parsed again, but now the lexer is in a state to recognize
      relevant Fortran keywords.\newline
      For example:
      \newline
      \verb|   subroutinesub(x)|\newline
      will succeed and returning kind=\verb|SUBROUTINE|.
\end{itemize}

\subsection{Keeping track of indents}
In \verb|indent_and_output()| (\verb|fortran.cpp|), a stack
is maintained containing the indents, along with the
current index. The actions are in principle quite simple: 
if after parsing a relevant keyword is found (\verb|SUBROUTINE|, \verb|DO|, ...)
the indent is changed as appropriate and put on the stack.
If a kind of \verb|END| (\verb|ENDIF|, \verb|END SUBROUTINE|, ...) is found, the indent 
is pulled from the stack.\newline\newline 
Some constructs deserve extra attention:
\begin{itemize}
   \item Labelled \verb|DO|-loops: if a labelled \verb|DO|-loop is encountered,
      the label involved is stored on a stack. When a corresponding
      statement label is encountered, appropriate action is taken,
      also in the case of nested \verb|DO|-loops sharing to the same 
      label.\footnote{Shared DO-termination is flagged as a 'deleted feature'
      by gfortran.}
   \item \verb|MODULE PROCEDURE| statements: at encountering a \verb|MODULE PROCEDURE|,
      indentation if the next full statement is classified as
      an executable statement.
   \item An ambiguity:
      \newline
      \verb|   MODULEPROCEDUREmyproc|
      \newline
      Should this be interpreted as:
      \newline
      \verb|   MODULE PROCEDUREmyproc|
      \newline
      or:
      \newline
      \verb|   MODULE PROCEDURE myproc|
      \newline
      \Findent{} assumes the last is correct.\footnote{This ambiguity arises
      from the fact that all spaces are removed in the preprocessing phase.
      In fixed format (where spaces do not count), this ambiguity
      is also present for the compiler.}
\end{itemize}


\subsection{Handling cpp and coco preprocessor statements}
It was a design goal that \findent{} should handle macro's more or less
intelligent.\newline 

\noindent{} For example: \newline\nopagebreak
\begin{tabular}{|l|l|l|}
   \hline
   \thead{Input}&\thead{Desired}&\thead{Not desired}\\ %thead-textbf
   \hline
   \begin{minipage}{7em}
      \begin{verbatim} 

#ifdef DIM2
do y=1,ny
#else
do y=1,1
#endif
do x=1,nx
call s(x,y)
enddo
enddo
       \end{verbatim}
   \end{minipage} & 
   \begin{minipage}{11em}
      \begin{verbatim}

#ifdef DIM2
   do y=1,ny
#else
   do y=1,1
#endif
      do x=1,nx
         call s(x,y)
      enddo
   enddo
         \end{verbatim}
   \end{minipage}& 
   \begin{minipage}{12em}
      \begin{verbatim}

#ifdef DIM2
   do y=1,ny
#else
      do y=1,1
#endif
         do x=1,nx
            call s(x,y)
         enddo
      enddo
     \end{verbatim}
   \end{minipage} \\
   \hline
\end{tabular}
\newline

\noindent{} The following preprocessor statements (defined in \verb|lexer.l|) are recognized: \newline
\begin{tabular}{|l|l|}
   \hline
   \thead{cpp}&\thead{coco}\\ % thead-textbf
   \hline
   \verb|# if|            & \verb|?? if|              \\
   \verb|# endif|         & \verb|?? endif|           \\
   \verb|# else|          & \verb|?? else|            \\
   \verb|# elif|          & \verb|?? elseif|          \\
   \verb|# include "..." |& \verb|?? include "..." |  \\
   \verb|# include <...> |& \verb|?? include <...> |  \\
   \hline
\end{tabular}
\newline\newline
Note1: the rest of the preprocessing line is ignored, so, for example, \verb|#if| has the
same effect as \verb|#ifdef|.

\noindent{} Note2: the \verb|include's| are only used when generating dependencies, and are ignored 
when indenting.
\newline

\noindent{} Most of the preprocessor-handling code is reached via \verb|handle_pre()| in 
\verb|Fortran.cpp| and \verb|Pre_analyzer()| in \verb|pre_analyzer.cpp|.\newline
The strategy is as follows:
\begin{itemize}
   \item A stack is maintained to store the relevant items (e.g.
      the indentation level and the stack of indentations) (see 
      \verb|push_all()|, \verb|top_all()| and \verb|pop_all()| in 
      \verb|fortran.h|.
   \item The relevant items are pushed on this stack after \verb|#if|.
   \item The relevant items are popped off the stack if appropriate after
      \verb|#endif|, \verb|#else| and \verb|#elif|.
   \item Handling the preprocessor statements is done recursively.
   \item After a construct like 
      \begin{verbatim}
      #if ...
      <fortran statements>
      #endif
      \end{verbatim}
      the indentation continues from the state before the \verb|#if|, but after a 
      construct like
      \begin{verbatim}
      #if ...
      <fortran statements>
      #else
      <fortran statements>
      #endif
      \end{verbatim}
      the indentation continues from the state after the \verb|#else|.
\end{itemize}
In this way, most of the times \findent{} will generate sensible indentation. 
If \findent{} makes an error, this can easily be fixed by inclusion of a
\verb|findentfix| statement, for example (admittedly somewhat constructed):\newline\newline
\begin{tabular}{|l|l|}
   \hline
   \thead{Original}&\thead{Corrected}\\ % thead-textbf
   \hline
   \begin{minipage}{10em}
      \begin{verbatim} 

    program p
#ifdef A
       do i=1,10
#else
       i = 1
#endif
       x = x+i
#ifdef A
    enddo
#endif
 end

       \end{verbatim}
   \end{minipage} & 
   \begin{minipage}{11em}
      \begin{verbatim}

    program p
#ifdef A
       do i=1,10
#else
       i = 1
#endif
       x = x+i
#ifdef A
!findentfix: do
       enddo
#endif
    end
         \end{verbatim}
   \end{minipage} \\
   \hline
\end{tabular}

\subsection{Relabelling}

Relabelling (renumbering of labels) is done in the following stages:

\begin{enumerate}
   \item Scan the input until a complete program unit (\verb|PROGRAM|, 
      \verb|SUBROUTINE|, \verb|FUNCTION|)
      is obtained, collecting the defined and used labels.
   \item Regenerate the program unit, now with the renumbered labels.
   \item Indent and output the renumbered program unit.
   \item Go to step 1.
\end{enumerate}

\noindent{} If some error is detected, (not defined label, label spanning continuation lines, ...)
relabelling is abandoned for the current and following program units, however,
indentation proceeds as normal. If relabelling fails, one can run \findent{}
with the flag \verb|--query-relabel|, to see the reason of failure.

\subsection{Generate miscellaneous text files}
Help files, man page, scripts for usage in editors {\it etc.} are 
generated in the file \verb|src/docs.cpp|. For generating the
man page and the help-file, the function \verb|manout()| is used so that
generating these files is based on the same input.\newline

\noindent{}The other files are include files, generated from the original text files. For 
example: \verb|vim_fortran.inc| is generated from \verb|vim/fortran.vim|, 
using the script \verb|src/tocpp|.
Details are available in the file \verb|src/Makefile.am|.

\section{Self replication}
\Findent{} has the capability to output a tar ball containing the complete 
source.\footnote{This can be disabled by giving the flag --disable-selfrep to configure.}
The method used is to create an include file for \verb'src/selfrep.cpp' based on
the output of \verb'make dist'. See \verb'src/Makefile.am' for details.
An issue is to maintain a reproducible build (see \url{https://reproducible-builds.org/}),
because the tar ball contains time stamps for the
containing files. This problem is solved by modifying the standard code to produce 
a tar ball. Normally, the file \verb'bootdate', created by \verb'bootstrap'
is used as time stamp. Most of the code is contained in \verb'configure.ac'.

\section{Copyright}
\Findent{} comes with the BSD-3 license:\newline
\hrule
\begin{center}
\begin{verbatim}
Copyright: 2015-2025 Willem Vermin

License: BSD-3-Clause
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions
 are met:
 1. Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.
 2. Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in the
    documentation and/or other materials provided with the distribution.
 3. Neither the name of the copyright holder nor the names of its
    contributors may be used to endorse or promote products derived
    from this software without specific prior written permission.
  
 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE HOLDERS OR
 CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\end{verbatim}
\end{center}
\hrule

\end{document}