%!TEX root = manual.tex
%
\chapter{ML-Antlr}

%\section{Overview}

Parsers analyze the syntactic structure of an input string, and are usually specified with some variant of context-free grammars.  \antlr{} is a parser generator for Standard ML based on Terence Parr's variant of $LL(k)$ parsing.  The details of the parsing algorithm are given in the companion implementation notes; the practical restrictions on grammars are discussed in Section~\ref{sec:antlr-llk}.  A parser generated by \antlr{} is a functor; it requires a module with the \texttt{ANTLR\char`\_LEXER} signature:
\begin{lstlisting}
signature ANTLR_LEXER = sig
    type strm
    val getPos : strm -> AntlrStreamPos.pos
  end
\end{lstlisting}
Applying the parser functor will yield a module containing a \texttt{parse} function:
\begin{lstlisting}
val parse : 
      (Lex.strm -> ParserToks.token * AntlrStreamPos.span * Lex.strm)
        -> Lex.strm
          -> result_ty option * strm * ParserToks.token AntlrRepair.repair list
\end{lstlisting}
where \texttt{result\char`\_ty} is determined by the semantic actions for the parser.  The \texttt{ParserTokens} module is generated by \antlr{} (see Section~\ref{sec:antlr-gencode}) and the \texttt{AntlrRepair} module is available in the \texttt{ml-lpt} library (see Chapter~\ref{ch:ml-lpt-lib}). 

Notable features of \antlr{} include:
\begin{itemize}
 \item Extended BNF format, including Kleene-closure (*), positive closure (+), and optional (?) operators.
 \item Robust, automatic error repair.
 \item Selective backtracking.
 \item ``Inherited attributes'': information can flow downward as well as upward during a parse.
 \item Semantic predicates: a syntactic match can be qualified by a semantic condition.
 \item Grammar inheritence.
 \item Convenient default actions, especially for EBNF constructions.
 \item Convenient abbreviations for token names (\emph{e{.}g{.}}, \texttt{"("} rather than \texttt{LP})
\end{itemize}
The tool is invoked from the command-line as follows:
\begin{verbatim}
    ml-antlr [options] file
\end{verbatim}
where \texttt{file} is the name of the input \ulex{} specification, and where \texttt{options} may be any combination of:

\vskip 12pt
\begin{tabular}{lp{0.65\textwidth}}
  \texttt{--dot} & generate DOT output (for graphviz; see \texttt{http://www.graphviz.org}).
    The generated file will be named \texttt{file.dot}, where \texttt{file} is the input file. \\
  \\
  \texttt{--latex} & generate a simple \LaTeX version of the grammar, named \texttt{file.tex}. \\
  \\
  \texttt{--unit-actions} & ignore the action code in the grammar, and instead return \texttt{()}
    for every production. \\
  \\
  \texttt{--debug} & add code to the actions to print the left-hand-side of the production.
\end{tabular}%

\vskip 10pt \noindent
The output file will be called \texttt{file.sml}.

\section{Background definitions}

Before describing \antlr{}, we need some terminology.  A \emph{context-free grammar} (CFG) is a set of \emph{token} (or \emph{terminal}) symbols, a set of \emph{nonterminal} symbols, a set of \emph{productions}, and a start symbol $S$, which must be a nonterminal.  
The general term \emph{symbol} refers to both tokens and nonterminals.  A production relates a nonterminal $A$ to a string of symbols $\alpha$; we write this relation as $A \ra \alpha$.  Suppose $\alpha A \beta$ is a symbol string, and $A$ is a nonterminal symbol.  We write $\alpha A \beta \Ra \alpha \gamma \beta$ if $A \ra \gamma$ is a production; this is called a one-step derivation.  In general, a CFG generates a language, which is a set of token strings.  The strings included in this language are exactly those token string derived in one or more steps from the start symbol $S$.

A parser recognizes whether an input string is in the language generated by a given CFG, usually computing some value (such as a parse tree) while doing so.  The computations performed during a parse are called \emph{semantic actions} (or just \emph{actions}).

\section{Specification format}\label{sec:antlr-spec}

A \antlr{} specification is a list of semicolon-terminated \emph{declarations}.  Each declaration is either a \emph{directive} or a \emph{nonterminal definition}.  Directives are used to alter global specification properties (such as the name of the functor that will be generated) or to define supporting infrastructure for the grammar.  The nonterminal definitions specify the grammar itself.  The grammar for \antlr{} is given in Figure~\ref{fig:antlr-syntax}.

\begin{figure}
\Grammar{
\GFirstB{spec}
	{$($ declaration \T{;} $)^*$}

\GFirstB{declaration}
	{directive}
\GNextB
	{nonterminal}

\GFirstB{directive}
	{\kw{defs} code}
\GNextB
	{\kw{entry} ID $($ \T{,} ID $)^*$}
\GNextB
	{\kw{import} STRING $($ \kw{dropping} symbol$^+$ $)^?$}
\GNextB
	{\kw{keywords} symbol $($ \T{,} symbol $)^*$}
\GNextB
	{\kw{value} ID code}
\GNextB
	{\kw{name} ID}
\GNextB
	{\kw{header} code}
\GNextB
	{\kw{refcell} ID \T{:} monotype \T{=} code}
\GNextB
	{\kw{start} ID}
\GNextB
	{\kw{tokentype} qualid}
\GNextB
	{\kw{tokens} \T{:} tokdef $($ \T{|} tokdef $)^*$}
\GNextB
	{\kw{nonterms} \T{:} datacon $($ \T{|} datacon $)^*$}
	
\GFirstB{code}
	{ \T{(} $\dots$ \T{)} }

\GFirstB{tokdef}
	{datacon $($ \T{(} STRING \T{)} $)^?$}

\GFirstB{datacon}
	{ID}
\GNextB
	{ID \T{of} monotype}

\GFirstB{nonterminal}
	{ID formals$^?$ \T{:} prodlist}

\GFirstB{formals}
	{ \T{(} ID $($ \T{,} ID $)^*$ \T{)} }
	
\GFirstB{prodlist}
	{production $($ \T{|} production $)^*$}
	
\GFirstB{production}
	{\kw{try}$^?$ named-item$^*$ $($ \kw{where} code $)^?$ $($ \T{=>} code $)^?$ }
\GFirstB{named-item}
	{$($ ID \T{:} $)^?$ item}
\GFirstB{item}
	{prim-item \T{?}}
\GNextB
	{prim-item \T{+}}
\GNextB
	{prim-item \T{*}}
	
\GFirstB{prim-item}
	{symbol $($ \T{@} code $)^?$}
\GNextB
	{ \T{(} prodlist \T{)} }

\GFirstB{symbol}
	{ID}
\GNextB
	{STRING}

\GFirstB{monotype}{\textrm{An SML monomorphic type expression}}
	
\GFirstB{qualid}{\textrm{An SML qualified identifier}}

\GFirstB{ID}{\textrm{An SML identifier}}
\GFirstB{STRING}{\textrm{An SML string literal}}
}
\caption{The \antlr{} grammar}\label{fig:antlr-syntax}
\end{figure}

SML-style comments (\texttt{(* ... *)}) are treated as ignored whitespace anywhere they occur in the specification, \emph{except} in segments of code.  The \textit{code} symbol represents a segment of SML code, enclosed in parentheses.  Extra parentheses occuring within strings or comments in code need not be balanced.
A complete example specification appears in Chapter~\ref{ch:example}.

Most \antlr{} declarations are \emph{cumulative}: they may appear multiple times in a grammar specification, with each new declaration adding to the effect of the previous ones.  Thus, for instance, the specification fragment
\begin{lstlisting}[language=MLAntlr]
    %tokens : foo ;
    %tokens : bar of string ;
\end{lstlisting}%
is equivalent to the single directive
\begin{lstlisting}[language=MLAntlr]
    %tokens : foo | bar of string ;
\end{lstlisting}%
and similarly for nonterminal definitions and so on.  All declarations are cumulative except for the \kw{start} and \kw{name} directives.
The reason for treating specifications in this way is to give the \kw{import} directive very simple semantics, as described below.

\section{Directives}

\subsection{The \kw{defs} directive}

The \kw{defs} directive is used to include a segment of code in the generated parser:  
\begin{lstlisting}[language=SML]
%defs (
  fun helperFn x = (* ... *)
);
\end{lstlisting}%
All definitions given will be in scope for the semantic actions (see Section~\ref{sec:antlr-actions}).

\subsection{The \kw{entry} directive}

It is often useful to parse input based on some fragment of a grammar.  When a nonterminal is declared to be an \emph{entry point} for the grammar via \kw{entry}, \antlr{} will generate a separate \texttt{parse} function that expects the input to be a string derived from that nonterminal.  Given a grammar with a nonterminal \texttt{exp} and the declaration
\begin{lstlisting}[language=MLAntlr]
    %entry exp;
\end{lstlisting}%
the generated parser will include a function
\begin{lstlisting}
val parseexp : (Lex.strm -> ParserToks.token * AntlrStreamPos.span * Lex.strm)
      -> Lex.strm
        -> exp_ty option * strm * ParserToks.token AntlrRepair.repair list
\end{lstlisting}
where \texttt{exp\char`\_ty} is the type of the actions for the \texttt{exp} nonterminal.  Note that if \texttt{exp} has inherited attributes (Section~\ref{sec:inh-attr}) they will appear as a tuple argument, curried after the lexer argument:
\begin{lstlisting}
val parseexp : (Lex.strm -> ParserToks.token * AntlrStreamPos.span * Lex.strm)
      -> attributes
        -> Lex.strm
          -> exp_ty option * strm * ParserToks.token AntlrRepair.repair list
\end{lstlisting}
Finally, the \emph{start} symbol (Section~\ref{sec:start}) is always an entry point to the grammar, but the generated function is simply called \texttt{parse}.

\subsection{The \kw{import} directive}

The \kw{import} directive is used to include one grammar inside another.  The string given in the directive should hold the path to a grammar file, and $\backslash$ characters must be escaped.  By default, all declarations appearing in the specified file are included in the resulting grammar, except for \kw{start}, \kw{entry}, and \kw{name} declarations.  However, individual tokens or nonterminals can be dropped by listing them in the \kw{dropping} clause of an \kw{import} declaration.  Since nonterminal definitions are cumulative (Section~\ref{sec:antlr-nt}), the imported nonterminals can be extended with new productions simply by listing them.
The final grammar must, of course, ensure that all used tokens and nonterminals are defined.

\subsection{The \kw{keywords} and \kw{value} directives}

\Antlr{} uses an error-repair mechanism that is based on inserting or deleting tokens until
a correct parse is found.
The \kw{keywords} and \kw{default} directives allow one to improve this process.
Since changes to the input involving keywords can drastically alter the meaning of the
input, it is usually desirable to favor non-keyword repairs.
The \kw{keywords} directive is used to tell \antlr{} which tokens should be considered keywords.
The \kw{value} directive is used to define a default argument for non-nullary tokens, which
is used to construct a token value that can be inserted into the token stream
when attempting to find a repair.
For example
\begin{lstlisting}[language=MLAntlr]
    %value NUMBER(0);
\end{lstlisting}%
would allow the \texttt{NUMBER} token to be inserted as an error repair.

Section~\ref{sec:antlr-gencode} describes how to report error repairs to the user.
%When a syntax error is discovered, \antlr{} attempts to find a single-token repair to the input that will %allow the parse to continue.

\subsection{The \kw{name} directive}

The prefix to use for the name of the generated parser functor is specified using \kw{name}.
In addition to the functor, \antlr{} will generate a module to define the \texttt{token} datatype.
If the declaration
\begin{lstlisting}[language=MLAntlr]
    %name Example;
\end{lstlisting}%
appears in the specification, then the parser functor will be named
\texttt{ExampleParseFn} and the tokens module will be called \texttt{ExampleTokens}.

\subsection{The \kw{header} directive}
The \kw{header} directive allows one to specify the parser's functor.
This directive is useful when you want to add additional parameters to the functor,
but the declaration must include the \texttt{Lex} structure with signature
\texttt{ANTLR\char`\_LEXER}.
For example,
\begin{lstlisting}
%header (
  functor ExampleParseFn (
      structure Lex : ANTLR_LEXER
      structure Extras : EXTRA_SIG));
\end{lstlisting}%

\noindent{}\textbf{Note:} the \kw{header} directive was added in SML/NJ version 110.72.

\subsection{The \kw{refcell} directive}

Because semantic actions must be pure (for backtracking and error repair), they cannot make use of standard reference cells to communicate information.
Nonterminals may inherit attributes (Section~\ref{sec:inh-attr}), which allows information to flow downward, but in some cases flowing information this way can become extremely tedious.
For example, a data structure may only need to be updated at a single point in the grammar, but in order to properly thread this state through the grammar, an inherited attribute would have to be added and propagated through every nonterminal.

The \kw{refcell} directive is used to declare a backtracking-safe reference cell and make it available to all semantic actions.  Reference cells are declared by giving the name, type, and initial value for the cell.  Each cell is bound in the semantic actions as a standard SML \texttt{ref} value.  Thus, for example, we might have the following specification fragment:
\begin{lstlisting}[language=MLAntlr]
    %refcell symbols : StringSet.set = ( StringSet.empty );
    
    exp
      : INT
      | (exp)
      | ID => ( symbols := StringSet.add(!symbols, ID); ID )
      ;
\end{lstlisting}%
The result of this fragment is that all symbol uses are tracked, in any use of
the \texttt{exp} nonterminal, but without having to manually thread the data
structure state through the grammar.

The final contents of a reference cell is returned as an extra result from the parse
following the repair list.

\subsection{The \kw{start} directive}\label{sec:start}

A particular nonterminal must be designated as the start symbol for the grammar.  The start symbol can be specified using \kw{start}; otherwise, the first nonterminal defined is assumed to be the start symbol.

\subsection{The \kw{tokentype} directive}

As noted above, \mlantlr{} synthesizes a structure that contains a datatype representing the
tokens specified in the grammar.
In some situations, it may be useful to specify the token datatype elsewhere (\eg{}, if you
want to use the same lexer for two different parsers).
In such a case, one can use the \kw{tokentype} directive to give a qualified name for the token
dataype.
This identifier will be used in the tokens structure to specify the tokens type.
For example, if the grammar specification contains the directive
\begin{lstlisting}[language=MLAntlr]
    %tokentype MyTokens.t;
\end{lstlisting}%
then the code
\begin{lstlisting}[language=SML]
    datatype token = MyTokens.t
\end{lstlisting}%
in the tokens structure.
Even when the \kw{tokentype} directive is used, one must also declare the grammar's tokens
using the \kw{tokens} directive.
Furthermore, the user-supplied token datatype must include a nullary \texttt{EOF} constructor
in addition to exactly the constructors specified by the \kw{tokens} directives.\footnote{
  When the \kw{tokentype} directive is specified, \mlantlr{} will generate a signature for
  the tokens structure.
}

\noindent{}\textbf{Note:} the \kw{tokentype} directive was added in SML/NJ version 110.81.

\subsection{The \kw{tokens} directive}

The alphabet of the parser is defined using \kw{tokens}.  The syntax for this directive resembles a datatype declaration in SML, except that optional abbreviations for tokens may be defined.  For example:
\begin{lstlisting}[language=MLAntlr]
    %tokens
      : KW_let  ("let")  | KW_in   ("in")
      | ID of string     | NUM of Int.int
      | EQ      ("=")    | PLUS    ("+")
      | LP      ("(")    | RP      (")")
      ;
\end{lstlisting}%
Within nonterminal definitions, tokens may be referenced either by their name or abbreviation; the latter must always be double-quoted.

\subsection{The \kw{nonterms} directive}
\mlantlr{} normally identifies non-terminal symbols by their appearance on the left-hand side
of a production and relies on SML type inference to determine the type returned by parsing
a string derived from the nonterminal.
It is also possible to declare the type of a non-terminal using a \kw{nonterms} directive.

\section{Nonterminal definitions}\label{sec:antlr-nt}

The syntax of nontermal definitions is given in Figure~\ref{fig:antlr-syntax}.  As an illustration of the grammar, consider the following example, which defines a nonterminal with three productions, taking a formal parameter \texttt{env}:
\begin{lstlisting}[language=MLAntlr]
    atomicExp(env)
      : ID => ( valOf(AtomMap.find (env, Atom.atom ID)) )
      | NUM
      | "(" exp@(env) ")"
      ;
\end{lstlisting}%
Note that actions are only allowed at the end of a production, and that they are optional.

As with most directives, the non-terminal definitions are cumulative.
For example, the definition of \texttt{atomicExp} above could also be
written as three separate rules.
\begin{lstlisting}[language=MLAntlr]
    atomicExp(env) : ID => ( valOf(AtomMap.find (env, Atom.atom ID)) );
    atomicExp(env) : NUM;
    atomicExp(env) : "(" exp@(env) ")";
\end{lstlisting}%

\subsection{Extended BNF constructions}

In standard BNF syntax, the right side of a production is a simple string of symbols.  Extended BNF allows regular expression-like operators to be used: \texttt{*}, \texttt{+}, and \texttt{?} can follow a symbol, denoting 0 or more, 1 or more, or 0 or 1 occurrences respectively.  In addition, parentheses can be used within a production to enclose a \emph{subrule}, which may list several \texttt{|}-separated alternatives, each of which may have its own action.  In the following example, the nonterminal \texttt{item\char`\_list} matches a semicolon-terminated list of identifiers and integers:
\begin{lstlisting}[language=MLAntlr]
    item_list : (( ID | INT ) ";")* ;
\end{lstlisting}%
All of the extended BNF constructions have implications for the actions of a production; see Section~\ref{sec:antlr-actions} for details.

\subsection{Inherited attributes}\label{sec:inh-attr}

In most parsers, information can flow upward during the parse through actions, but not downard.  In attribute grammar terminology, the former refers to \emph{synthesized} attributes, while the latter refers to \emph{inherited attributes}.  Since \antlr{} is a predictive parser, it allows both kinds of attributes.  Inherited attributes are treated as parameters to nonterminals, which can be used in their actions or semantic predicates.  Formal parameters are introduced by enclosing them in parentheses after the name of a nonterminal and before its production list; the list of parameters will become a tuple.  In the following, the nonterminal \texttt{expr} takes a single parameter called \texttt{env}:
\begin{lstlisting}[language=MLAntlr]
    expr(env) : (* ... *) ;
\end{lstlisting}%
If a nonterminal has a formal parameter, any use of that nonterminal is required to apply it to an actual parameter.  Actual parameters are introduced in a production by giving the name of a nonterminal, followed by the \texttt{@} sign, followed by the code to compute the parameter.  For example:
\begin{lstlisting}[language=MLAntlr]
    assignment : ID ":=" expr@(Env.emptyEnv) ;
\end{lstlisting}%

\subsection{Selective backtracking}

Sometimes it is inconvenient or impossible to construct a nonterminal definition which can be unambiguously resolved with finite lookahead.
% (see Section~\ref{sec:antlr-llk} for examples).  
The \kw{try} keyword can be used to mark ambiguous \emph{productions} for selective backtracking.  For backtracking to take place, each involved production must be so marked.  Consider the following:
\begin{lstlisting}[language=MLAntlr]
    A : %try B* ";"
      | %try B* "(" C+ ")"
      ;
\end{lstlisting}%
As written, the two productions cannot be distinguished with finite lookahead, since they share an arbitrary long prefix of \texttt{B} nonterminal symbols.  Adding the \kw{try} markers tells \antlr{} to attempt to parse the first alternative, and if that fails to try the second.  Another way to resolve the ambiguity is the use of subrules, which do not incur a performance penalty:
\begin{lstlisting}[language=MLAntlr]
    A : B* ( ";"
           | "(" C+ ")"
           )
      ;
\end{lstlisting}%
This is essentially \emph{left-factoring}. See Section~\ref{sec:antlr-llk} for more guidance on working with the $LL(k)$ restriction.

\subsection{Semantic predicates}

A production can be qualified by a \emph{semantic predicate} by introducting a \kw{where} clause.  Even if the production is syntactically matched by the input, it will not be used unless its semantic predicate evaluates to \texttt{true}.  A \kw{where} clause can thus introduce context-sensitivity into a grammar.  The following example uses an inherited \texttt{env} attribute, containing a variable-value environment:
\begin{lstlisting}[language=MLAntlr]
    atomicExp(env)
      : ID %where ( AtomMap.inDomain(env, Atom.atom ID) )
           => ( valOf(AtomMap.find (env, Atom.atom ID)) )
      | NUM
      | "(" exp@(env) ")"
      ;
\end{lstlisting}%
In this example, if a variable is mentioned that has not been defined, the error is detected and reported during the parse as a syntax error.

Semantic predicates are most powerful when combined with selective backtracking.  The combination allows two syntactically identical phrases to be distinguished by contextual, semantic information.

\subsection{Actions}\label{sec:antlr-actions}

Actions for productions are just SML code enclosed in parentheses.  Because of potential backtracking and error repair, action code should be pure (except that they may update \antlr{} \texttt{refcell}s; see the \kw{refcell} directive).

In scope for an action are all the user definitions from the \kw{defs} directive.  In addition, the formal parameters of the production are in scope, as are the semantic yield of all symbols to the left of the action (the yield of a token is the data associated with that token's constructor).  In the following example, the first action has \texttt{env} and \texttt{exp} in scope, while the second action has \texttt{env} and \texttt{NUM} in scope:
\begin{lstlisting}[language=MLAntlr]
    atomicExp(env)
      : "(" exp@(env) ")" => ( exp )
      | NUM => ( NUM )
      ;
\end{lstlisting}%
Notice also that the actual parameter to \texttt{exp} in the first production is \texttt{env}, which is in scope at the point the parameter is given; \texttt{exp} itself would not be in scope at that point.

An important aspect of actions is naming: in the above example, \texttt{exp} and \texttt{NUM} were the default names given to the symbols in the production.  In general, the default name of a symbol is just the symbol's name.  If the same name appears multiple times in a production, a number is appended to the name of each yield, start from 1, going from left to right.  A subrule (any items enclosed in parentheses) is by default called \texttt{SR}.  Any default name may be overriden using the syntax \texttt{name=symbol}.  Overriding a default name does \emph{not} change the automatic number for other default names.  Consider:
\begin{lstlisting}[language=MLAntlr]
    foo : A bar=A A ("," A)* A*
        ;
\end{lstlisting}%
In this production, the names in scope from left to right are: \texttt{A1}, \texttt{bar}, \texttt{A3}, \texttt{SR}, \texttt{A4}.

The EBNF operators \texttt{*}, \texttt{+} and \texttt{?} have a special effect on the semantic yield of the symbols to which they are applied.  Both \texttt{*} and \texttt{+} yield a \emph{list} of the type of their symbol, while \texttt{?} yields an option.  For example, if \texttt{ID*} appeared in a production, its default name would be \texttt{ID}, and if the type of value of \texttt{ID} was \texttt{string}, it would yield a \texttt{string list}; likewise \texttt{ID?} would yield a \texttt{string option}.  

Subrules can have embedded actions that determine their yield:
\begin{lstlisting}[language=MLAntlr]
    plusList : ((exp "+" exp => ( exp1 + exp2 )) ";" => ( SR ))* => ( SR )
\end{lstlisting}%
The \texttt{plusList} nonterminal matches a list of semicolon-terminated additions.  The innermost subrule, containing the addition, yields the value of the addition; that subrule is contained in a larger subrule terminated by a semicolon, which yield the value of the inner subrule.  Finally, the semicolon-terminated subrule is itself within a subrule, which is repeated zero or more times.  Note that the numbering scheme for names is restarted within each subrule.

Actions are \emph{optional}: if an action is not specified, the default behavior is to return all nonterminals and non-nullary tokens in scope.  Thus, the last example can be written as 
\begin{lstlisting}[language=MLAntlr]
    plusList : ((exp "+" exp => ( exp1 + exp2 )) ";")*
\end{lstlisting}%
since \texttt{"+"} and \texttt{";"} represent nullary token values.

\section{The $LL(k)$ restriction}\label{sec:antlr-llk}

When working with any parser, one must be aware of the restrictions its algorithm places on grammars.
When \antlr{} analyzes a grammar, it attempts to create a prediction-decision
tree for each nonterminal.
In the usual case, this decision is made using lookahead token sets.  The tool will start with $k = 1$
lookahead and increment up to a set maximum until it can
uniquely predict each production.  Subtrees of the decision tree
remember the tokens chosen by their parents, and take this into account
when computing lookahead.  For example, suppose we have two productions
at the top level that generate the following sentences:
\begin{verbatim}
    prod1 ==> AA
    prod1 ==> AB
    prod1 ==> BC
    prod2 ==> AC
    prod2 ==> C
\end{verbatim}
At $k = 1$, the productions can generate the following sets:
\begin{verbatim}
    prod1 {A, B}
    prod2 {A, C}
\end{verbatim}
and $k = 2$,
\begin{verbatim}
    prod1 {A, B, C}
    prod2 {C, <EOF>}
\end{verbatim}
Examining the lookahead sets alone, this grammar fragment looks ambiguous
even for $k = 2$.  However, \antlr{} will generate the following decision
tree:
\begin{lstlisting}[language=SML]
if LA(0) = A then
  if LA(1) = A or LA(1) = B then
    predict prod1
  else if LA(1) = C then
    predict prod2
else if LA(0) = B then
  predict prod1
else if LA(1) = C then
  predict prod2
\end{lstlisting}%

In \antlr{}, only a small amount of lookahead is used by default ($k = 3$).  Thus, the following grammar is ambiguous for \antlr{}:
\begin{lstlisting}[language=MLAntlr]
    foo : A A A B
        | A A A A
        ;
\end{lstlisting}%
and will generate the following error message:
\begin{verbatim}
    Error: lookahead computation failed for 'foo',
    with a conflict for the following productions:
      foo ::= A A A A EOF
      foo ::= A A A B EOF
    The conflicting token sets are:
      k = 1: {A}
      k = 2: {A}
      k = 3: {A}
\end{verbatim}
Whenever a lookahead ambiguity is detected, an error message of this form is given.  The listed productions are the point of conflict.
The \texttt{k = ...} sets together give examples that can cause the ambiguity, in this case an input of \texttt{AAA}.

The problem with this example is that the two \texttt{foo} productions can only be distinguished by a token at $k = 4$ depth.  This situation can usually be resolved using \emph{left-factoring}, which lifts the common prefix of multiple productions into a single production, and then distinguishes the old productions through a subrule:
\begin{lstlisting}[language=MLAntlr]
    foo : A A A (B | A)
        ;
\end{lstlisting}%
Recall that subrule alternatives can have their own actions:
\begin{lstlisting}[language=MLAntlr]
    foo : A A A ( B => ( "got a B" ) 
                | A => ( "got an A" )
                )
        ;
\end{lstlisting}%
making left-factoring a fairly versatile technique.

Another limitation of predictive parsing is \emph{left-recursion}, where a nonterminal recurs without any intermediate symbols:
\begin{lstlisting}[language=MLAntlr]
    foo : foo A A
        | B
        ;
\end{lstlisting}%
Left-recursion breaks predictive parsing, because it is impossible to make a prediction for a left-recursive production without already having a prediction in hand.  Usually, this is quite easily resolved using EBNF operators, since left-recursion is most often used for specifying lists.  Thus, the previous example can be rewritten as
\begin{lstlisting}[language=MLAntlr]
    foo : B (A A)*
        ;
\end{lstlisting}%
which is both more readable and more amenable to $LL(k)$ parsing.

\section{Position tracking}

\antlr{} includes built-in support for propagating position information.  Because the lexer module is required to provide a \texttt{getPos} function, the tokens themselves do not need to carry explicit position information.
A position \emph{span} is a pair to two lexer positions (the type \texttt{AntlrStreamPos.span} is
an abbreviation for \texttt{AntlrStreamPos.pos * AntlrStreamPos.pos}).
Within action code, the position span of any symbol (token, nonterminal, subrule) is available as a value; if the yield of the symbol is named \texttt{Sym}, its span is called \texttt{Sym\char`\_SPAN}.
Note that the span of a symbol after applying the \texttt{*} or \texttt{+} operators is the
span of the entire matched list:
\begin{lstlisting}[language=MLAntlr]
    foo : A* => (* A_SPAN starts at the first A and ends at the last *)
\end{lstlisting}%
In addition, the span of the entire current production is available as \texttt{FULL\char`\_SPAN}.

%\section{Handling precedence}

\section{Using the generated code}\label{sec:antlr-gencode}

When \antlr{} is run, it generates a tokens module and a parser functor.
If the parser is given the name \texttt{Xyz} via the \kw{name} directive, these structures will be called \texttt{XyzParseFn} and \texttt{XyzTokens} respectively.  
The tokens module will contain a single datatype, called \texttt{token}.  The data constructors for the \texttt{token} type have the same name and type as those given in the \kw{tokens} directive; in addition, a nullary constructor called \texttt{EOF} will be available.

The generated parser functor includes the following:
\begin{lstlisting}[language=SML]
val parse : (Lex.strm -> ParserToks.token * AntlrStreamPos.span * Lex.strm)
      -> Lex.strm
        -> result_ty option * strm * ParserToks.token AntlrRepair.repair list
\end{lstlisting}%
where \texttt{result\char`\_ty} is the type of the semantic action for the grammar's start symbol.  
The \texttt{parse} function is given a lexer function and a stream.
The result of a parse is the semantic yield of the parse, the value of the stream at the end of the parse, and a list of error repairs.  
If an unrepairable error occurred, \texttt{NONE} is returned for the yield of the parse.

Note that if the start symbol for the grammar includes an inherited attribute (or a tuple of attributes), it will appear as an additional, curried parameter to the parser following the lexer parameter.  Suppose, for example, that a grammar has a start symbol with an inherited \texttt{Int.int AtomMap.map}, and that the grammar yields \texttt{Int.int} values.  The type of its \texttt{parse} function is as follows:
\begin{lstlisting}
    val parse : 
      (strm -> ParserToks.token * strm) -> 
      Int.int AtomMap.map -> 
      strm -> 
      Int.int option * strm * ParserToks.token AntlrRepair.repair list
\end{lstlisting}

The \texttt{AntlrRepair} module is part of the \texttt{ml-lpt-lib} library; it is fully described in Chapter~\ref{ch:ml-lpt-lib}.
It includes a function \texttt{repairToString}:
\begin{lstlisting}
val repairToString : 
      ('token -> string) -> AntlrStreamPos.sourcemap
        -> 'token repair -> string

\end{lstlisting}
which can be used to produce error messages from the parser's repair actions.
There is a more refined version of this function that allows one to specialize
the string representation of tokens based on whether the repair adds or deletes
them.
\begin{lstlisting}
datatype add_or_delete = ADD | DEL

val repairToString : 
      (add_or_delete -> 'token -> string) -> AntlrStreamPos.sourcemap
        ->'token repair -> string
\end{lstlisting}


Likewise, the tokens module (\texttt{ParserTokens} in this example) includes a function:
\begin{lstlisting}
val toString : token -> string
\end{lstlisting}
Thus, although error reporting is customizable, a reasonable default is provided, as illustrated below:
\begin{lstlisting}
let
  val sm = AntlrStreamPos.mkSourcemap()
  val (result, strm', errs) = Parser.parse (Lexer.lex sm) strm
  val errStrings = 
    map (AntlrRepair.repairToString ParserTokens.toString sm)
	errs
in
  print (String.concatWith "\n" errStrings)
end
\end{lstlisting}
The \texttt{toString} function will convert each token to its symbol as given in a \kw{tokens} directive,
using abbreviations when they are available.
By substituting a different function for \texttt{toString}, this behavior can be altered.
