Author: Paul McGuire
Date: September, 2004

# Using the pyparsing module

Author: | Paul McGuire
Address: |
    ptmcg@users.sourceforge.net
Revision: | 1.2.2
Date: | September, 2004
Copyright: | Copyright © 2003,2004 Paul McGuire.

abstract: | This document provides how-to instructions for the pyparsing library, an easy-to-use Python module for constructing and executing basic text parsers. The pyparsing module is useful for evaluating user-definable expressions, processing custom application language commands, or extracting data from formatted reports.

Contents

# 1 Steps to follow

To parse an incoming data string, the client code must follow these steps:

1. First define the tokens and patterns to be matched, and assign this to a program variable. Optional results names or parsing actions can also be defined at this time.
2. Call `parseString()` or `scanString()` on this variable, passing in the string to be parsed. During the matching process, whitespace between tokens is skipped by default (although this can be changed). When token matches occur, any defined parse action methods are called.
3. Process the parsed results, returned as a list of strings. Matching results may also be accessed as named attributes of the returned results, if names are defined in the definition of the token pattern, using `setResultsName()`.

## 1.1 Hello, World!

The following complete Python program will parse the greeting "Hello, World!", or any other greeting of the form "<salutation>, <addressee>!":

    from pyparsing import Word, alphas

    greet = Word( alphas ) + "," + Word( alphas ) + "!"
    greeting = greet.parseString( "Hello, World!" )
    print greeting

The parsed tokens are returned in the following form:

    ['Hello', ',', 'World', '!']

## 1.2 Usage notes

- The pyparsing module can be used to interpret simple command strings or algebraic expressions, or can be used to extract data from text reports with complicated format and structure ("screen or report scraping"). However, it is possible that your defined matching patterns may accept invalid inputs. Use pyparsing to extract data from strings assumed to be well-formatted.

- To keep up the readability of your code, use the `+`, `|`, and `^` operators to combine expressions. You can also combine string literals with ParseExpressions - they will be automatically converted to Literal objects. For example:

      integer  = Word( nums )            # simple unsigned integer
      variable = Word( alphas, max=1 )   # single letter variable, such as x, z, m, etc.
      arithOp  = Word( "+-*/", max=1 )   # arithmetic operators
      equation = variable + "=" + integer + arithOp + integer    # will match "x=2+2", etc.

  In the definition of `equation`, the string `"="` will get added as a `Literal("=")`, but in a more readable way.

- The pyparsing module's default behavior is to ignore whitespace. This is the case for 99% of all parsers ever written. This allows you to write simple, clean, grammars, such as the above `equation`, without having to clutter it up with extraneous `ws` markers. The `equation` grammar will successfully parse all of the following statements:

      x=2+2
      x = 2+2
      a = 10   *   4
      r= 1234/ 100000

  Of course, it is quite simple to extend this example to support more elaborate expressions, with nesting with parentheses, floating point numbers, scientific notation, and named constants (such as `e` or `pi`). See `fourFn.py`, included in the examples directory.

- `MatchFirst` expressions are matched left-to-right, and the first match found will skip all later expressions within, so be sure to define less-specific patterns after more-specific patterns. If you are not sure which expressions are most specific, use Or expressions (defined using the `^` operator) - they will always match the longest expression, although they are more compute-intensive.

- `Or` expressions will evaluate all of the specified subexpressions to determine which is the "best" match, that is, which matches the longest string in the input data. In case of a tie, the left-most expression in the `Or` list will win.

- If parsing the contents of an entire file, pass it to the parseFile method using:

      expr.parseFile( sourceFile )

- `ParseExceptions` will report the location where an expected token or expression failed to match. In the case of complex expressions, the reported location may not be exactly where you would expect.

- Use the `Group` class to enclose logical groups of tokens within a sublist. This will help organize your results into more hierarchical form (the default behavior is to return matching tokens as a flat list of matching input strings).

- Punctuation may be significant for matching, but is rarely of much interest in the parsed results. Use the `suppress()` method to keep these tokens from cluttering up your returned lists of tokens. For example, `delimitedList()` matches a succession of one or more expressions, separated by delimiters (commas by default), but only returns a list of the actual expressions - the delimiters are used for parsing, but are suppressed from the returned output.

- Parse actions can be used to convert values from strings to other data types (ints, floats, booleans, etc.). But be careful not to include converted data within a `Combine` object.

- Be careful when defining parse actions that modify global variables or data structures (as in `fourFn.py`), especially for low level tokens or expressions that may occur within an `And` expression; an early element of an `And` may match, but the overall expression may fail.

- Performance of pyparsing may be slow for complex grammars and/or large input strings. The psyco package can be used to improve the speed of the pyparsing module with no changes to grammar or program logic - observed improvments have been in the 20-50% range.

# 2 Classes

## 2.1 Classes in the pyparsing module

`ParserElement` - abstract base class for all pyparsing classes; methods for code to use are:

- `parseString( sourceString )` - only called once, on the overall matching pattern; returns a ParseResults object that makes the matched tokens available as a list, and optionally as a dictionary, or as an object with named attributes

- `parseFile( sourceFile )` - a convenience function, that accepts an input file object or filename. The file contents are passed as a string to `parseString()`.

- `scanString( sourceString )` - generator function, used to find and extract matching text in the given source string; for each matched text, returns a tuple of:

  - matched tokens (packaged as a ParseResults object)
  - start location of the matched text in the given source string
  - end location in the given source string

  `scanString` allows you to scan through the input source string for random matches, instead of exhaustively defining the grammar for the entire source text (as would be required with `parseString`).

- `transformString( sourceString )` - convenience wrapper function for `scanString`, to process the input source string, and replace matching text with the tokens returned from parse actions defined in the grammar (see setParseAction).

- `setName( name )` - associate a short descriptive name for this element, useful in displaying exceptions and trace information

- `setResultsName( string, listAllMatches=False )` - name to be given to tokens matching the element; returns a copy of the element so that a single basic element can be referenced multiple times and given different names within a complex grammar; if multiple tokens within a repetition group (such as `ZeroOrMore` or `delimitedList`) the default is to return only the last matching token - if listAllMatches is set to True, then a list of matching tokens is returned.

- `setParseAction( fn )` - function to call after successful matching of the element; the function is defined as `fn( s, loc, toks )`, where:

  - `s` is the original parse string
  - `loc` is the location in the string where matching started
  - `toks` is a list of the matched tokens

  `fn` can return a modified `toks` list, to perform conversion, or string modifications. For brevity, `fn` may also be a lambda - here is an example of using a parse action to convert matched integer tokens from strings to integers:

      intNumber = Word(nums).setParseAction( lambda s,l,t: [ int(t[0]) ] )

  If `fn` does not modify the `toks` list, it does not need to return anything at all.

- `leaveWhiteSpace()` - change default behavior of skipping whitespace before starting matching (mostly used internally to the pyparsing module, rarely used by client code)

- `suppress()` - convenience function to suppress the output of the given element, instead of wrapping it with a Suppress object.

- `ignore( expr )` - function to specify parse expression to be ignored while matching defined patterns; can be called repeatedly to specify multiple expressions; useful to specify patterns of comment syntax, for example

- `setDebug( dbgFlag=True )` - function to enable/disable tracing output when trying to match this element

- `validate()` - function to verify that the defined grammar does not contain infinitely recursive constructs

- `parseWithTabs` - function to override default behavior of converting tabs to spaces before parsing the input string; rarely used, except when specifying whitespace-significant grammars using the White class.

## 2.2 Basic ParserElement subclasses

- `Literal` - construct with a string to be matched exactly
- `CaselessLiteral` - construct with a string to be matched, but without case checking; results are always returned as the defining literal, NOT as they are found in the input string

- `Word` - one or more contiguous characters; construct with a string containing the set of allowed initial characters, and an optional second string of allowed body characters; if only one string given, it specifies that the same character set defined for the initial character is used for the word body; a Word may also be constructed with any of the following optional parameters:

  - min - indicating a minimum length of matching characters
  - max - indicating a maximum length of matching characters
  - exact - indicating an exact length of matching characters

  If exact is specified, it will override any values for min or max.

- `CharsNotIn` - similar to Word, but matches characters not in the given constructor string (accepts only one string for both initial and body characters); also supports min, max, and exact optional parameters.

- `SkipTo` - skips ahead in the input string, accepting any characters up to the specified pattern; may be constructed with the following optional parameters:

  - include - if set to true, also consumes the match expression (default is false)
  - ignore - allows the user to specify patterns to not be matched, to prevent false matches

- `White` - also similar to Word, but matches whitespace characters. Not usually needed, as whitespace is implicitly ignored by pyparsing. However, some grammars are whitespace-sensitive, such as those that use leading tabs or spaces to indicating grouping or hierarchy. (If matching on tab characters, be sure to call parseWithTabs on the top-level parse element.)
- `Empty` - a null expression, requiring no characters - will always match; useful for debugging and for specialized grammars
- `NoMatch` - opposite of Empty, will never match; useful for debugging and for specialized grammars

## 2.3 Expression subclasses

- `And` - construct with a list of ParserElements, all of which must match for And to match; can also be created using the '+' operator
- `Or` - construct with a list of ParserElements, any of which must match for Or to match; if more than one expression matches, the expression that makes the longest match will be used; can also be created using the '^' operator
- `MatchFirst` - construct with a list of ParserElements, any of which must match for MatchFirst to match; matching is done left-to-right, taking the first expression that matches; can also be created using the '|' operator
- `Optional` - construct with a ParserElement, but this element is not required to match; can be constructed with an optional `default` argument, containing a default string to be supplied if the given optional parse element is not found in the input string; parse action will only be called if a match is found, or if a default is specified
- `ZeroOrMore` - similar to Optional, but can be repeated
- `OneOrMore` - similar to ZeroOrMore, but at least one match must be present
- `FollowedBy` - a lookahead expression, requires matching of the given expressions, but does not advance the parsing position within the input string
- `NotAny` - a negative lookahead expression, prevents matching of named expressions, does not advance the parsing position within the input string; can also be created using the unary '~' operator

## 2.4 Positional subclasses

- `StringStart` - matches beginning of the text
- `StringEnd` - matches the end of the text
- `LineStart` - matches beginning of a line (lines delimited by `\n` characters)
- `LineEnd` - matches the end of a line

## 2.5 Converter subclasses

- `Upcase` - converts matched tokens to uppercase
- `Combine` - joins all matched tokens into a single string, using specified joinString (default `joinString=""`); expects all matching tokens to be adjacent, with no intervening whitespace (can be overridden by specifying `adjacent=False` in constructor)
- `Suppress` - clears matched tokens; useful to keep returned results from being cluttered with required but uninteresting tokens (such as list delimiters)

## 2.6 Special subclasses and Exception classes

- `Group` - causes the matched tokens to be enclosed in a list; useful in repeated elements like `ZeroOrMore` and `OneOrMore` to break up matched tokens into groups for each repeated pattern

- `Dict` - like `Group`, but also constructs a dictionary, using the [0]'th elements of all enclosed token lists as the keys, and each token list as the value

- `SkipTo` - catch-all matching expression that accepts all characters up until the given pattern is found to match; useful for specifying incomplete grammars

- `Forward` - placeholder token used to define recursive token patterns; when defining the actual expression later in the program, insert it into the `Forward` object using the `<<` operator (see `fourFn.py` for an example).

- `ParseException` - exception returned when a grammar parse fails; ParseExceptions have attributes loc, msg, line, lineno, and column

- `RecursiveGrammarException` - exception returned by `validate()` if the grammar contains a recursive infinite loop, such as:

      badGrammar = Forward()
      goodToken = Literal("A")
      badGrammar << Optional(goodToken) + badGrammar

- `ParseResults` - class used to contain and manage the lists of tokens created from parsing the input using the user-defined parse expression. ParseResults can be accessed in a number of ways:

  - as a list
    - total list of elements can be found using len()
    - individual elements can be found using [0], [1], [-1], etc.
    - elements can be deleted using del
  - as a dictionary
    - if `setResultsName()` is used to name elements within the overall parse expression, then these fields can be referenced as dictionary elements or as attributes
    - the Dict class generates dictionary entries using the data of the input text - in addition to ParseResults listed as `[ [ a1, b1, c1, ...], [ a2, b2, c2, ...] ]` it also acts as a dictionary with entries defined as `{ a1 : [ b1, c1, ... ] }, { a2 : [ b2, c2, ... ] }`; this is especially useful when processing tabular data where the first column contains a key value for that line of data
    - list elements that are deleted using `del` will still be accessible by their dictionary keys
    - supports `items()` and `keys()` methods, similar to a dictionary
  - as a nested list
    - results returned from the Group class are encapsulated within their own list structure, so that the tokens can be handled as a hierarchical tree

  ParseResults can also be converted to an ordinary list of strings by calling `asList()`. Note that this will strip the results of any field names that have been defined for any embedded parse elements. (The `pprint` module is especially good at printing out the nested contents given by `asList()`.)

  Finally, ParseResults can be converted to an XML string by calling `asXML()`. Where possible, results will be tagged using the results names defined for the respective ParseExpressions. `asXML()` takes two optional arguments:

  - doctagname - for ParseResults that do not have a defined name, this argument will wrap the resulting XML in a set of opening and closing tags `<doctagname>` and `</doctagname>`.
  - namedItemsOnly (default=False) - flag to indicate if the generated XML should skip items that do not have defined names. If a nested group item is named, then all embedded items will be included, whether they have names or not.

# 3 Miscellaneous attributes and methods

## 3.1 Helper methods

- `delimitedList( expr, delim=',')` - convenience function for matching one or more occurrences of expr, separated by delim. By default, the delimiters are suppressed, so the returned results contain only the separate list elements. Can optionally specify `combine=True`, indicating that the expressions and delimiters should be returned as one combined value (useful for scoped variables, such as `a.b.c`, or `a::b::c`, or paths such as `a/b/c`).
- `oneOf( string, caseless=False )` - convenience function for quickly declaring an alternative set of `Literal` tokens, by splitting the given string on whitespace boundaries. The tokens are sorted so that longer matches are attempted first; this ensures that a short token does not mask a longer one that starts with the same characters. If `caseless=True`, will create an alternative set of CaselessLiteral tokens.
- `dictOf( key, value )` - convenience function for quickly declaring a dictionary pattern of `Dict( ZeroOrMore( Group( key + value ) ) )`.
- `lineno( loc, string )` - function to give the line number of the location within the string; the first line is line 1, newlines start new rows
- `col( loc, string )` - function to give the column number of the location within the string; the first column is column 1, newlines reset the column number to 1
- `line( loc, string )` - function to retrieve the line of text representing `lineno( loc, string )`; useful when printing out diagnostic messages for exceptions

## 3.2 Common string and token constants

- `alphas` - same as `string.letters`
- `nums` - same as `string.digits`
- `alphanums` - a string containing `alphas + nums`
- `printables` - same as `string.printable`, minus the space (`' '`) character
- `empty` - a global `Empty()`; will always match
- `sglQuotedString` - a string of characters enclosed in 's; may include whitespace, but not newlines
- `dblQuotedString` - a string of characters enclosed in "s; may include whitespace, but not newlines
- `quotedString` - `sglQuotedString | dblQuotedString`
- `cStyleComment` - a comment block delimited by `'/*'` and `'*/'` sequences; can span multiple lines, but does not support nesting of comments
- `htmlComment` - a comment block delimited by `'<!--'` and `'-->'` sequences; can span multiple lines, but does not support nesting of comments
- `commaSeparatedList` - similar to `delimitedList`, except that the list expressions can be any text value, or a quoted string; quoted strings can safely include commas without incorrectly breaking the string into two tokens
- `restOfLine` - all remaining printable characters up to but not including the next newline