The Interface to an Alex-generated lexer

Alex User Guide
Prev |  | Next

# Chapter 5. The Interface to an Alex-generated lexer

This section answers the question: "How do I include an Alex lexer in my program?

Alex provides for a great deal of flexibility in how the lexer is exposed to the rest of the program. For instance, there's no need to parse a `String` directly if you have some special character-buffer operations that avoid the overheads of ordinary Haskell `String`s. You might want Alex to keep track of the line and column number in the input text, or you might wish to do it yourself (perhaps you use a different tab width from the standard 8-columns, for example).

The general story is this: Alex provides a basic interface to the generated lexer (described in the next section), which you can use to parse tokens given an abstract input type with operations over it. You also have the option of including a wrapper, which provides a higher-level abstraction over the basic interface; Alex comes with several wrappers.

# 5.1. Basic interface

If you compile your Alex file without a `%wrapper` declaration, then you get access to the lowest-level API to the lexer. You must provide definitions for the following, either in the same module or imported from another module:

        type AlexInput
        alexGetChar       :: AlexInput -> Maybe (Char,AlexInput)
        alexInputPrevChar :: AlexInput -> Char

The generated lexer is independent of the input type, which is why you have to provide a definition for the input type yourself. Note that the input type needs to keep track of the previous character in the input stream; this is used for implementing patterns with a left-context (those that begin with `^` or ``set`^`). If you don't ever use patterns with a left-context in your lexical specification, then you can safely forget about the previous character in the input stream, and have `alexInputPrevChar` return `undefined`.

Alex will provide the following function:

        alexScan :: AlexInput	 -- The current input
                 -> Int		 -- The "start code"
                 -> Maybe (          -- Nothing on error or EOF
                         AlexInput,  -- The remaining input
                         Int,        -- Length of this token
                         action      -- The action (an unknown type)
            	  )

Calling `alexScan` will scan a single token from the input stream, and return the remaining input stream, the length of the token, and the `action`.

The `action` is simply the value of the expression inside `{...}` on the right-hand-side of the appropriate rule in the Alex file. Alex doesn't specify what type these expressions should have, it simply requires that they all have the same type, or else you'll get a type error when you try to compile the generated lexer.

Once you have the `action`, it is up to you what to do with it. The type of `action` could be a function which takes the `String` representation of the token and returns a value in some token type, or it could be a continuation that takes the new input and calls `alexScan` again, building a list of tokens as it goes.

This is pretty low-level stuff; you have complete flexibility about how you use the lexer, but there might be a fair amount of support code to write before you can actually use it. For this reason, we also provide a selection of wrappers that add some common functionality to this basic scheme. Wrappers are described in the next section.

There is another entry point, which is useful if your grammar contains any predicates (see Section 3.2.2.1):

        alexScanUser
                 :: user             -- predicate state
                 -> AlexInput	 -- The current input
                 -> Int		 -- The "start code"
                 -> Maybe (          -- Nothing on error or EOF
                         AlexInput,  -- The remaining input
                         Int,        -- Length of this token
                         action      -- The action (an unknown type)
            	  )

The extra argument, of some type `user`, is passed to each predicate.

Prev | Home | Next
Syntax of character sets |  | Wrappers