1[/==============================================================================
2    Copyright (C) 2001-2011 Joel de Guzman
3    Copyright (C) 2001-2011 Hartmut Kaiser
4
5    Distributed under the Boost Software License, Version 1.0. (See accompanying
6    file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
7===============================================================================/]
8
9[section:lexer_tutorials __lex__ Tutorials Overview]
10
11The __lex__ library implements several components on top of possibly different
12lexer generator libraries. It exposes a pair of iterators, which, when
13dereferenced, return a stream of tokens generated from the underlying character
14stream. The generated tokens are based on the token definitions supplied by the
15user.
16
17Currently, __lex__ is built on top of Ben Hanson's excellent __lexertl__
18library (which is a proposed Boost library). __lexertl__ provides the necessary
19functionality to build state machines based on a set of supplied regular
20expressions. But __lex__ is not restricted to be used with __lexertl__. We
21expect it to be usable in conjunction with any other lexical scanner generator
22library, all what needs to be implemented is a set of wrapper objects exposing a
23well defined interface as described in this documentation.
24
25[note   For the sake of clarity all examples in this documentation assume
26        __lex__ to be used on top of __lexertl__.]
27
28Building a lexer using __lex__ is highly configurable, where most of this
29configuration is done at compile time. Almost all of the configurable
30parameters have generally useful default values, allowing project startup to be
31a easy and straightforward task. Here is a (non-complete) list of features you
32can tweak to adjust the generated lexer instance to the actual needs:
33
34* Select and customize the token type to be generated by the lexer instance.
35* Select and customize the token value types the generated token instances will
36  be able to hold.
37* Select the iterator type of the underlying input stream, which will be used
38  as the source for the character stream to tokenize.
39* Customize the iterator type returned by the lexer to enable debug support,
40  special handling of certain input sequences, etc.
41* Select the /dynamic/ or the /static/ runtime model for the lexical
42  analyzer.
43
44Special care has been taken during the development of the library that
45optimal code will be generated regardless of the configuration options
46selected.
47
48The series of tutorial examples of this section will guide you through some
49common use cases helping to understand the big picture. The first two quick
50start examples (__sec_lex_quickstart_1__ and __sec_lex_quickstart_2__)
51introduce the __lex__ library while building two stand alone applications, not
52being connected to or depending on any other part of __spirit__. The section
53__sec_lex_quickstart_3__ demonstrates how to use a lexer in conjunction with a
54parser (where obviously the parser is built using __qi__).
55
56[endsect]
57
58