Manpages - lex.1p

This manual page is part of the POSIX Programmer’s Manual. The Linux implementation of this interface may differ (consult the corresponding Linux manual page for details of Linux behavior), or the interface may not be implemented on Linux.

lex — generate programs for lexical tasks (DEVELOPMENT)

  lex [-t] [-n|-v] [file...]

The lex utility shall generate C programs to be used in lexical processing of character input, and that can be used as an interface to yacc. The C programs shall be generated from lex source code and conform to the ISO C standard, without depending on any undefined, unspecified, or implementation-defined behavior, except in cases where the code is copied directly from the supplied source, or in cases that are documented by the implementation. Usually, the lex utility shall write the program it generates to the file lex.yy.c; the state of this file is unspecified if lex exits with a non-zero exit status. See the EXTENDED DESCRIPTION section for a complete description of the lex input language.

The lex utility shall conform to the Base Definitions volume of POSIX.1‐2017, Section 12.2, Utility Syntax Guidelines, except for Guideline 9.

The following options shall be supported:

-n

Suppress the summary of statistics usually written with the -v option. If no table sizes are specified in the lex source code and the -v option is not specified, then -n is implied.

-t

Write the resulting program to standard output instead of lex.yy.c.

-v

Write a summary of lex statistics to the standard output. (See the discussion of lex table sizes in Definitions in lex.) If the -t option is specified and -n is not specified, this report shall be written to standard error. If table sizes are specified in the lex source code, and if the -n option is not specified, the -v option may be enabled.

The following operand shall be supported:

file

A pathname of an input file. If more than one such file is specified, all files shall be concatenated to produce a single lex program. If no file operands are specified, or if a file operand is ’-’, the standard input shall be used.

The standard input shall be used if no file operands are specified, or if a file operand is ’-’. See INPUT FILES.

The input files shall be text files containing lex source code, as described in the EXTENDED DESCRIPTION section.

The following environment variables shall affect the execution of lex:

LANG

Provide a default value for the internationalization variables that are unset or null. (See the Base Definitions volume of POSIX.1‐2017, Section 8.2, Internationalization Variables for the precedence of internationalization variables used to determine the values of locale categories.)

LC_ALL

If set to a non-empty string value, override the values of all the other internationalization variables.

LC_COLLATE

Determine the locale for the behavior of ranges, equivalence classes, and multi-character collating elements within regular expressions. If this variable is not set to the POSIX locale, the results are unspecified.

LC_CTYPE

Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments and input files), and the behavior of character classes within regular expressions. If this variable is not set to the POSIX locale, the results are unspecified.

LC_MESSAGES

Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error.

NLSPATH

Determine the location of message catalogs for the processing of LC_MESSAGES.

Default.

If the -t option is specified, the text file of C source code output of lex shall be written to standard output.

If the -t option is not specified:

*

Implementation-defined informational, error, and warning messages concerning the contents of lex source code input shall be written to either the standard output or standard error.

*

If the -v option is specified and the -n option is not specified, lex statistics shall also be written to either the standard output or standard error, in an implementation-defined format. These statistics may also be generated if table sizes are specified with a ’%’ operator in the Definitions section, as long as the -n option is not specified.

If the -t option is specified, implementation-defined informational, error, and warning messages concerning the contents of lex source code input shall be written to the standard error.

If the -t option is not specified:

1.

Implementation-defined informational, error, and warning messages concerning the contents of lex source code input shall be written to either the standard output or standard error.

2.

If the -v option is specified and the -n option is not specified, lex statistics shall also be written to either the standard output or standard error, in an implementation-defined format. These statistics may also be generated if table sizes are specified with a ’%’ operator in the Definitions section, as long as the -n option is not specified.

A text file containing C source code shall be written to lex.yy.c, or to the standard output if the -t option is present.

Each input file shall contain lex source code, which is a table of regular expressions with corresponding actions in the form of C program fragments.

When lex.yy.c is compiled and linked with the lex library (using the -l l operand with c99), the resulting program shall read character input from the standard input and shall partition it into strings that match the given expressions.

When an expression is matched, these actions shall occur:

*

The input string that was matched shall be left in yytext as a null-terminated string; yytext shall either be an external character array or a pointer to a character string. As explained in Definitions in lex, the type can be explicitly selected using the %array or %pointer declarations, but the default is implementation-defined.

*

The external int yyleng shall be set to the length of the matching string.

*

The expression’s corresponding program fragment, or action, shall be executed.

During pattern matching, lex shall search the set of patterns for the single longest possible match. Among rules that match the same number of characters, the rule given first shall be chosen.

The general format of lex source shall be:

Definitions %% Rules %% /User/Subroutines

The first “%%” is required to mark the beginning of the rules (regular expressions and actions); the second “%%” is required only if user subroutines follow.

Any line in the Definitions section beginning with a <blank> shall be assumed to be a C program fragment and shall be copied to the external definition area of the lex.yy.c file. Similarly, anything in the Definitions section included between delimiter lines containing only “%{” and “%}” shall also be copied unchanged to the external definition area of the lex.yy.c file.

Any such input (beginning with a <blank> or within “%{” and “%}” delimiter lines) appearing at the beginning of the Rules section before any rules are specified shall be written to lex.yy.c after the declarations of variables for the /yylex/() function and before the first line of code in /yylex/(). Thus, user variables local to /yylex/() can be declared here, as well as application code to execute upon entry to /yylex/().

The action taken by lex when encountering any input beginning with a <blank> or within “%{” and “%}” delimiter lines appearing in the Rules section but coming after one or more rules is undefined. The presence of such input may result in an erroneous definition of the /yylex/() function.

C-language code in the input shall not contain C-language trigraphs. The C-language code within “%{” and “%}” delimiter lines shall not contain any lines consisting only of “%}”, or only of “%%”.

The following exit values shall be returned:

 0

Successful completion.

>0

An error occurred.

Default.

The following sections are informative.

Conforming applications are warned that in the Rules section, an ERE without an action is not acceptable, but need not be detected as erroneous by lex. This may result in compilation or runtime errors.

The purpose of /input/() is to take characters off the input stream and discard them as far as the lexical analysis is concerned. A common use is to discard the body of a comment once the beginning of a comment is recognized.

The lex utility is not fully internationalized in its treatment of regular expressions in the lex source code or generated lexical analyzer. It would seem desirable to have the lexical analyzer interpret the regular expressions given in the lex source according to the environment specified when the lexical analyzer is executed, but this is not possible with the current lex technology. Furthermore, the very nature of the lexical analyzers produced by lex must be closely tied to the lexical requirements of the input language being described, which is frequently locale-specific anyway. (For example, writing an analyzer that is used for French text is not automatically useful for processing other languages.)

The following is an example of a lex program that implements a rudimentary scanner for a Pascal-like syntax:

    %{
    /* Need this for the call to atof() below. */
    #include <math.h>
    /* Need this for printf(), fopen(), and stdin below. */
    #include <stdio.h>
    %}
    DIGIT    [0-9]
    ID       [a-z][a-z0-9]*
    %%
    {DIGIT}+ {
        printf("An integer: %s (%d)\n", yytext,
            atoi(yytext));
        }
    {DIGIT}+"."{DIGIT}*        {
        printf("A float: %s (%g)\n", yytext,
            atof(yytext));
        }
    if|then|begin|end|procedure|function        {
        printf("A keyword: %s\n", yytext);
        }
    {ID}    printf("An identifier: %s\n", yytext);
    "+"|"-"|"*"|"/"        printf("An operator: %s\n", yytext);
    "{"[^}\n]*"}"    /* Eat up one-line comments. */
    [ \t\n]+        /* Eat up white space. */
    .  printf("Unrecognized character: %s\n", yytext);
    %%
    int main(int argc, char *argv[])
    {
        ++argv, --argc;  /* Skip over program name. */
        if (argc > 0)
            yyin = fopen(argv[0], "r");
        else
            yyin = stdin;
        yylex();
    }

Even though the -c option and references to the C language are retained in this description, lex may be generalized to other languages, as was done at one time for EFL, the Extended FORTRAN Language. Since the lex input specification is essentially language-independent, versions of this utility could be written to produce Ada, Modula-2, or Pascal code, and there are known historical implementations that do so.

The current description of lex bypasses the issue of dealing with internationalized EREs in the lex source code or generated lexical analyzer. If it follows the model used by awk (the source code is assumed to be presented in the POSIX locale, but input and output are in the locale specified by the environment variables), then the tables in the lexical analyzer produced by lex would interpret EREs specified in the lex source in terms of the environment variables specified when lex was executed. The desired effect would be to have the lexical analyzer interpret the EREs given in the lex source according to the environment specified when the lexical analyzer is executed, but this is not possible with the current lex technology.

The description of octal and hexadecimal-digit escape sequences agrees with the ISO C standard use of escape sequences.

Earlier versions of this standard allowed for implementations with bytes other than eight bits, but this has been modified in this version.

There is no detailed output format specification. The observed behavior of lex under four different historical implementations was that none of these implementations consistently reported the line numbers for error and warning messages. Furthermore, there was a desire that lex be allowed to output additional diagnostic messages. Leaving message formats unspecified avoids these formatting questions and problems with internationalization.

Although the %x specifier for exclusive start conditions is not historical practice, it is believed to be a minor change to historical implementations and greatly enhances the usability of lex programs since it permits an application to obtain the expected functionality with fewer statements.

The %array and %pointer declarations were added as a compromise between historical systems. The System V-based lex copies the matched text to a yytext array. The flex program, supported in BSD and GNU systems, uses a pointer. In the latter case, significant performance improvements are available for some scanners. Most historical programs should require no change in porting from one system to another because the string being referenced is null-terminated in both cases. (The method used by flex in its case is to null-terminate the token in place by remembering the character that used to come right after the token and replacing it before continuing on to the next scan.) Multi-file programs with external references to yytext outside the scanner source file should continue to operate on their historical systems, but would require one of the new declarations to be considered strictly portable.

The description of EREs avoids unnecessary duplication of ERE details because their meanings within a lex ERE are the same as that for the ERE in this volume of POSIX.1‐2017.

The reason for the undefined condition associated with text beginning with a <blank> or within “%{” and “%}” delimiter lines appearing in the Rules section is historical practice. Both the BSD and System V lex copy the indented (or enclosed) input in the Rules section (except at the beginning) to unreachable areas of the yylex/() function (the code is written directly after a /break statement). In some cases, the System V lex generates an error message or a syntax error, depending on the form of indented input.

The intention in breaking the list of functions into those that may appear in lex.yy.c versus those that only appear in libl.a is that only those functions in libl.a can be reliably redefined by a conforming application.

The descriptions of standard output and standard error are somewhat complicated because historical lex implementations chose to issue diagnostic messages to standard output (unless -t was given). POSIX.1‐2008 allows this behavior, but leaves an opening for the more expected behavior of using standard error for diagnostics. Also, the System V behavior of writing the statistics when any table sizes are given is allowed, while BSD-derived systems can avoid it. The programmer can always precisely obtain the desired results by using either the -t or -n options.

The OPERANDS section does not mention the use of - as a synonym for standard input; not all historical implementations support such usage for any of the file operands.

A description of the translation table was deleted from early proposals because of its relatively low usage in historical applications.

The change to the definition of the /input/() function that allows buffering of input presents the opportunity for major performance gains in some applications.

The following examples clarify the differences between lex regular expressions and regular expressions appearing elsewhere in this volume of POSIX.1‐2017. For regular expressions of the form “r/x”, the string matching r is always returned; confusion may arise when the beginning of x matches the trailing portion of r. For example, given the regular expression “a*b/cc” and the input “aaabcc”, yytext would contain the string “aaab” on this match. But given the regular expression “x*/xy” and the input “xxxy”, the token xxx, not xx, is returned by some implementations because xxx matches “x“*.

In the rule “ab*/bc”, the “b“* at the end of r extends r’s match into the beginning of the trailing context, so the result is unspecified. If this rule were “ab/bc”, however, the rule matches the text “ab” when it is followed by the text “bc”. In this latter case, the matching of r cannot extend into the beginning of x, so the result is specified.

None.

/c99 , //ed , //yacc /

The Base Definitions volume of POSIX.1‐2017, Chapter 5, File Format Notation, Chapter 8, Environment Variables, Chapter 9, Regular Expressions, Section 12.2, Utility Syntax Guidelines

Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1-2017, Standard for Information Technology – Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, Copyright (C) 2018 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. In the event of any discrepancy between this version and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Standard is the referee document. The original Standard can be obtained online at http://www.opengroup.org/unix/online.html .

Any typographical or formatting errors that appear in this page are most likely to have been introduced during the conversion of the source files to man page format. To report such errors, see https://www.kernel.org/doc/man-pages/reporting_bugs.html .