red specifies the list of tokens that must have lex start conditions
%red
lex-token names ...
The red section of a source description file specifies the red tokens in a specification grammar. A red token is only recognized if it follows another token that can validly precede it according to the grammar. To determine which tokens can follow other tokens ivy*meta SDTB automatically computes the token follow graph of the grammar.
The follow graph is used to compute the lex start conditions for the red tokens; the start conditions are automatically included in the generated lex.l file. When start conditions are used, each token that is recognized by the lexer will leave the lexer in a specific state. This state defines exactly which set of tokens may be recognized next in the specification; only tokens in that set will be recognized. This allows great freedom in the description of tokens that overlap each other. For example, consider the following grammar description:
spec: ( id+ «%%» codeline+ )
codeline:<.*>
id:<[a-z]*>
Without start conditions, an id could never be recognized because codeline always has precedence (because codeline is more general, and because it is defined earlier that id). However, with the use of start conditions, a codeline will only be recognized following %% or following another codeline token.
Normally all lex-tokens in the grammar section are considered to be red tokens. The red section should only be used in the very rare occurrence that the grammar defines too many tokens, and one or more limits in the lex program are exceeded.
The red section is used only in conjunction with the R feature of ivy*meta SDTB (see FEATURES(2) ). If the R feature is not selected, then the red section is ignored. The R option specifies ivy*meta SDTB to use start conditions only for those tokens specified in the red section.
Occasionally you may encounter a ivy*meta SDTB error condition, ``Too many red tokens.'' This can be confusing, especially when you have not specified any red tokens. This error occurs because all tokens are red unless a red section is used to designate specific red tokens. The error message means that there are too many tokens in the grammar. Generally, the number of tokens can be reduced by using the following methods:
Look for repeated examples of literal tokens, for example «(» and «)". Make these named tokens instead (e.g., bparen:"(» and eparen:")"), and then use the names rather than the literals.
Look for rules that consist of literal alternatives, and turn them into tokens (e.g., boolean:<TRUE|FALSE>, instead of boolean: ( «TRUE» | «FALSE» )).
These techniques also apply to the lex error messages, ``Too many start conditions'' and ``Start condition too long.'' These error messages all refer to internal lex buffer lengths that are exceeded when there are too many red tokens in the grammar.
Lex A Lexical Analyzer Generator, M.E. Lesk and E. Schmidt