clang 10.0.0git

Classes | Public Member Functions | Friends | List of all members
clang::syntax::TokenBuffer Class Reference

A list of tokens obtained by preprocessing a text buffer and operations to map between the expanded and spelled tokens, i.e. More...

`#include "clang/Tooling/Syntax/Tokens.h"`

## Classes
struct | Expansion
| An expansion produced by the preprocessor, includes macro expansions and preprocessor directives. More...

## Public Member Functions
| TokenBuffer (const SourceManager &SourceMgr)
llvm::ArrayRef< syntax::Token > | expandedTokens () const
| All tokens produced by the preprocessor after all macro replacements, directives, etc. More...
llvm::ArrayRef< syntax::Token > | expandedTokens (SourceRange R) const
| Returns the subrange of expandedTokens() corresponding to the closed token range R. More...
llvm::Optional< llvm::ArrayRef< syntax::Token > > | spelledForExpanded (llvm::ArrayRef< syntax::Token > Expanded) const
| Find the subrange of spelled tokens that produced the corresponding `Expanded` tokens. More...
llvm::Optional< Expansion > | expansionStartingAt (const syntax::Token *Spelled) const
| If `Spelled` starts a mapping (e.g. More...
llvm::ArrayRef< syntax::Token > | spelledTokens (FileID FID) const
| Lexed tokens of a file before preprocessing. More...
std::vector< const syntax::Token * > | macroExpansions (FileID FID) const
| Get all tokens that expand a macro in `FID`. More...
const SourceManager & | sourceManager () const
std::string | dumpForTests () const

## Friends
class | TokenCollector

## Detailed Description

A list of tokens obtained by preprocessing a text buffer and operations to map between the expanded and spelled tokens, i.e.

TokenBuffer has information about two token streams:

1. Expanded tokens: tokens produced by the preprocessor after all macro replacements,
2. Spelled tokens: corresponding directly to the source code of a file before any macro replacements occurred. Here's an example to illustrate a difference between those two: #define FOO 10 int a = FOO;

Spelled tokens are {'#','define','FOO','10','int','a','=','FOO',';'}. Expanded tokens are {'int','a','=','10',';','eof'}.

Note that the expanded token stream has a tok::eof token at the end, the spelled tokens never store a 'eof' token.

The full list expanded tokens can be obtained with expandedTokens(). Spelled tokens for each of the files can be obtained via spelledTokens(FileID).

To map between the expanded and spelled tokens use findSpelledByExpanded().

To build a token buffer use the TokenCollector class. You can also compute the spelled tokens of a file using the tokenize() helper.

FIXME: allow to map from spelled to expanded tokens when use-case shows up. FIXME: allow mappings into macro arguments.

Definition at line 176 of file Tokens.h.

## Constructor & Destructor Documentation

## ◆ TokenBuffer()

clang::syntax::TokenBuffer::TokenBuffer | ( | const SourceManager & | SourceMgr | ) |
| inline

Definition at line 178 of file Tokens.h.

## Member Function Documentation

## ◆ dumpForTests()

std::string TokenBuffer::dumpForTests | ( |  | ) | const

Definition at line 615 of file Tokens.cpp.

References clang::frontend::DumpTokens, and clang::comments::tok::eof.

## ◆ expandedTokens() [1/2]

llvm::ArrayRef<syntax::Token> clang::syntax::TokenBuffer::expandedTokens | ( |  | ) | const
| inline

All tokens produced by the preprocessor after all macro replacements, directives, etc.

Source locations found in the clang AST will always point to one of these tokens. Tokens are in TU order (per SourceManager::isBeforeInTranslationUnit()). FIXME: figure out how to handle token splitting, e.g. '>>' can be split into two '>' tokens by the parser. However, TokenBuffer currently keeps it as a single '>>' token.

Definition at line 186 of file Tokens.h.

## ◆ expandedTokens() [2/2]

llvm::ArrayRef< syntax::Token > TokenBuffer::expandedTokens | ( | SourceRange | R | ) | const

Returns the subrange of expandedTokens() corresponding to the closed token range R.

Definition at line 122 of file Tokens.cpp.

References clang::SourceRange::getBegin(), clang::SourceRange::getEnd(), clang::SourceRange::isInvalid(), and clang::syntax::Token::location().

## ◆ expansionStartingAt()

llvm::Optional< TokenBuffer::Expansion > TokenBuffer::expansionStartingAt | ( | const syntax::Token * | Spelled | ) | const

If `Spelled` starts a mapping (e.g.

if it's a macro name or '#' starting a preprocessor directive) return the subrange of expanded tokens that the macro expands to.

Definition at line 232 of file Tokens.cpp.

References clang::syntax::TokenBuffer::Expansion::Expanded, clang::SourceLocation::isFileID(), clang::syntax::Token::location(), clang::None, and clang::syntax::TokenBuffer::Expansion::Spelled.

## ◆ macroExpansions()

std::vector< const syntax::Token * > TokenBuffer::macroExpansions | ( | FileID | FID | ) | const

Get all tokens that expand a macro in `FID`.

For the following input #define FOO B #define FOO2(X) int X FOO2(XY) int B; FOO; macroExpansions() returns {"FOO2", "FOO"} (from line 3 and 5 respecitvely).

Definition at line 282 of file Tokens.cpp.

References clang::syntax::Token::kind().

## ◆ sourceManager()

const SourceManager& clang::syntax::TokenBuffer::sourceManager | ( |  | ) | const
| inline

Definition at line 258 of file Tokens.h.

## ◆ spelledForExpanded()

llvm::Optional< llvm::ArrayRef< syntax::Token > > TokenBuffer::spelledForExpanded | ( | llvm::ArrayRef< syntax::Token > | Expanded | ) | const

Find the subrange of spelled tokens that produced the corresponding `Expanded` tokens.

EXPECTS: `Expanded` is a subrange of expandedTokens().

Will fail if the expanded tokens do not correspond to a sequence of spelled tokens. E.g. for the following example:

#define FIRST f1 f2 f3 #define SECOND s1 s2 s3

a FIRST b SECOND c // expanded tokens are: a f1 f2 f3 b s1 s2 s3 c

the results would be:

## expanded => spelled

a => a s1 s2 s3 => SECOND a f1 f2 f3 => a FIRST a f1 => can't map s1 s2 => can't map

If `Expanded` is empty, the returned value is llvm::None. Complexity is logarithmic.

Definition at line 191 of file Tokens.cpp.

References clang::syntax::Token::location(), and clang::None.

## ◆ spelledTokens()

llvm::ArrayRef< syntax::Token > TokenBuffer::spelledTokens | ( | FileID | FID | ) | const

Lexed tokens of a file before preprocessing.

E.g. for the following input #define DECL(name) int name = 10 DECL(a); spelledTokens() returns {"#", "define", "DECL", "(", "name", ")", "int", "name", "=", "10", "DECL", "(", "a", ")", ";"}

Definition at line 179 of file Tokens.cpp.

## Friends And Related Function Documentation

## ◆ TokenCollector

friend class TokenCollector
| friend

Definition at line 299 of file Tokens.h.

The documentation for this class was generated from the following files:

- include/clang/Tooling/Syntax/Tokens.h
- lib/Tooling/Syntax/Tokens.cpp