clang 10.0.0git

Public Member Functions | Static Public Member Functions | Friends | List of all members
clang::Lexer Class Reference

Lexer - This provides a simple interface that turns a text buffer into a stream of tokens. More...

`#include "clang/Lex/Lexer.h"`

Inheritance diagram for clang::Lexer:
[legend]
Collaboration diagram for clang::Lexer:
[legend]

## Public Member Functions
| Lexer (FileID FID, const llvm::MemoryBuffer *InputFile, Preprocessor &PP)
| Lexer constructor - Create a new lexer object for the specified buffer with the specified preprocessor managing the lexing process. More...
| Lexer (SourceLocation FileLoc, const LangOptions &LangOpts, const char *BufStart, const char *BufPtr, const char *BufEnd)
| Lexer constructor - Create a new raw lexer object. More...
| Lexer (FileID FID, const llvm::MemoryBuffer *FromFile, const SourceManager &SM, const LangOptions &LangOpts)
| Lexer constructor - Create a new raw lexer object. More...
| Lexer (const Lexer &)=delete
Lexer & | operator= (const Lexer &)=delete
const LangOptions & | getLangOpts () const
| getLangOpts - Return the language features currently enabled. More...
SourceLocation | getFileLoc () const
| getFileLoc - Return the File Location for the file we are lexing out of. More...
bool | isPragmaLexer () const
| isPragmaLexer - Returns true if this Lexer is being used to lex a pragma. More...
bool | LexFromRawLexer (Token &Result)
| LexFromRawLexer - Lex a token from a designated raw lexer (one with no associated preprocessor object. More...
bool | isKeepWhitespaceMode () const
| isKeepWhitespaceMode - Return true if the lexer should return tokens for every character in the file, including whitespace and comments. More...
void | SetKeepWhitespaceMode (bool Val)
| SetKeepWhitespaceMode - This method lets clients enable or disable whitespace retention mode. More...
bool | inKeepCommentMode () const
| inKeepCommentMode - Return true if the lexer should return comments as tokens. More...
void | SetCommentRetentionState (bool Mode)
| SetCommentRetentionMode - Change the comment retention mode of the lexer to the specified mode. More...
void | resetExtendedTokenMode ()
| Sets the extended token mode back to its initial value, according to the language options and preprocessor. More...
StringRef | getBuffer () const
| Gets source code buffer. More...
void | ReadToEndOfLine (SmallVectorImpl< char > *Result=nullptr)
| ReadToEndOfLine - Read the rest of the current preprocessor line as an uninterpreted string. More...
DiagnosticBuilder | Diag (const char *Loc, unsigned DiagID) const
| Diag - Forwarding function for diagnostics. More...
SourceLocation | getSourceLocation (const char *Loc, unsigned TokLen=1) const
| getSourceLocation - Return a source location identifier for the specified offset in the current file. More...
SourceLocation | getSourceLocation () override
| getSourceLocation - Return a source location for the next character in the current file. More...
const char * | getBufferLocation () const
| Return the current location in the buffer. More...
unsigned | getCurrentBufferOffset ()
| Returns the current lexing offset. More...
bool | skipOver (unsigned NumBytes)
| Skip over `NumBytes` bytes. More...
Public Member Functions inherited from clang::PreprocessorLexer
| PreprocessorLexer (const PreprocessorLexer &)=delete
PreprocessorLexer & | operator= (const PreprocessorLexer &)=delete
void | LexIncludeFilename (Token &FilenameTok)
| Lex a token, producing a header-name token if possible. More...
void | setParsingPreprocessorDirective (bool f)
| Inform the lexer whether or not we are currently lexing a preprocessor directive. More...
bool | isLexingRawMode () const
| Return true if this lexer is in raw mode or not. More...
Preprocessor * | getPP () const
| Return the preprocessor object for this lexer. More...
FileID | getFileID () const
unsigned | getInitialNumSLocEntries () const
| Number of SLocEntries before lexing the file. More...
const FileEntry * | getFileEntry () const
| getFileEntry - Return the FileEntry corresponding to this FileID. More...
conditional_iterator | conditional_begin () const
conditional_iterator | conditional_end () const
void | setConditionalLevels (ArrayRef< PPConditionalInfo > CL)

## Static Public Member Functions
static Lexer * | Create_PragmaLexer (SourceLocation SpellingLoc, SourceLocation ExpansionLocStart, SourceLocation ExpansionLocEnd, unsigned TokLen, Preprocessor &PP)
| Create_PragmaLexer: Lexer constructor - Create a new lexer object for _Pragma expansion. More...
static std::string | Stringify (StringRef Str, bool Charify=false)
| Stringify - Convert the specified string into a C string by i) escaping '\' and " characters and ii) replacing newline character(s) with "\n". More...
static void | Stringify (SmallVectorImpl< char > &Str)
| Stringify - Convert the specified string into a C string by i) escaping '\' and " characters and ii) replacing newline character(s) with "\n". More...
static unsigned | getSpelling (const Token &Tok, const char *&Buffer, const SourceManager &SourceMgr, const LangOptions &LangOpts, bool *Invalid=nullptr)
| getSpelling - This method is used to get the spelling of a token into a preallocated buffer, instead of as an std::string. More...
static std::string | getSpelling (const Token &Tok, const SourceManager &SourceMgr, const LangOptions &LangOpts, bool *Invalid=nullptr)
| getSpelling() - Return the 'spelling' of the Tok token. More...
static StringRef | getSpelling (SourceLocation loc, SmallVectorImpl< char > &buffer, const SourceManager &SM, const LangOptions &options, bool *invalid=nullptr)
| getSpelling - This method is used to get the spelling of the token at the given source location. More...
static unsigned | MeasureTokenLength (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
| MeasureTokenLength - Relex the token at the specified location and return its length in bytes in the input file. More...
static bool | getRawToken (SourceLocation Loc, Token &Result, const SourceManager &SM, const LangOptions &LangOpts, bool IgnoreWhiteSpace=false)
| Relex the token at the specified location. More...
static SourceLocation | GetBeginningOfToken (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
| Given a location any where in a source buffer, find the location that corresponds to the beginning of the token in which the original source location lands. More...
static unsigned | getTokenPrefixLength (SourceLocation TokStart, unsigned CharNo, const SourceManager &SM, const LangOptions &LangOpts)
| Get the physical length (including trigraphs and escaped newlines) of the first `Characters` characters of the token starting at TokStart. More...
static SourceLocation | AdvanceToTokenCharacter (SourceLocation TokStart, unsigned Characters, const SourceManager &SM, const LangOptions &LangOpts)
| AdvanceToTokenCharacter - If the current SourceLocation specifies a location at the start of a token, return a new location that specifies a character within the token. More...
static SourceLocation | getLocForEndOfToken (SourceLocation Loc, unsigned Offset, const SourceManager &SM, const LangOptions &LangOpts)
| Computes the source location just past the end of the token at this source location. More...
static CharSourceRange | getAsCharRange (SourceRange Range, const SourceManager &SM, const LangOptions &LangOpts)
| Given a token range, produce a corresponding CharSourceRange that is not a token range. More...
static CharSourceRange | getAsCharRange (CharSourceRange Range, const SourceManager &SM, const LangOptions &LangOpts)
static bool | isAtStartOfMacroExpansion (SourceLocation loc, const SourceManager &SM, const LangOptions &LangOpts, SourceLocation *MacroBegin=nullptr)
| Returns true if the given MacroID location points at the first token of the macro expansion. More...
static bool | isAtEndOfMacroExpansion (SourceLocation loc, const SourceManager &SM, const LangOptions &LangOpts, SourceLocation *MacroEnd=nullptr)
| Returns true if the given MacroID location points at the last token of the macro expansion. More...
static CharSourceRange | makeFileCharRange (CharSourceRange Range, const SourceManager &SM, const LangOptions &LangOpts)
| Accepts a range and returns a character range with file locations. More...
static StringRef | getSourceText (CharSourceRange Range, const SourceManager &SM, const LangOptions &LangOpts, bool *Invalid=nullptr)
| Returns a string for the source that the range encompasses. More...
static StringRef | getImmediateMacroName (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
| Retrieve the name of the immediate macro expansion. More...
static StringRef | getImmediateMacroNameForDiagnostics (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
| Retrieve the name of the immediate macro expansion. More...
static PreambleBounds | ComputePreamble (StringRef Buffer, const LangOptions &LangOpts, unsigned MaxLines=0)
| Compute the preamble of the given file. More...
static Optional< Token > | findNextToken (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
| Finds the token that comes right after the given location. More...
static SourceLocation | findLocationAfterToken (SourceLocation loc, tok::TokenKind TKind, const SourceManager &SM, const LangOptions &LangOpts, bool SkipTrailingWhitespaceAndNewLine)
| Checks that the given token is the first token that occurs after the given location (this excludes comments and whitespace). More...
static bool | isIdentifierBodyChar (char c, const LangOptions &LangOpts)
| Returns true if the given character could appear in an identifier. More...
static bool | isNewLineEscaped (const char *BufferStart, const char *Str)
| Checks whether new line pointed by Str is preceded by escape sequence. More...
static char | getCharAndSizeNoWarn (const char *Ptr, unsigned &Size, const LangOptions &LangOpts)
| getCharAndSizeNoWarn - Like the getCharAndSize method, but does not ever emit a warning. More...
static StringRef | getIndentationForLine (SourceLocation Loc, const SourceManager &SM)
| Returns the leading whitespace for line that corresponds to the given location `Loc`. More...

## Friends
class | Preprocessor

## Additional Inherited Members
Public Types inherited from clang::PreprocessorLexer
using | conditional_iterator = SmallVectorImpl< PPConditionalInfo >::const_iterator
| Iterator that traverses the current stack of preprocessor conditional directives (#if/#ifdef/#ifndef). More...
Protected Member Functions inherited from clang::PreprocessorLexer
| PreprocessorLexer ()
| PreprocessorLexer (Preprocessor *pp, FileID fid)
virtual | ~PreprocessorLexer ()=default
void | pushConditionalLevel (SourceLocation DirectiveStart, bool WasSkipping, bool FoundNonSkip, bool FoundElse)
| pushConditionalLevel - When we enter a #if directive, this keeps track of what we are currently in for diagnostic emission (e.g. More...
void | pushConditionalLevel (const PPConditionalInfo &CI)
bool | popConditionalLevel (PPConditionalInfo &CI)
| popConditionalLevel - Remove an entry off the top of the conditional stack, returning information about it. More...
PPConditionalInfo & | peekConditionalLevel ()
| Return the top of the conditional stack. More...
unsigned | getConditionalStackDepth () const
Protected Attributes inherited from clang::PreprocessorLexer
Preprocessor * | PP = nullptr
const FileID | FID
| The SourceManager FileID corresponding to the file being lexed. More...
unsigned | InitialNumSLocEntries = 0
| Number of SLocEntries before lexing the file. More...
bool | ParsingPreprocessorDirective = false
| True when parsing #XXX; turns '\n' into a tok::eod token. More...
bool | ParsingFilename = false
| True after #include; turns <xx> or "xxx" into a tok::header_name token. More...
bool | LexingRawMode = false
| True if in raw mode. More...
MultipleIncludeOpt | MIOpt
| A state machine that detects the #ifndef-wrapping a file idiom for the multiple-include optimization. More...
SmallVector< PPConditionalInfo, 4 > | ConditionalStack
| Information about the set of #if/#ifdef/#ifndef blocks we are currently in. More...

## Detailed Description

Lexer - This provides a simple interface that turns a text buffer into a stream of tokens.

This provides no support for file reading or buffering, or buffering/seeking of tokens, only forward lexing is supported. It relies on the specified Preprocessor object to handle preprocessor directives, etc.

Definition at line 76 of file Lexer.h.

## Constructor & Destructor Documentation

## ◆ Lexer() [1/4]

Lexer::Lexer | ( | FileID | FID,
|  | const llvm::MemoryBuffer * | InputFile,
|  | Preprocessor & | PP
| ) |  |

Lexer constructor - Create a new lexer object for the specified buffer with the specified preprocessor managing the lexing process.

This lexer assumes that the associated file buffer and Preprocessor objects will outlive it, so it doesn't take ownership of either of them.

Definition at line 133 of file Lexer.cpp.

References resetExtendedTokenMode().

Referenced by Create_PragmaLexer().

## ◆ Lexer() [2/4]

Lexer::Lexer | ( | SourceLocation | fileloc,
|  | const LangOptions & | langOpts,
|  | const char * | BufStart,
|  | const char * | BufPtr,
|  | const char * | BufEnd
| ) |  |

Lexer constructor - Create a new raw lexer object.

This object is only suitable for calls to 'LexFromRawLexer'. This lexer assumes that the text range will outlive it, so it doesn't take ownership of it.

Definition at line 146 of file Lexer.cpp.

References clang::PreprocessorLexer::LexingRawMode.

## ◆ Lexer() [3/4]

Lexer::Lexer | ( | FileID | FID,
|  | const llvm::MemoryBuffer * | FromFile,
|  | const SourceManager & | SM,
|  | const LangOptions & | langOpts
| ) |  |

Lexer constructor - Create a new raw lexer object.

This object is only suitable for calls to 'LexFromRawLexer'. This lexer assumes that the text range will outlive it, so it doesn't take ownership of it.

Definition at line 158 of file Lexer.cpp.

## ◆ Lexer() [4/4]

clang::Lexer::Lexer | ( | const Lexer & |  | ) |
| delete

## Member Function Documentation

## ◆ AdvanceToTokenCharacter()

static SourceLocation clang::Lexer::AdvanceToTokenCharacter | ( | SourceLocation | TokStart,
|  | unsigned | Characters,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| inline static

AdvanceToTokenCharacter - If the current SourceLocation specifies a location at the start of a token, return a new location that specifies a character within the token.

This handles trigraphs and escaped newlines.

Definition at line 363 of file Lexer.h.

References clang::SourceLocation::getLocWithOffset(), and Offset.

Referenced by Diag(), clang::StringLiteral::getLocationOfByte(), getUDSuffixLoc(), and MakeCharSourceRange().

## ◆ ComputePreamble()

PreambleBounds Lexer::ComputePreamble | ( | StringRef | Buffer,
|  | const LangOptions & | LangOpts,
|  | unsigned | MaxLines = `0`
| ) |  |
| static

Compute the preamble of the given file.

The preamble of a file contains the initial comments, include directives, and other preprocessor directives that occur before the code in this particular file actually begins. The preamble of the main source file is a potential prefix header.

Parameters

  Buffer | The memory buffer containing the file's contents.
  MaxLines | If non-zero, restrict the length of the preamble to fewer than this number of lines.

Returns
  The offset into the file where the preamble ends and the rest of the file begins along with a boolean value indicating whether the preamble ends at the beginning of a new line.

Definition at line 582 of file Lexer.cpp.

References End, clang::comments::tok::eof, clang::SourceLocation::getFromRawEncoding(), clang::Token::getKind(), clang::Token::getLocation(), clang::SourceLocation::getRawEncoding(), clang::Token::getRawIdentifier(), clang::Token::isAtStartOfLine(), clang::SourceLocation::isInvalid(), clang::SourceLocation::isValid(), clang::Token::needsCleaning(), and SetCommentRetentionState().

Referenced by clang::ComputePreambleBounds(), and clang::PrintPreambleAction::ExecuteAction().

## ◆ Create_PragmaLexer()

Lexer * Lexer::Create_PragmaLexer | ( | SourceLocation | SpellingLoc,
|  | SourceLocation | ExpansionLocStart,
|  | SourceLocation | ExpansionLocEnd,
|  | unsigned | TokLen,
|  | Preprocessor & | PP
| ) |  |
| static

Create_PragmaLexer: Lexer constructor - Create a new lexer object for _Pragma expansion.

This has a variety of magic semantics that this method sets up. It returns a new'd Lexer that must be delete'd when done.

This has a variety of magic semantics that this method sets up. It returns a new'd Lexer that must be delete'd when done.

On entrance to this routine, TokStartLoc is a macro location which has a spelling loc that indicates the bytes to be lexed for the token and an expansion location that indicates where all lexed tokens should be "expanded from".

TODO: It would really be nice to make _Pragma just be a wrapper around a normal lexer that remaps tokens as they fly by. This would require making Preprocessor::Lex virtual. Given that, we could just dump in a magic lexer interface that could handle this stuff. This would pull GetMappedTokenLoc out of the critical path of the lexer!

Definition at line 186 of file Lexer.cpp.

References clang::SourceManager::createExpansionLoc(), clang::SourceManager::getBuffer(), clang::SourceManager::getCharacterData(), clang::SourceManager::getFileID(), clang::SourceManager::getLocForStartOfFile(), clang::Preprocessor::getSourceManager(), Lexer(), clang::PreprocessorLexer::ParsingPreprocessorDirective, and SM.

## ◆ Diag()

DiagnosticBuilder Lexer::Diag | ( | const char * | Loc,
|  | unsigned | DiagID
| ) |  | const

Diag - Forwarding function for diagnostics.

This translate a source position in the current buffer into a SourceLocation object for rendering.

Definition at line 1145 of file Lexer.cpp.

References clang::Preprocessor::Diag(), and getSourceLocation().

Referenced by DecodeTrigraphChar(), FindConflictEnd(), findLocationAfterToken(), findPlaceholderEnd(), isEndOfBlockCommentWithEscapedNewLine(), maybeDiagnoseUTF8Homoglyph(), and ReadToEndOfLine().

## ◆ findLocationAfterToken()

SourceLocation Lexer::findLocationAfterToken | ( | SourceLocation | Loc,
|  | tok::TokenKind | TKind,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts,
|  | bool | SkipTrailingWhitespaceAndNewLine
| ) |  |
| static

Checks that the given token is the first token that occurs after the given location (this excludes comments and whitespace).

Returns the location immediately after the specified token. If the token is not found or the location is inside a macro, the returned source location will be invalid.

Definition at line 1270 of file Lexer.cpp.

References clang::C, DecodeTrigraphChar(), Diag(), findNextToken(), clang::SourceManager::getCharacterData(), clang::SourceLocation::getLocWithOffset(), GetTrigraphCharForLetter(), clang::isHorizontalWhitespace(), clang::PreprocessorLexer::isLexingRawMode(), clang::isWhitespace(), clang::Token::NeedsCleaning, Offset, clang::Token::setFlag(), and clang::format::Tok.

Referenced by clang::Sema::AddCFAuditedAttribute(), DiagnoseMismatchedNewDelete(), clang::Sema::DiagnoseUnterminatedPragmaPack(), and GenerateFixForUnusedDecl().

## ◆ findNextToken()

Optional< Token > Lexer::findNextToken | ( | SourceLocation | Loc,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| static

Finds the token that comes right after the given location.

Returns the next token, or none if the location is inside a macro.

Definition at line 1237 of file Lexer.cpp.

References clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), getLocForEndOfToken(), clang::SourceManager::getLocForStartOfFile(), isAtEndOfMacroExpansion(), clang::SourceLocation::isMacroID(), clang::None, and clang::format::Tok.

Referenced by clang::tooling::ExtractionSemicolonPolicy::compute(), findLocationAfterToken(), and LookThroughTransitiveAssignmentsAndCommaOperators().

## ◆ getAsCharRange() [1/2]

static CharSourceRange clang::Lexer::getAsCharRange | ( | SourceRange | Range,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| inline static

Given a token range, produce a corresponding CharSourceRange that is not a token range.

This allows the source range to be used by components that don't have access to the lexer and thus can't find the end of the range for themselves.

Definition at line 394 of file Lexer.h.

References End, clang::SourceRange::getBegin(), clang::SourceRange::getEnd(), clang::SourceLocation::isInvalid(), and SM.

Referenced by clang::arcmt::writeARCDiagsToPlist().

## ◆ getAsCharRange() [2/2]

static CharSourceRange clang::Lexer::getAsCharRange | ( | CharSourceRange | Range,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| inline static

Definition at line 402 of file Lexer.h.

References clang::CharSourceRange::getAsRange(), clang::CharSourceRange::isTokenRange(), and SM.

## ◆ GetBeginningOfToken()

SourceLocation Lexer::GetBeginningOfToken | ( | SourceLocation | Loc,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| static

Given a location any where in a source buffer, find the location that corresponds to the beginning of the token in which the original source location lands.

Definition at line 554 of file Lexer.cpp.

References getBeginningOfFileToken(), clang::SourceManager::getDecomposedLoc(), clang::SourceLocation::getLocWithOffset(), clang::SourceManager::getSpellingLoc(), clang::SourceLocation::isFileID(), and clang::SourceManager::isMacroArgExpansion().

Referenced by adjustRemoval(), and findPreviousTokenStart().

## ◆ getBuffer()

StringRef clang::Lexer::getBuffer | ( |  | ) | const
| inline

Gets source code buffer.

Definition at line 242 of file Lexer.h.

References Diag().

Referenced by clang::html::SyntaxHighlight().

## ◆ getBufferLocation()

const char* clang::Lexer::getBufferLocation | ( |  | ) | const
| inline

Return the current location in the buffer.

Definition at line 266 of file Lexer.h.

## ◆ getCharAndSizeNoWarn()

static char clang::Lexer::getCharAndSizeNoWarn | ( | const char * | Ptr,
|  | unsigned & | Size,
|  | const LangOptions & | LangOpts
| ) |  |
| inline static

getCharAndSizeNoWarn - Like the getCharAndSize method, but does not ever emit a warning.

Definition at line 543 of file Lexer.h.

References AttributeLangSupport::C, P, clang::Token::setKind(), clang::Token::setLength(), and clang::Token::setLocation().

Referenced by getSpellingSlow(), getTokenPrefixLength(), and maybeDiagnoseUTF8Homoglyph().

## ◆ getCurrentBufferOffset()

unsigned clang::Lexer::getCurrentBufferOffset | ( |  | ) |
| inline

Returns the current lexing offset.

Definition at line 269 of file Lexer.h.

References SM, and clang::format::Tok.

## ◆ getFileLoc()

SourceLocation clang::Lexer::getFileLoc | ( |  | ) | const
| inline

getFileLoc - Return the File Location for the file we are lexing out of.

The physical location encodes the location where the characters come from, the virtual location encodes where we should claim the characters came from. Currently this is only used by _Pragma handling.

Definition at line 174 of file Lexer.h.

## ◆ getImmediateMacroName()

StringRef Lexer::getImmediateMacroName | ( | SourceLocation | Loc,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| static

Retrieve the name of the immediate macro expansion.

This routine starts from a source location, and finds the name of the macro responsible for its immediate expansion. It looks through any intervening macro argument expansions to compute this. It returns a StringRef which refers to the SourceManager-owned buffer of the source where that macro name is spelled. Thus, the result shouldn't out-live that SourceManager.

Definition at line 975 of file Lexer.cpp.

References clang::PreprocessorLexer::FID, clang::CharSourceRange::getBegin(), clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::SrcMgr::SLocEntry::getExpansion(), clang::SrcMgr::ExpansionInfo::getExpansionLocStart(), clang::SourceManager::getFileID(), clang::SourceManager::getImmediateExpansionRange(), clang::SourceManager::getSLocEntry(), clang::SrcMgr::ExpansionInfo::getSpellingLoc(), clang::SourceManager::getSpellingLoc(), clang::SourceLocation::isFileID(), clang::SourceManager::isInFileID(), clang::SrcMgr::ExpansionInfo::isMacroArgExpansion(), clang::SourceLocation::isMacroID(), and MeasureTokenLength().

Referenced by CheckMemaccessSize(), getMacroName(), clang::ento::CheckerContext::getMacroNameOrSpelling(), and IsEnumConstOrFromMacro().

## ◆ getImmediateMacroNameForDiagnostics()

StringRef Lexer::getImmediateMacroNameForDiagnostics | ( | SourceLocation | Loc,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| static

Retrieve the name of the immediate macro expansion.

This routine starts from a source location, and finds the name of the macro responsible for its immediate expansion. It looks through any intervening macro argument expansions to compute this. It returns a StringRef which refers to the SourceManager-owned buffer of the source where that macro name is spelled. Thus, the result shouldn't out-live that SourceManager.

This differs from Lexer::getImmediateMacroName in that any macro argument location will result in the topmost function macro that accepted it. e.g.

MAC1( MAC2(foo) )

for location of 'foo' token, this function will return "MAC1" while Lexer::getImmediateMacroName will return "MAC2".

Definition at line 1022 of file Lexer.cpp.

References clang::CharSourceRange::getBegin(), clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::SourceManager::getFileEntryForID(), clang::SourceManager::getFileID(), clang::SourceManager::getImmediateExpansionRange(), clang::SourceManager::getSpellingLoc(), clang::SourceManager::isMacroArgExpansion(), clang::SourceLocation::isMacroID(), and MeasureTokenLength().

Referenced by DiagnoseNullConversion(), and mapDiagnosticRanges().

## ◆ getIndentationForLine()

StringRef Lexer::getIndentationForLine | ( | SourceLocation | Loc,
|  | const SourceManager & | SM
| ) |  |
| static

Returns the leading whitespace for line that corresponds to the given location `Loc`.

Definition at line 1071 of file Lexer.cpp.

References findBeginningOfLine(), clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), GetMappedTokenLoc(), clang::SourceLocation::isInvalid(), clang::SourceLocation::isMacroID(), Line, and clang::PreprocessorLexer::PP.

## ◆ getLangOpts()

const LangOptions& clang::Lexer::getLangOpts | ( |  | ) | const
| inline

getLangOpts - Return the language features currently enabled.

NOTE: this lexer modifies features as a file is parsed!

Definition at line 168 of file Lexer.h.

Referenced by DecodeTrigraphChar(), findPlaceholderEnd(), isEndOfBlockCommentWithEscapedNewLine(), and maybeDiagnoseUTF8Homoglyph().

## ◆ getLocForEndOfToken()

SourceLocation Lexer::getLocForEndOfToken | ( | SourceLocation | Loc,
|  | unsigned | Offset,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| static

Computes the source location just past the end of the token at this source location.

This routine can be used to produce a source location that points just past the end of the token referenced by `Loc`, and is generally used when a diagnostic needs to point just after a token where it expected something different that it received. If the returned source location would not be meaningful (e.g., if it points into a macro), this routine returns an invalid source location.

Parameters

  Offset | an offset from the end of the token, where the source location should refer to. The default offset (0) produces a source location pointing just past the end of the token; an offset of 1 produces a source location pointing to the last character in the token, etc.

Definition at line 778 of file Lexer.cpp.

References clang::SourceLocation::getLocWithOffset(), isAtEndOfMacroExpansion(), clang::SourceLocation::isInvalid(), clang::SourceLocation::isMacroID(), MeasureTokenLength(), and Offset.

Referenced by clang::transformer::after(), findNextToken(), clang::arcmt::trans::findSemiAfterLocation(), clang::Sema::getLocForEndOfToken(), clang::diff::SyntaxTree::getSourceRangeOffsets(), LookThroughTransitiveAssignmentsAndCommaOperators(), makeRangeFromFileLocs(), clang::Parser::ParseUnqualifiedId(), clang::edit::Commit::replaceText(), clang::tooling::RecursiveSymbolVisitor< T >::TraverseNestedNameSpecifierLoc(), clang::tooling::RecursiveSymbolVisitor< T >::VisitCXXConstructorDecl(), clang::tooling::RecursiveSymbolVisitor< T >::VisitTypedefTypeLoc(), and clang::tooling::RecursiveSymbolVisitor< T >::VisitTypeLoc().

## ◆ getRawToken()

bool Lexer::getRawToken | ( | SourceLocation | Loc,
|  | Token & | Result,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts,
|  | bool | IgnoreWhiteSpace = `false`
| ) |  |
| static

Relex the token at the specified location.

Returns
  true if there was a failure, false on success.

Definition at line 455 of file Lexer.cpp.

References clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::SourceManager::getExpansionLoc(), clang::SourceManager::getLocForStartOfFile(), clang::isWhitespace(), and SetCommentRetentionState().

Referenced by findPreviousTokenKind(), and MeasureTokenLength().

## ◆ getSourceLocation() [1/2]

SourceLocation Lexer::getSourceLocation | ( | const char * | Loc,
|  | unsigned | TokLen = `1`
| ) |  | const

getSourceLocation - Return a source location identifier for the specified offset in the current file.

Definition at line 1126 of file Lexer.cpp.

References clang::SourceLocation::getLocWithOffset(), GetMappedTokenLoc(), and clang::SourceLocation::isFileID().

Referenced by DetectEOL(), and makeCharRange().

## ◆ getSourceLocation() [2/2]

SourceLocation clang::Lexer::getSourceLocation | ( |  | ) |
| inline override virtual

getSourceLocation - Return a source location for the next character in the current file.

Implements clang::PreprocessorLexer.

Definition at line 261 of file Lexer.h.

Referenced by Diag(), findPlaceholderEnd(), isEndOfBlockCommentWithEscapedNewLine(), and ReadToEndOfLine().

## ◆ getSourceText()

StringRef Lexer::getSourceText | ( | CharSourceRange | Range,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts,
|  | bool * | Invalid = `nullptr`
| ) |  |
| static

Returns a string for the source that the range encompasses.

Definition at line 939 of file Lexer.cpp.

References clang::CharSourceRange::getBegin(), clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::CharSourceRange::getEnd(), clang::SourceManager::isInFileID(), clang::CharSourceRange::isInvalid(), and makeFileCharRange().

Referenced by clang::edit::EditedSource::clearRewrites(), constructDebugPieceForTrackedCondition(), diagnoseXorMisusedAsPow(), FindLocsWithCommonFileID(), GCRewriteFinalize(), clang::diff::SyntaxTree::Impl::getDeclValue(), GetDefaultValueString(), getLiteralInfo(), isReleased(), printExprAsWritten(), clang::tooling::replaceStmtWithStmt(), and clang::tooling::ReplaceNodeWithTemplate::run().

## ◆ getSpelling() [1/3]

unsigned Lexer::getSpelling | ( | const Token & | Tok,
|  | const char *& | Buffer,
|  | const SourceManager & | SourceMgr,
|  | const LangOptions & | LangOpts,
|  | bool * | Invalid = `nullptr`
| ) |  |
| static

getSpelling - This method is used to get the spelling of a token into a preallocated buffer, instead of as an std::string.

The caller is required to allocate enough space for the token, which is guaranteed to be at least Tok.getLength() bytes long. The length of the actual result is returned.

Note that this method may do two possible things: it may either fill in the buffer specified with characters, or it may change the input pointer to point to a constant buffer with the data already in it (avoiding a copy). The caller is not allowed to modify the returned buffer pointer if an internal buffer is returned.

The caller is required to allocate enough space for the token, which is guaranteed to be at least Tok.getLength() bytes long. The actual length of the token is returned.

Note that this method may do two possible things: it may either fill in the buffer specified with characters, or it may change the input pointer to point to a constant buffer with the data already in it (avoiding a copy). The caller is not allowed to modify the returned buffer pointer if an internal buffer is returned.

Definition at line 397 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), clang::Token::getIdentifierInfo(), clang::Token::getLength(), clang::Token::getLiteralData(), clang::Token::getLocation(), clang::Token::getRawIdentifier(), getSpellingSlow(), clang::Token::hasUCN(), clang::Token::is(), clang::Token::isLiteral(), and clang::Token::needsCleaning().

Referenced by findDirectives(), clang::ModuleMapParser::getLocation(), clang::ento::CheckerContext::getMacroNameOrSpelling(), clang::StringLiteralParser::getOffsetOfStringByte(), ReadOriginalFileName(), clang::edit::EditsReceiver::remove(), and clang::StringLiteralParser::StringLiteralParser().

## ◆ getSpelling() [2/3]

std::string Lexer::getSpelling | ( | const Token & | Tok,
|  | const SourceManager & | SourceMgr,
|  | const LangOptions & | LangOpts,
|  | bool * | Invalid = `nullptr`
| ) |  |
| static

getSpelling() - Return the 'spelling' of the Tok token.

getSpelling() - Return the 'spelling' of this token.

The spelling of a token is the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

The spelling of a token are the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

Definition at line 365 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), clang::Token::getLength(), clang::Token::getLocation(), getSpellingSlow(), clang::Token::needsCleaning(), and clang::Result.

## ◆ getSpelling() [3/3]

StringRef Lexer::getSpelling | ( | SourceLocation | loc,
|  | SmallVectorImpl< char > & | buffer,
|  | const SourceManager & | SM,
|  | const LangOptions & | options,
|  | bool * | invalid = `nullptr`
| ) |  |
| static

getSpelling - This method is used to get the spelling of the token at the given source location.

getSpelling() - Return the 'spelling' of this token.

If, as is usually true, it is not necessary to copy any data, then the returned string may not point into the provided buffer.

This method lexes at the expansion depth of the given location and does not jump to the expansion or spelling location.

The spelling of a token are the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

Definition at line 324 of file Lexer.cpp.

References clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::SourceManager::getLocForStartOfFile(), getSpellingSlow(), and length().

## ◆ getTokenPrefixLength()

unsigned Lexer::getTokenPrefixLength | ( | SourceLocation | TokStart,
|  | unsigned | CharNo,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| static

Get the physical length (including trigraphs and escaped newlines) of the first `Characters` characters of the token starting at TokStart.

Definition at line 718 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), and getCharAndSizeNoWarn().

## ◆ inKeepCommentMode()

bool clang::Lexer::inKeepCommentMode | ( |  | ) | const
| inline

inKeepCommentMode - Return true if the lexer should return comments as tokens.

Definition at line 220 of file Lexer.h.

Referenced by findPlaceholderEnd(), and isEndOfBlockCommentWithEscapedNewLine().

## ◆ isAtEndOfMacroExpansion()

bool Lexer::isAtEndOfMacroExpansion | ( | SourceLocation | loc,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts,
|  | SourceLocation * | MacroEnd = `nullptr`
| ) |  |
| static

Returns true if the given MacroID location points at the last token of the macro expansion.

Parameters

  MacroEnd | If non-null and function returns true, it is set to end location of the macro.

Definition at line 822 of file Lexer.cpp.

References clang::SourceLocation::getLocWithOffset(), clang::SourceManager::getSpellingLoc(), clang::SourceManager::isAtEndOfImmediateMacroExpansion(), clang::SourceLocation::isFileID(), clang::SourceLocation::isMacroID(), clang::SourceLocation::isValid(), and MeasureTokenLength().

Referenced by FindLocsWithCommonFileID(), findNextToken(), clang::arcmt::trans::findSemiAfterLocation(), getLocForEndOfToken(), makeFileCharRange(), and clang::edit::Commit::replaceText().

## ◆ isAtStartOfMacroExpansion()

bool Lexer::isAtStartOfMacroExpansion | ( | SourceLocation | loc,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts,
|  | SourceLocation * | MacroBegin = `nullptr`
| ) |  |
| static

Returns true if the given MacroID location points at the first token of the macro expansion.

Parameters

  MacroBegin | If non-null and function returns true, it is set to begin location of the macro.

Definition at line 800 of file Lexer.cpp.

References clang::SourceManager::isAtStartOfImmediateMacroExpansion(), clang::SourceLocation::isMacroID(), and clang::SourceLocation::isValid().

Referenced by FindLocsWithCommonFileID(), makeFileCharRange(), and clang::edit::Commit::replaceText().

## ◆ isIdentifierBodyChar()

bool Lexer::isIdentifierBodyChar | ( | char | c,
|  | const LangOptions & | LangOpts
| ) |  |
| static

Returns true if the given character could appear in an identifier.

Definition at line 1047 of file Lexer.cpp.

References clang::isIdentifierBody().

Referenced by addFixitForObjCARCConversion(), and canBeJoined().

## ◆ isKeepWhitespaceMode()

bool clang::Lexer::isKeepWhitespaceMode | ( |  | ) | const
| inline

isKeepWhitespaceMode - Return true if the lexer should return tokens for every character in the file, including whitespace and comments.

This should only be used in raw mode, as the preprocessor is not prepared to deal with the excess tokens.

Definition at line 206 of file Lexer.h.

Referenced by findPlaceholderEnd(), and isEndOfBlockCommentWithEscapedNewLine().

## ◆ isNewLineEscaped()

bool Lexer::isNewLineEscaped | ( | const char * | BufferStart,
|  | const char * | Str
| ) |  |
| static

Checks whether new line pointed by Str is preceded by escape sequence.

Definition at line 1051 of file Lexer.cpp.

References clang::isHorizontalWhitespace(), and clang::isVerticalWhitespace().

Referenced by findBeginningOfLine().

## ◆ isPragmaLexer()

bool clang::Lexer::isPragmaLexer | ( |  | ) | const
| inline

isPragmaLexer - Returns true if this Lexer is being used to lex a pragma.

Definition at line 183 of file Lexer.h.

Referenced by clang::Preprocessor::GetCurDirLookup(), and ReadToEndOfLine().

## ◆ LexFromRawLexer()

bool clang::Lexer::LexFromRawLexer | ( | Token & | Result | ) |
| inline

LexFromRawLexer - Lex a token from a designated raw lexer (one with no associated preprocessor object.

Return true if the 'next character to read' pointer points at the end of the lexer buffer, false otherwise.

Definition at line 194 of file Lexer.h.

Referenced by DetectEOL(), findDirectives(), and clang::TokenRewriter::TokenRewriter().

## ◆ makeFileCharRange()

CharSourceRange Lexer::makeFileCharRange | ( | CharSourceRange | Range,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| static

Accepts a range and returns a character range with file locations.

Returns a null range if a part of the range resides inside a macro expansion or the range does not reside on the same FileID.

This function is trying to deal with macros and return a range based on file locations. The cases where it can successfully handle macros are:

-begin or end range lies at the start or end of a macro expansion, in which case the location will be set to the expansion point, e.g: #define M 1 2 a M If you have a range [a, 2] (where 2 came from the macro), the function will return a range for "a M" if you have range [a, 1], the function will fail because the range overlaps with only a part of the macro

-The macro is a function macro and the range can be mapped to the macro arguments, e.g: #define M 1 2 #define FM(x) x FM(a b M) if you have range [b, 2], the function will return the file range "b M" inside the macro arguments. if you have range [a, 2], the function will return the file range "FM(a b M)" since the range includes all of the macro expansion.

Definition at line 875 of file Lexer.cpp.

References Begin, End, clang::CharSourceRange::getBegin(), clang::CharSourceRange::getEnd(), clang::SrcMgr::SLocEntry::getExpansion(), clang::SrcMgr::ExpansionInfo::getExpansionLocStart(), clang::SourceManager::getFileID(), clang::SourceManager::getImmediateSpellingLoc(), clang::SourceManager::getSLocEntry(), isAtEndOfMacroExpansion(), isAtStartOfMacroExpansion(), clang::CharSourceRange::isCharRange(), clang::SourceLocation::isFileID(), clang::SourceLocation::isInvalid(), clang::SrcMgr::ExpansionInfo::isMacroArgExpansion(), clang::SourceLocation::isMacroID(), clang::CharSourceRange::isTokenRange(), makeRangeFromFileLocs(), clang::CharSourceRange::setBegin(), and clang::CharSourceRange::setEnd().

Referenced by clang::tooling::getRangeForEdit(), getSourceText(), makeStandaloneRange(), and clang::edit::Commit::replaceText().

## ◆ MeasureTokenLength()

unsigned Lexer::MeasureTokenLength | ( | SourceLocation | Loc,
|  | const SourceManager & | SM,
|  | const LangOptions & | LangOpts
| ) |  |
| static

MeasureTokenLength - Relex the token at the specified location and return its length in bytes in the input file.

If the token needs cleaning (e.g. includes a trigraph or an escaped newline) then this count includes bytes that are part of that.

Definition at line 444 of file Lexer.cpp.

References clang::Token::getLength(), and getRawToken().

Referenced by buildFixItInsertionLine(), EmitAlphaCounter(), clang::TextDiagnostic::emitDiagnosticLoc(), getImmediateMacroName(), getImmediateMacroNameForDiagnostics(), getLocForEndOfToken(), clang::Rewriter::getRangeSize(), getRangeSize(), clang::Rewriter::getRewrittenText(), clang::html::HighlightRange(), highlightRange(), isAtEndOfMacroExpansion(), clang::edit::Commit::replaceText(), clang::CoverageSourceInfo::SourceRangeSkipped(), and clang::JSONNodeDumper::Visit().

## ◆ operator=()

Lexer& clang::Lexer::operator= | ( | const Lexer & |  | ) |
| delete

## ◆ ReadToEndOfLine()

void Lexer::ReadToEndOfLine | ( | SmallVectorImpl< char > * | Result = `nullptr` | ) |

ReadToEndOfLine - Read the rest of the current preprocessor line as an uninterpreted string.

This switches the lexer out of directive mode.

Definition at line 2657 of file Lexer.cpp.

References clang::Preprocessor::CodeCompleteNaturalLanguage(), clang::PreprocessorLexer::ConditionalStack, clang::FixItHint::CreateInsertion(), Diag(), clang::Preprocessor::Diag(), clang::comments::tok::eof, clang::Preprocessor::getCodeCompletionFileLoc(), clang::Preprocessor::getDiagnostics(), getSourceLocation(), clang::Preprocessor::HandleEndOfFile(), clang::Token::is(), clang::DiagnosticsEngine::isIgnored(), clang::Preprocessor::isInPrimaryFile(), clang::PreprocessorLexer::isLexingRawMode(), isPragmaLexer(), clang::Preprocessor::isRecordingPreamble(), clang::PreprocessorLexer::LexingRawMode, clang::PreprocessorLexer::ParsingFilename, clang::PreprocessorLexer::ParsingPreprocessorDirective, resetExtendedTokenMode(), clang::Preprocessor::setRecordedPreambleConditionalStack(), clang::Token::startToken(), and clang::format::Tok.

## ◆ resetExtendedTokenMode()

void Lexer::resetExtendedTokenMode | ( |  | ) |

Sets the extended token mode back to its initial value, according to the language options and preprocessor.

This controls whether the lexer produces comment and whitespace tokens.

This requires the lexer to have an associated preprocessor. A standalone lexer has nothing to reset to.

Definition at line 163 of file Lexer.cpp.

References clang::Preprocessor::getCommentRetentionState(), clang::PreprocessorLexer::PP, SetCommentRetentionState(), and SetKeepWhitespaceMode().

Referenced by findPlaceholderEnd(), Lexer(), and ReadToEndOfLine().

## ◆ SetCommentRetentionState()

void clang::Lexer::SetCommentRetentionState | ( | bool | Mode | ) |
| inline

SetCommentRetentionMode - Change the comment retention mode of the lexer to the specified mode.

This is really only useful when lexing in raw mode, because otherwise the lexer needs to manage this.

Definition at line 227 of file Lexer.h.

Referenced by ComputePreamble(), DetectEOL(), findDirectives(), getBeginningOfFileToken(), getRawToken(), LexRawTokensFromMainFile(), and resetExtendedTokenMode().

## ◆ SetKeepWhitespaceMode()

void clang::Lexer::SetKeepWhitespaceMode | ( | bool | Val | ) |
| inline

SetKeepWhitespaceMode - This method lets clients enable or disable whitespace retention mode.

Definition at line 212 of file Lexer.h.

Referenced by DetectEOL(), clang::DumpRawTokensAction::ExecuteAction(), resetExtendedTokenMode(), and clang::TokenRewriter::TokenRewriter().

## ◆ skipOver()

bool Lexer::skipOver | ( | unsigned | NumBytes | ) |

Skip over `NumBytes` bytes.

If the skip is successful, the next token will be lexed from the new offset. The lexer also assumes that we skipped to the start of the line.

Returns
  true if the skip failed (new offset would have been past the end of the buffer), false otherwise.

Definition at line 221 of file Lexer.cpp.

## ◆ Stringify() [1/2]

std::string Lexer::Stringify | ( | StringRef | Str,
|  | bool | Charify = `false`
| ) |  |
| static

Stringify - Convert the specified string into a C string by i) escaping '\' and " characters and ii) replacing newline character(s) with "\n".

If Charify is true, this escapes the ' character instead of ".

Definition at line 255 of file Lexer.cpp.

References clang::Result, and StringifyImpl().

Referenced by getFunctionSourceLocation(), PrintHeaderInfo(), remapMacroPath(), and clang::MacroArgs::StringifyArgument().

## ◆ Stringify() [2/2]

void Lexer::Stringify | ( | SmallVectorImpl< char > & | Str | ) |
| static

Stringify - Convert the specified string into a C string by i) escaping '\' and " characters and ii) replacing newline character(s) with "\n".

Definition at line 262 of file Lexer.cpp.

References StringifyImpl().

## Friends And Related Function Documentation

## ◆ Preprocessor

friend class Preprocessor
| friend

Definition at line 77 of file Lexer.h.

The documentation for this class was generated from the following files:

- include/clang/Lex/Lexer.h
- lib/Lex/Lexer.cpp