clang 10.0.0git

Public Types | Public Member Functions | List of all members
clang::Token Class Reference

Token - This structure provides full information about a lexed token. More...

`#include "clang/Lex/Token.h"`

## Public Types
enum | TokenFlags { StartOfLine = 0x01, LeadingSpace = 0x02, DisableExpand = 0x04, NeedsCleaning = 0x08, LeadingEmptyMacro = 0x10, HasUDSuffix = 0x20, HasUCN = 0x40, IgnoredComma = 0x80, StringifiedInMacro = 0x100, CommaAfterElided = 0x200, IsEditorPlaceholder = 0x400, IsReinjected = 0x800 }

## Public Member Functions
tok::TokenKind | getKind () const
void | setKind (tok::TokenKind K)
bool | is (tok::TokenKind K) const
| is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}". More...
bool | isNot (tok::TokenKind K) const
bool | isOneOf (tok::TokenKind K1, tok::TokenKind K2) const
template<typename... Ts>
bool | isOneOf (tok::TokenKind K1, tok::TokenKind K2, Ts... Ks) const
bool | isAnyIdentifier () const
| Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode). More...
bool | isLiteral () const
| Return true if this is a "literal", like a numeric constant, string, etc. More...
bool | isAnnotation () const
| Return true if this is any of tok::annot_* kind tokens. More...
SourceLocation | getLocation () const
| Return a source location identifier for the specified offset in the current file. More...
unsigned | getLength () const
void | setLocation (SourceLocation L)
void | setLength (unsigned Len)
SourceLocation | getAnnotationEndLoc () const
void | setAnnotationEndLoc (SourceLocation L)
SourceLocation | getLastLoc () const
SourceLocation | getEndLoc () const
SourceRange | getAnnotationRange () const
| SourceRange of the group of tokens that this annotation token represents. More...
void | setAnnotationRange (SourceRange R)
const char * | getName () const
void | startToken ()
| Reset all flags to cleared. More...
IdentifierInfo * | getIdentifierInfo () const
void | setIdentifierInfo (IdentifierInfo *II)
const void * | getEofData () const
void | setEofData (const void *D)
StringRef | getRawIdentifier () const
| getRawIdentifier - For a raw identifier token (i.e., an identifier lexed in raw mode), returns a reference to the text substring in the buffer if known. More...
void | setRawIdentifierData (const char *Ptr)
const char * | getLiteralData () const
| getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise. More...
void | setLiteralData (const char *Ptr)
void * | getAnnotationValue () const
void | setAnnotationValue (void *val)
void | setFlag (TokenFlags Flag)
| Set the specified flag. More...
bool | getFlag (TokenFlags Flag) const
| Get the specified flag. More...
void | clearFlag (TokenFlags Flag)
| Unset the specified flag. More...
unsigned | getFlags () const
| Return the internal represtation of the flags. More...
void | setFlagValue (TokenFlags Flag, bool Val)
| Set a flag to either true or false. More...
bool | isAtStartOfLine () const
| isAtStartOfLine - Return true if this token is at the start of a line. More...
bool | hasLeadingSpace () const
| Return true if this token has whitespace before it. More...
bool | isExpandDisabled () const
| Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2. More...
bool | isObjCAtKeyword (tok::ObjCKeywordKind objcKey) const
| Return true if we have an ObjC keyword identifier. More...
tok::ObjCKeywordKind | getObjCKeywordID () const
| Return the ObjC keyword kind. More...
bool | needsCleaning () const
| Return true if this token has trigraphs or escaped newlines in it. More...
bool | hasLeadingEmptyMacro () const
| Return true if this token has an empty macro before it. More...
bool | hasUDSuffix () const
| Return true if this token is a string or character literal which has a ud-suffix. More...
bool | hasUCN () const
| Returns true if this token contains a universal character name. More...
bool | stringifiedInMacro () const
| Returns true if this token is formed by macro by stringizing or charizing operator. More...
bool | commaAfterElided () const
| Returns true if the comma after this token was elided. More...
bool | isEditorPlaceholder () const
| Returns true if this token is an editor placeholder. More...

## Detailed Description

Token - This structure provides full information about a lexed token.

It is not intended to be space efficient, it is intended to return as much information as possible about each returned token. This is expected to be compressed into a smaller form if memory footprint is important.

The parser can create a special "annotation token" representing a stream of tokens that were parsed and semantically resolved, e.g.: "foo::MyClass<int>" can be represented by a single typename annotation token that carries information about the SourceRange of the tokens and the type object.

Definition at line 34 of file Token.h.

## Member Enumeration Documentation

## ◆ TokenFlags

enum clang::Token::TokenFlags

Enumerator
StartOfLine |
LeadingSpace |
DisableExpand |
NeedsCleaning |
LeadingEmptyMacro |
HasUDSuffix |
HasUCN |
IgnoredComma |
StringifiedInMacro |
CommaAfterElided |
IsEditorPlaceholder |
IsReinjected |

Definition at line 72 of file Token.h.

## Member Function Documentation

## ◆ clearFlag()

void clang::Token::clearFlag | ( | TokenFlags | Flag | ) |
| inline

Unset the specified flag.

Definition at line 246 of file Token.h.

Referenced by findPlaceholderEnd(), GenerateNewArgTokens(), isGCCAsmStatement(), clang::TokenLexer::Lex(), remapMacroPath(), and setFlagValue().

## ◆ commaAfterElided()

bool clang::Token::commaAfterElided | ( |  | ) | const
| inline

Returns true if the comma after this token was elided.

Definition at line 303 of file Token.h.

References CommaAfterElided, and getFlag().

## ◆ getAnnotationEndLoc()

SourceLocation clang::Token::getAnnotationEndLoc | ( |  | ) | const
| inline

Definition at line 140 of file Token.h.

References clang::SourceLocation::getFromRawEncoding(), and isAnnotation().

Referenced by getAnnotationRange(), getEndLoc(), and getLastLoc().

## ◆ getAnnotationRange()

SourceRange clang::Token::getAnnotationRange | ( |  | ) | const
| inline

SourceRange of the group of tokens that this annotation token represents.

Definition at line 160 of file Token.h.

References getAnnotationEndLoc(), and getLocation().

Referenced by clang::Parser::ParseUnqualifiedId(), and clang::Parser::TryAnnotateTypeOrScopeToken().

## ◆ getAnnotationValue()

void* clang::Token::getAnnotationValue | ( |  | ) | const
| inline

Definition at line 226 of file Token.h.

References isAnnotation().

Referenced by addConstevalToLambdaDeclSpecifier(), clang::Parser::getTypeAnnotation(), ParseAlignPragma(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PragmaLoopHintString(), and PrintPreprocessedTokens().

## ◆ getEndLoc()

SourceLocation clang::Token::getEndLoc | ( |  | ) | const
| inline

Definition at line 153 of file Token.h.

References getAnnotationEndLoc(), getLength(), getLocation(), clang::SourceLocation::getLocWithOffset(), and isAnnotation().

Referenced by clang::Preprocessor::checkModuleIsAvailable(), diagnoseDynamicExceptionSpecification(), clang::Preprocessor::Lex(), parseDeclareSimdClauses(), and VersionNumberSeparator().

## ◆ getEofData()

const void* clang::Token::getEofData | ( |  | ) | const
| inline

Definition at line 192 of file Token.h.

References clang::comments::tok::eof, and is().

Referenced by TypeTraitArity().

## ◆ getFlag()

bool clang::Token::getFlag | ( | TokenFlags | Flag | ) | const
| inline

Get the specified flag.

Definition at line 241 of file Token.h.

Referenced by commaAfterElided(), hasLeadingEmptyMacro(), hasLeadingSpace(), hasUCN(), hasUDSuffix(), isAtStartOfLine(), isEditorPlaceholder(), isExpandDisabled(), clang::Preprocessor::Lex(), needsCleaning(), and stringifiedInMacro().

## ◆ getFlags()

unsigned clang::Token::getFlags | ( |  | ) | const
| inline

Return the internal represtation of the flags.

This is only intended for low-level operations such as writing tokens to disk.

Definition at line 254 of file Token.h.

Referenced by clang::ASTWriter::AddToken(), and GenerateNewArgTokens().

## ◆ getIdentifierInfo()

IdentifierInfo* clang::Token::getIdentifierInfo | ( |  | ) | const
| inline

Definition at line 179 of file Token.h.

References clang::comments::tok::eof, is(), isAnnotation(), isLiteral(), and isNot().

Referenced by addConstevalToLambdaDeclSpecifier(), AddDefinedMacroToHash(), clang::ASTWriter::AddToken(), clang::MacroArgs::ArgNeedsPreexpansion(), attributeParsedArgsUnevaluated(), clang::TokenConcatenation::AvoidConcat(), CheckArrayDesignatorSyntax(), clang::Preprocessor::CheckMacroName(), clang::Preprocessor::checkModuleIsAvailable(), DetectEOL(), diagnoseAutoModuleImport(), diagnoseDynamicExceptionSpecification(), EvaluateDefined(), EvaluateFeatureLikeBuiltinMacro(), EvaluateValue(), ExpectFeatureIdentifierInfo(), FindLocsWithCommonFileID(), clang::PreprocessingRecord::findMacroDefinition(), GenerateNewArgTokens(), clang::arcmt::getFileRemappings(), GetFirstChar(), getIdentifier(), getMissingDeclaratorIdLoc(), getObjCKeywordID(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandleIdentifier(), clang::Preprocessor::HandlePoisonedIdentifier(), clang::PragmaNamespace::HandlePragma(), clang::Preprocessor::HandleSkippedDirectiveWhileUsingPCH(), clang::TokenLexer::Init(), clang::format::FormatToken::is(), IsBuiltInOrStandardCXX11Attribute(), IsCommonTypo(), isConfigurationPattern(), clang::format::AdditionalKeywords::isCSharpKeyword(), clang::MacroInfo::isIdenticalTo(), clang::format::AdditionalKeywords::IsJavaScriptIdentifier(), isObjCAtKeyword(), isSameToken(), isTrivialSingleTokenExpansion(), clang::VAOptDefinitionContext::isVAOptToken(), isWideStringLiteralFromMacro(), clang::TokenLexer::Lex(), clang::Preprocessor::Lex(), clang::Preprocessor::LexAfterModuleImport(), LexModuleNameComponent(), clang::Preprocessor::LexOnOffSwitch(), clang::MacroPPCallbacks::MacroDefined(), clang::MacroPPCallbacks::MacroUndefined(), clang::Preprocessor::MaybeHandlePoisonedIdentifier(), clang::format::mustBeJSIdent(), clang::TokenValue::operator==(), ParseAlignPragma(), clang::Parser::ParseConstraintLogicalOrExpression(), parseDeclareSimdClauses(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), parseOpenMPReductionId(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PragmaLoopHintString(), PrintMacroDefinition(), ReadLineMarkerFlags(), remapMacroPath(), takeDeclAttributes(), clang::Parser::TryAnnotateOptionalCXXScopeToken(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), clang::Preprocessor::TypoCorrectToken(), and VersionNumberSeparator().

## ◆ getKind()

tok::TokenKind clang::Token::getKind | ( |  | ) | const
| inline

Definition at line 92 of file Token.h.

Referenced by clang::Sema::ActOnCharacterConstant(), addConstevalToLambdaDeclSpecifier(), clang::ASTWriter::AddToken(), clang::TokenConcatenation::AvoidConcat(), clang::Sema::CheckConstraintExpression(), clang::Preprocessor::CheckMacroName(), clang::Lexer::ComputePreamble(), diagnoseDynamicExceptionSpecification(), clang::Preprocessor::DumpToken(), EvaluateDirectiveSubExpr(), EvaluateFeatureLikeBuiltinMacro(), EvaluateHasIncludeCommon(), EvaluateValue(), GenerateNewArgTokens(), getBeginningOfFileToken(), clang::ModuleMapParser::getLocation(), getMissingDeclaratorIdLoc(), clang::Parser::getNullabilityKeyword(), clang::format::FormatToken::getPrecedence(), getSpellingSlow(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandlePragmaModuleBuild(), isAnnotation(), isAnyIdentifier(), isConfigurationPattern(), clang::format::AdditionalKeywords::isCSharpKeyword(), clang::format::FormatToken::isFunctionLikeKeyword(), clang::MacroInfo::isIdenticalTo(), isLiteral(), isPipeDeclerator(), clang::Preprocessor::IsPreviousCachedToken(), isSameToken(), clang::format::FormatToken::isSimpleTypeSpecifier(), clang::format::FormatToken::isStringLiteral(), isTypeQualifier(), clang::format::FormatToken::isUnaryOperator(), clang::Preprocessor::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexNonComment(), clang::Preprocessor::LexOnOffSwitch(), clang::Preprocessor::LexUnexpandedNonComment(), clang::TokenValue::operator==(), ParseAlignPragma(), clang::Parser::ParseConstraintLogicalAndExpression(), clang::Parser::ParseConstraintLogicalOrExpression(), parseOpenMPReductionId(), ParseReductionId(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), ReadOriginalFileName(), remapMacroPath(), clang::Parser::SkipMalformedDecl(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), clang::html::SyntaxHighlight(), clang::Parser::TryAnnotateCXXScopeToken(), tryConsumeLambdaSpecifierToken(), and TypeTraitArity().

## ◆ getLastLoc()

SourceLocation clang::Token::getLastLoc | ( |  | ) | const
| inline

Definition at line 149 of file Token.h.

References getAnnotationEndLoc(), getLocation(), and isAnnotation().

Referenced by clang::Parser::TryAnnotateTypeOrScopeToken().

## ◆ getLength()

unsigned clang::Token::getLength | ( |  | ) | const
| inline

Definition at line 129 of file Token.h.

References isAnnotation().

Referenced by clang::Sema::ActOnNumericConstant(), addConstevalToLambdaDeclSpecifier(), clang::ASTWriter::AddToken(), clang::TokenConcatenation::AvoidConcat(), DetectEOL(), clang::Preprocessor::DumpToken(), getBeginningOfFileToken(), getEndLoc(), GetFirstChar(), GetLineValue(), clang::ModuleMapParser::getLocation(), clang::StringLiteralParser::getOffsetOfStringByte(), getRawIdentifier(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), getSpellingSlow(), IsBuiltInOrStandardCXX11Attribute(), IsStringPrefix(), isTrivialSingleTokenExpansion(), clang::TokenLexer::Lex(), clang::Preprocessor::LexHeaderName(), clang::Lexer::MeasureTokenLength(), PrintPreprocessedTokens(), remapMacroPath(), clang::RewriteMacrosInInput(), SelectDigraphErrorMessage(), clang::MacroArgs::StringifyArgument(), clang::html::SyntaxHighlight(), updateConsecutiveMacroArgTokens(), UseNSOptionsMacro(), and VersionNumberSeparator().

## ◆ getLiteralData()

const char* clang::Token::getLiteralData | ( |  | ) | const
| inline

getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise.

Definition at line 217 of file Token.h.

References isLiteral().

Referenced by GetFirstChar(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::TokenLexer::Lex(), and UseNSOptionsMacro().

## ◆ getLocation()

SourceLocation clang::Token::getLocation | ( |  | ) | const
| inline

Return a source location identifier for the specified offset in the current file.

Definition at line 126 of file Token.h.

References clang::SourceLocation::getFromRawEncoding().

Referenced by clang::Sema::ActOnCharacterConstant(), clang::Sema::ActOnNumericConstant(), addConstevalToLambdaDeclSpecifier(), clang::ASTWriter::AddToken(), attributeParsedArgsUnevaluated(), AuditedType(), clang::TokenConcatenation::AvoidConcat(), CheckArrayDesignatorSyntax(), clang::Preprocessor::CheckEndOfDirective(), clang::Preprocessor::CheckMacroName(), clang::Preprocessor::checkModuleIsAvailable(), ComputeDATE_TIME(), clang::Lexer::ComputePreamble(), clang::Parser::ConsumeToken(), DetectEOL(), clang::Parser::Diag(), clang::Preprocessor::Diag(), diagnoseDynamicExceptionSpecification(), diagnoseUnexpectedOperator(), clang::Preprocessor::DiscardUntilEndOfDirective(), clang::DoPrintPreprocessedInput(), clang::Preprocessor::DumpToken(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateFeatureLikeBuiltinMacro(), EvaluateHasIncludeCommon(), EvaluateValue(), clang::Parser::ExitScope(), ExpectFeatureIdentifierInfo(), findDirectives(), FindLocsWithCommonFileID(), clang::PreprocessingRecord::findMacroDefinition(), findPreviousTokenKind(), FixDigraph(), GenerateNewArgTokens(), getAnnotationRange(), getBeginningOfFileToken(), getEndLoc(), clang::arcmt::getFileRemappings(), GetFirstChar(), getLastLoc(), GetLineValue(), clang::ModuleMapParser::getLocation(), getMissingDeclaratorIdLoc(), clang::StringLiteralParser::getOffsetOfStringByte(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandleEndOfFile(), clang::Preprocessor::HandleIdentifier(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaHdrstop(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::Preprocessor::HandlePragmaPopMacro(), clang::Preprocessor::HandlePragmaSystemHeader(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), IsBuiltInOrStandardCXX11Attribute(), IsCommonTypo(), isConfigurationPattern(), isGCCAsmStatement(), clang::Preprocessor::IsPreviousCachedToken(), IsStringPrefix(), isTrivialSingleTokenExpansion(), clang::TokenLexer::Lex(), clang::Preprocessor::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexHeaderName(), LexModuleNameComponent(), clang::Preprocessor::LexOnOffSwitch(), clang::Preprocessor::LookUpIdentifierInfo(), clang::MacroPPCallbacks::MacroDefined(), TokenCollector::CollectPPExpansions::MacroExpands(), clang::MacroPPCallbacks::MacroUndefined(), outputPrintable(), ParseAlignPragma(), clang::Parser::ParseAssignmentExpression(), clang::Parser::ParseConstraintLogicalOrExpression(), parseDeclareSimdClauses(), parseDeviceSelector(), parseImplementationSelector(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), parseOpenMPReductionId(), parseOpenMPSimpleClause(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::Preprocessor::parseSimpleIntegerLiteral(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PragmaLoopHintString(), PrintPreprocessedTokens(), ReadLineMarkerFlags(), ReadOriginalFileName(), remapMacroPath(), resyncUTF8(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), SelectDigraphErrorMessage(), clang::html::SyntaxHighlight(), takeDeclAttributes(), clang::TokenRewriter::TokenRewriter(), clang::Parser::TryAnnotateCXXScopeToken(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), tryConsumeLambdaSpecifierToken(), clang::Parser::TryConsumeToken(), TypeTraitArity(), updateConsecutiveMacroArgTokens(), and VersionNumberSeparator().

## ◆ getName()

const char* clang::Token::getName | ( |  | ) | const
| inline

Definition at line 168 of file Token.h.

References clang::tok::getTokenName().

## ◆ getObjCKeywordID()

tok::ObjCKeywordKind Token::getObjCKeywordID | ( |  | ) | const

Return the ObjC keyword kind.

getObjCKeywordID - Return the ObjC keyword kind.

Definition at line 66 of file Lexer.cpp.

References clang::CMK_None, getIdentifierInfo(), clang::IdentifierInfo::getObjCKeywordID(), and isAnnotation().

Referenced by isExpandDisabled().

## ◆ getRawIdentifier()

StringRef clang::Token::getRawIdentifier | ( |  | ) | const
| inline

getRawIdentifier - For a raw identifier token (i.e., an identifier lexed in raw mode), returns a reference to the text substring in the buffer if known.

Definition at line 205 of file Token.h.

References getLength(), and is().

Referenced by clang::Preprocessor::CheckEndOfDirective(), clang::Lexer::ComputePreamble(), clang::ModuleMapParser::getLocation(), clang::Lexer::getSpelling(), clang::Preprocessor::HandlePragmaModuleBuild(), and clang::Preprocessor::LookUpIdentifierInfo().

## ◆ hasLeadingEmptyMacro()

bool clang::Token::hasLeadingEmptyMacro | ( |  | ) | const
| inline

Return true if this token has an empty macro before it.

Definition at line 289 of file Token.h.

References getFlag(), and LeadingEmptyMacro.

Referenced by findPlaceholderEnd(), and clang::Preprocessor::LexHeaderName().

## ◆ hasLeadingSpace()

bool clang::Token::hasLeadingSpace | ( |  | ) | const
| inline

Return true if this token has whitespace before it.

Definition at line 272 of file Token.h.

References getFlag(), and LeadingSpace.

Referenced by buildMSAsmString(), clang::Preprocessor::DumpToken(), findPlaceholderEnd(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), IsBuiltInOrStandardCXX11Attribute(), clang::MacroInfo::isIdenticalTo(), isTrivialSingleTokenExpansion(), clang::TokenLexer::Lex(), clang::Preprocessor::LexHeaderName(), outputPrintable(), PrintPreprocessedTokens(), remapMacroPath(), clang::RewriteMacrosInInput(), clang::MacroArgs::StringifyArgument(), and updateConsecutiveMacroArgTokens().

## ◆ hasUCN()

bool clang::Token::hasUCN | ( |  | ) | const
| inline

Returns true if this token contains a universal character name.

Definition at line 296 of file Token.h.

References getFlag(), and HasUCN.

Referenced by clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), and clang::Preprocessor::LookUpIdentifierInfo().

## ◆ hasUDSuffix()

bool clang::Token::hasUDSuffix | ( |  | ) | const
| inline

Return true if this token is a string or character literal which has a ud-suffix.

Definition at line 293 of file Token.h.

References getFlag(), and HasUDSuffix.

Referenced by clang::TokenConcatenation::AvoidConcat(), EvaluateValue(), clang::Preprocessor::FinishLexStringLiteral(), GetLineValue(), clang::ModuleMapParser::getLocation(), LexModuleNameComponent(), clang::Preprocessor::LexOnOffSwitch(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), and ReadLineMarkerFlags().

## ◆ is()

bool clang::Token::is | ( | tok::TokenKind | K | ) | const
| inline

is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}".

Definition at line 97 of file Token.h.

Referenced by addConstevalToLambdaDeclSpecifier(), attributeParsedArgsUnevaluated(), clang::TokenConcatenation::AvoidConcat(), buildMSAsmString(), CheckArrayDesignatorSyntax(), clang::Sema::CheckConstraintExpression(), clang::Preprocessor::CheckEndOfDirective(), clang::Preprocessor::CheckMacroName(), clang::Parser::ConsumeAnyToken(), clang::BalancedDelimiterTracker::consumeClose(), clang::BalancedDelimiterTracker::consumeOpen(), clang::Preprocessor::CreateString(), DetectEOL(), diagnoseDynamicExceptionSpecification(), diagnoseUnexpectedOperator(), clang::DoPrintPreprocessedInput(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateHasIncludeCommon(), EvaluateValue(), findDirectives(), FindLocsWithCommonFileID(), findPreviousTokenKind(), clang::Preprocessor::FinishLexStringLiteral(), GenerateNewArgTokens(), getEofData(), getIdentifier(), getIdentifierInfo(), GetLineValue(), clang::ModuleMapParser::getLocation(), getMissingDeclaratorIdLoc(), clang::Parser::getNullabilityKeyword(), getRawIdentifier(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), clang::Preprocessor::HandleDirective(), clang::Preprocessor::HandleIdentifier(), clang::Preprocessor::HandleMicrosoftCommentPaste(), clang::Preprocessor::HandlePragmaHdrstop(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::Preprocessor::HandlePragmaPoison(), clang::format::hash_combine(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), clang::format::FormatToken::is(), IsBuiltInOrStandardCXX11Attribute(), IsCommonTypo(), isConfigurationPattern(), isEndOfTemplateArgument(), isGCCAsmStatement(), isMapModifier(), clang::TokenLexer::isNextTokenLParen(), isOneOf(), clang::TokenLexer::isParsingPreprocessorDirective(), isPipeDeclerator(), isResultTypeOrTemplate(), isValidAfterIdentifierInDeclarator(), isWideStringLiteralFromMacro(), clang::TokenLexer::Lex(), clang::Preprocessor::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexHeaderName(), LexModuleNameComponent(), clang::Preprocessor::LexOnOffSwitch(), LexRawTokensFromMainFile(), clang::Parser::ConditionDeclarationOrInitStatementState::markNotExpression(), clang::Parser::MightBeCXXScopeToken(), outputPrintable(), ParseAlignPragma(), clang::Parser::ParseAssignmentExpression(), clang::Parser::ParseConstraintLogicalAndExpression(), clang::Parser::ParseConstraintLogicalOrExpression(), parseContextScore(), parseDeclareSimdClauses(), parseDeviceSelector(), parseImplementationSelector(), ParseLoopHintValue(), parseMapType(), clang::Parser::ParseMSAsmIdentifier(), parseOpenMPReductionId(), clang::Preprocessor::parseSimpleIntegerLiteral(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PragmaLoopHintString(), clang::format::printDebugInfo(), PrintPreprocessedTokens(), ReadLineMarkerFlags(), clang::Lexer::ReadToEndOfLine(), remapMacroPath(), clang::RewriteIncludesInInput(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), setEofData(), setRawIdentifierData(), clang::Parser::SkipMalformedDecl(), clang::Preprocessor::SkipTokensWhileUsingPCH(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), takeDeclAttributes(), clang::Parser::TryAnnotateCXXScopeToken(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), TypeTraitArity(), and VersionNumberSeparator().

## ◆ isAnnotation()

bool clang::Token::isAnnotation | ( |  | ) | const
| inline

Return true if this is any of tok::annot_* kind tokens.

Definition at line 120 of file Token.h.

References getKind(), and clang::tok::isAnnotation().

Referenced by clang::Preprocessor::AnnotateCachedTokens(), clang::TokenConcatenation::AvoidConcat(), clang::Parser::ConsumeAnyToken(), diagnoseDynamicExceptionSpecification(), ExpectFeatureIdentifierInfo(), FindLocsWithCommonFileID(), GenerateNewArgTokens(), getAnnotationEndLoc(), getAnnotationValue(), getEndLoc(), getIdentifierInfo(), getLastLoc(), getLength(), clang::Parser::getNullabilityKeyword(), getObjCKeywordID(), isObjCAtKeyword(), clang::TokenLexer::Lex(), LexModuleNameComponent(), parseDeviceSelector(), parseOpenMPDirectiveKind(), parseOpenMPSimpleClause(), remapMacroPath(), clang::Preprocessor::ReplaceLastTokenWithAnnotation(), setAnnotationEndLoc(), setAnnotationValue(), setLength(), clang::syntax::Token::Token(), clang::syntax::TokenCollector::TokenCollector(), clang::Parser::TryAnnotateOptionalCXXScopeToken(), and clang::Parser::TryAnnotateTypeOrScopeToken().

## ◆ isAnyIdentifier()

bool clang::Token::isAnyIdentifier | ( |  | ) | const
| inline

Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode).

Definition at line 109 of file Token.h.

References getKind(), and clang::tok::isAnyIdentifier().

Referenced by clang::TokenLexer::Lex().

## ◆ isAtStartOfLine()

bool clang::Token::isAtStartOfLine | ( |  | ) | const
| inline

isAtStartOfLine - Return true if this token is at the start of a line.

Definition at line 268 of file Token.h.

References getFlag(), and StartOfLine.

Referenced by buildMSAsmString(), clang::Preprocessor::CheckEndOfDirective(), clang::Lexer::ComputePreamble(), DetectEOL(), clang::Preprocessor::DumpToken(), findPlaceholderEnd(), clang::ModuleMapParser::getLocation(), getMissingDeclaratorIdLoc(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), IsBuiltInOrStandardCXX11Attribute(), IsCommonTypo(), isGCCAsmStatement(), clang::MacroInfo::isIdenticalTo(), isTrivialSingleTokenExpansion(), clang::TokenLexer::Lex(), clang::Preprocessor::LexHeaderName(), PrintPreprocessedTokens(), ReadOriginalFileName(), remapMacroPath(), clang::RewriteMacrosInInput(), clang::Parser::SkipMalformedDecl(), clang::MacroArgs::StringifyArgument(), clang::html::SyntaxHighlight(), and updateConsecutiveMacroArgTokens().

## ◆ isEditorPlaceholder()

bool clang::Token::isEditorPlaceholder | ( |  | ) | const
| inline

Returns true if this token is an editor placeholder.

Editor placeholders are produced by the code-completion engine and are represented as characters between '<#' and '#>' in the source code. The lexer uses identifier tokens to represent placeholders.

Definition at line 310 of file Token.h.

References getFlag(), and IsEditorPlaceholder.

Referenced by clang::Parser::ParseTopLevelDecl(), and clang::Parser::TryAnnotateTypeOrScopeToken().

## ◆ isExpandDisabled()

bool clang::Token::isExpandDisabled | ( |  | ) | const
| inline

Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2.

Definition at line 276 of file Token.h.

References DisableExpand, getFlag(), getObjCKeywordID(), and isObjCAtKeyword().

Referenced by clang::Preprocessor::DumpToken(), and clang::Preprocessor::HandleIdentifier().

## ◆ isLiteral()

bool clang::Token::isLiteral | ( |  | ) | const
| inline

Return true if this is a "literal", like a numeric constant, string, etc.

Definition at line 115 of file Token.h.

References getKind(), and clang::tok::isLiteral().

Referenced by clang::Preprocessor::CreateString(), GetFirstChar(), getIdentifierInfo(), getLiteralData(), clang::Lexer::getSpelling(), isWideStringLiteralFromMacro(), clang::format::mustBeJSIdentOrValue(), setLiteralData(), and UseNSOptionsMacro().

## ◆ isNot()

bool clang::Token::isNot | ( | tok::TokenKind | K | ) | const
| inline

Definition at line 98 of file Token.h.

Referenced by addConstevalToLambdaDeclSpecifier(), clang::MacroArgs::ArgNeedsPreexpansion(), attributeParsedArgsUnevaluated(), clang::TokenConcatenation::AvoidConcat(), CheckArrayDesignatorSyntax(), clang::Preprocessor::CheckEndOfDirective(), clang::Preprocessor::CheckMacroName(), clang::Preprocessor::checkModuleIsAvailable(), DetectEOL(), diagnoseDynamicExceptionSpecification(), clang::Preprocessor::DiscardUntilEndOfDirective(), DoPrintMacros(), EvaluateDefined(), EvaluateDirectiveSubExpr(), EvaluateFeatureLikeBuiltinMacro(), EvaluateHasIncludeCommon(), EvaluateValue(), clang::PrintDependencyDirectivesSourceMinimizerAction::ExecuteAction(), clang::DumpRawTokensAction::ExecuteAction(), clang::DumpTokensAction::ExecuteAction(), clang::PreprocessOnlyAction::ExecuteAction(), findDirectives(), FindLocsWithCommonFileID(), clang::Preprocessor::FinishLexStringLiteral(), GenerateNewArgTokens(), clang::MacroArgs::getArgLength(), getIdentifierInfo(), GetLineValue(), getMissingDeclaratorIdLoc(), clang::Preprocessor::getSpelling(), clang::Preprocessor::HandleMicrosoftCommentPaste(), clang::Preprocessor::HandlePragmaDependency(), clang::Preprocessor::HandlePragmaHdrstop(), clang::Preprocessor::HandlePragmaIncludeAlias(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::Preprocessor::HandlePragmaPoison(), clang::html::HighlightMacros(), clang::TokenLexer::Init(), IsBuiltInOrStandardCXX11Attribute(), isPipeDeclerator(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexHeaderName(), LexModuleName(), clang::Preprocessor::LexOnOffSwitch(), LexRawTokensFromMainFile(), clang::Parser::ConditionDeclarationOrInitStatementState::markNotExpression(), outputPrintable(), ParseAlignPragma(), clang::Parser::ParseConstraintLogicalOrExpression(), parseDeclareSimdClauses(), parseDeviceSelector(), parseImplementationSelector(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), parseOpenMPSimpleClause(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PragmaLoopHintString(), ReadLineMarkerFlags(), ReadOriginalFileName(), remapMacroPath(), clang::RewriteIncludesInInput(), clang::RewriteMacrosInInput(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), clang::html::SyntaxHighlight(), takeDeclAttributes(), clang::format::tokenCanStartNewLine(), clang::TokenRewriter::TokenRewriter(), clang::Parser::TryConsumeToken(), TypeTraitArity(), and VersionNumberSeparator().

## ◆ isObjCAtKeyword()

bool Token::isObjCAtKeyword | ( | tok::ObjCKeywordKind | objcKey | ) | const

Return true if we have an ObjC keyword identifier.

isObjCAtKeyword - Return true if we have an ObjC keyword identifier.

Definition at line 57 of file Lexer.cpp.

References getIdentifierInfo(), and isAnnotation().

Referenced by isExpandDisabled(), clang::format::FormatToken::isObjCAtKeyword(), clang::Parser::ParseTopLevelDecl(), clang::Parser::SkipMalformedDecl(), and takeDeclAttributes().

## ◆ isOneOf() [1/2]

bool clang::Token::isOneOf | ( | tok::TokenKind | K1,
|  | tok::TokenKind | K2
| ) |  | const
| inline

Definition at line 99 of file Token.h.

References is().

Referenced by addConstevalToLambdaDeclSpecifier(), clang::TokenConcatenation::AvoidConcat(), EvaluateFeatureLikeBuiltinMacro(), GenerateNewArgTokens(), getMissingDeclaratorIdLoc(), clang::Parser::getNullabilityKeyword(), clang::TokenLexer::Init(), IsBuiltInOrStandardCXX11Attribute(), isConfigurationPattern(), isEndOfTemplateArgument(), isMapType(), isOneOf(), isValidAfterIdentifierInDeclarator(), clang::Preprocessor::LexHeaderName(), clang::Parser::ParseConstraintLogicalAndExpression(), clang::Parser::ParseConstraintLogicalOrExpression(), ParseLoopHintValue(), clang::Parser::ParseUnqualifiedId(), remapMacroPath(), clang::Parser::SkipMalformedDecl(), and takeDeclAttributes().

## ◆ isOneOf() [2/2]

template<typename... Ts>

bool clang::Token::isOneOf | ( | tok::TokenKind | K1,
|  | tok::TokenKind | K2,
|  | Ts... | Ks
| ) |  | const
| inline

Definition at line 103 of file Token.h.

References is(), and isOneOf().

## ◆ needsCleaning()

bool clang::Token::needsCleaning | ( |  | ) | const
| inline

Return true if this token has trigraphs or escaped newlines in it.

Definition at line 285 of file Token.h.

References getFlag(), and NeedsCleaning.

Referenced by clang::Preprocessor::CheckEndOfDirective(), clang::Lexer::ComputePreamble(), clang::Preprocessor::DumpToken(), GetFirstChar(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), getSpellingSlow(), IsStringPrefix(), and clang::Preprocessor::LookUpIdentifierInfo().

## ◆ setAnnotationEndLoc()

void clang::Token::setAnnotationEndLoc | ( | SourceLocation | L | ) |
| inline

Definition at line 144 of file Token.h.

References clang::SourceLocation::getRawEncoding(), and isAnnotation().

Referenced by clang::Preprocessor::EnterAnnotationToken(), clang::Preprocessor::HandleEndOfFile(), ParseAlignPragma(), ParseLoopHintValue(), setAnnotationRange(), clang::Parser::TryAnnotateTypeOrScopeToken(), and clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec().

## ◆ setAnnotationRange()

void clang::Token::setAnnotationRange | ( | SourceRange | R | ) |
| inline

Definition at line 163 of file Token.h.

References clang::SourceRange::getBegin(), clang::SourceRange::getEnd(), setAnnotationEndLoc(), and setLocation().

Referenced by clang::Preprocessor::LexOnOffSwitch(), and clang::Parser::ParseTopLevelDecl().

## ◆ setAnnotationValue()

void clang::Token::setAnnotationValue | ( | void * | val | ) |
| inline

Definition at line 230 of file Token.h.

References isAnnotation().

Referenced by clang::Preprocessor::EnterAnnotationToken(), clang::Parser::getTypeAnnotation(), clang::Preprocessor::HandleEndOfFile(), clang::Preprocessor::LexOnOffSwitch(), ParseAlignPragma(), ParseLoopHintValue(), and clang::Parser::ParseTopLevelDecl().

## ◆ setEofData()

void clang::Token::setEofData | ( | const void * | D | ) |
| inline

Definition at line 196 of file Token.h.

References clang::comments::tok::eof, and is().

Referenced by TypeTraitArity().

## ◆ setFlag()

void clang::Token::setFlag | ( | TokenFlags | Flag | ) |
| inline

Set the specified flag.

Definition at line 236 of file Token.h.

Referenced by clang::Preprocessor::Backtrack(), clang::Lexer::findLocationAfterToken(), findPlaceholderEnd(), GenerateNewArgTokens(), clang::Preprocessor::HandleIdentifier(), clang::TokenLexer::Init(), isEndOfBlockCommentWithEscapedNewLine(), isGCCAsmStatement(), isTrivialSingleTokenExpansion(), clang::TokenLexer::Lex(), maybeDiagnoseUTF8Homoglyph(), clang::ASTReader::ReadToken(), and setFlagValue().

## ◆ setFlagValue()

void clang::Token::setFlagValue | ( | TokenFlags | Flag,
|  | bool | Val
| ) |  |
| inline

Set a flag to either true or false.

Definition at line 259 of file Token.h.

References clearFlag(), and setFlag().

Referenced by isTrivialSingleTokenExpansion(), clang::TokenLexer::Lex(), clang::Preprocessor::LexHeaderName(), and remapMacroPath().

## ◆ setIdentifierInfo()

void clang::Token::setIdentifierInfo | ( | IdentifierInfo * | II | ) |
| inline

Definition at line 188 of file Token.h.

Referenced by EvaluateHasIncludeCommon(), findPlaceholderEnd(), clang::Preprocessor::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LookUpIdentifierInfo(), clang::ASTReader::ReadToken(), and remapMacroPath().

## ◆ setKind()

void clang::Token::setKind | ( | tok::TokenKind | K | ) |
| inline

Definition at line 93 of file Token.h.

Referenced by clang::TokenRewriter::AddTokenBefore(), attributeParsedArgsUnevaluated(), clang::Preprocessor::CheckMacroName(), clang::Preprocessor::EnterAnnotationToken(), EvaluateFeatureLikeBuiltinMacro(), findDirectives(), FixDigraph(), GenerateNewArgTokens(), clang::Lexer::getCharAndSizeNoWarn(), clang::Preprocessor::HandleIdentifier(), clang::html::HighlightMacros(), IsBuiltInOrStandardCXX11Attribute(), isConfigurationPattern(), clang::format::FormatTokenLexer::lex(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexHeaderName(), clang::Preprocessor::LexOnOffSwitch(), clang::Preprocessor::LookUpIdentifierInfo(), maybeDiagnoseUTF8Homoglyph(), ParseAlignPragma(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), clang::Parser::Parser(), clang::Parser::ParseTopLevelDecl(), clang::ASTReader::ReadToken(), remapMacroPath(), clang::MacroArgs::StringifyArgument(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), TypeTraitArity(), and clang::VAOptExpansionContext::VAOptExpansionContext().

## ◆ setLength()

void clang::Token::setLength | ( | unsigned | Len | ) |
| inline

Definition at line 135 of file Token.h.

References isAnnotation().

Referenced by clang::TokenRewriter::AddTokenBefore(), clang::Preprocessor::CreateString(), FixDigraph(), GenerateNewArgTokens(), clang::Lexer::getCharAndSizeNoWarn(), IsBuiltInOrStandardCXX11Attribute(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::ASTReader::ReadToken(), and remapMacroPath().

## ◆ setLiteralData()

void clang::Token::setLiteralData | ( | const char * | Ptr | ) |
| inline

Definition at line 221 of file Token.h.

References isLiteral().

Referenced by clang::Preprocessor::CreateString(), and IsBuiltInOrStandardCXX11Attribute().

## ◆ setLocation()

void clang::Token::setLocation | ( | SourceLocation | L | ) |
| inline

Definition at line 134 of file Token.h.

References clang::SourceLocation::getRawEncoding().

Referenced by clang::TokenRewriter::AddTokenBefore(), clang::Preprocessor::CreateString(), clang::Preprocessor::EnterAnnotationToken(), FindLocsWithCommonFileID(), FixDigraph(), GenerateNewArgTokens(), clang::Lexer::getCharAndSizeNoWarn(), clang::Preprocessor::HandleEndOfFile(), IsBuiltInOrStandardCXX11Attribute(), isTrivialSingleTokenExpansion(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), ParseAlignPragma(), ParseLoopHintValue(), clang::ASTReader::ReadToken(), remapMacroPath(), setAnnotationRange(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), TypeTraitArity(), and updateConsecutiveMacroArgTokens().

## ◆ setRawIdentifierData()

void clang::Token::setRawIdentifierData | ( | const char * | Ptr | ) |
| inline

Definition at line 209 of file Token.h.

References is().

Referenced by clang::Preprocessor::CreateString(), findPlaceholderEnd(), clang::TokenLexer::Lex(), and maybeDiagnoseUTF8Homoglyph().

## ◆ startToken()

void clang::Token::startToken | ( |  | ) |
| inline

Reset all flags to cleared.

Definition at line 171 of file Token.h.

References clang::SourceLocation::getRawEncoding().

Referenced by clang::TokenRewriter::AddTokenBefore(), clang::Preprocessor::checkModuleIsAvailable(), ComputeDATE_TIME(), clang::Preprocessor::EnterAnnotationToken(), FindLocsWithCommonFileID(), findPlaceholderEnd(), GenerateNewArgTokens(), clang::Preprocessor::HandleEndOfFile(), clang::Preprocessor::HandlePragmaHdrstop(), IsBuiltInOrStandardCXX11Attribute(), clang::TokenLexer::Lex(), clang::Preprocessor::LexAfterModuleImport(), clang::Preprocessor::LexHeaderName(), clang::Preprocessor::LexOnOffSwitch(), outputPrintable(), ParseAlignPragma(), ParseLoopHintValue(), clang::Parser::ParseMSAsmIdentifier(), clang::Preprocessor::ParsePragmaPushOrPopMacro(), clang::Parser::Parser(), PrintPreprocessedTokens(), clang::Lexer::ReadToEndOfLine(), clang::ASTReader::ReadToken(), clang::arcmt::trans::MigrationContext::rewritePropertyAttribute(), clang::MacroArgs::StringifyArgument(), TypeTraitArity(), and clang::VAOptExpansionContext::VAOptExpansionContext().

## ◆ stringifiedInMacro()

bool clang::Token::stringifiedInMacro | ( |  | ) | const
| inline

Returns true if this token is formed by macro by stringizing or charizing operator.

Definition at line 300 of file Token.h.

References getFlag(), and StringifiedInMacro.

Referenced by isWideStringLiteralFromMacro().

The documentation for this class was generated from the following files:

- include/clang/Lex/Token.h
- lib/Lex/Lexer.cpp