clang 8.0.0

Classes | Typedefs | Enumerations | Functions
Token extraction and manipulation
libclang: C Interface to Clang

The routines in this group provide access to the tokens within a translation unit, along with a semantic mapping of those tokens to their corresponding cursors. More...

Collaboration diagram for Token extraction and manipulation:

## Classes
struct | CXToken
| Describes a single preprocessing token. More...

## Typedefs
typedef enum CXTokenKind | CXTokenKind
| Describes a kind of token. More...

## Enumerations
enum | CXTokenKind { CXToken_Punctuation, CXToken_Keyword, CXToken_Identifier, CXToken_Literal, CXToken_Comment }
| Describes a kind of token. More...

## Functions
CINDEX_LINKAGE CXToken * | clang_getToken (CXTranslationUnit TU, CXSourceLocation Location)
| Get the raw lexical token starting with the given location. More...
CINDEX_LINKAGE CXTokenKind | clang_getTokenKind (CXToken)
| Determine the kind of the given token. More...
CINDEX_LINKAGE CXString | clang_getTokenSpelling (CXTranslationUnit, CXToken)
| Determine the spelling of the given token. More...
CINDEX_LINKAGE CXSourceLocation | clang_getTokenLocation (CXTranslationUnit, CXToken)
| Retrieve the source location of the given token. More...
CINDEX_LINKAGE CXSourceRange | clang_getTokenExtent (CXTranslationUnit, CXToken)
| Retrieve a source range that covers the given token. More...
CINDEX_LINKAGE void | clang_tokenize (CXTranslationUnit TU, CXSourceRange Range, CXToken **Tokens, unsigned *NumTokens)
| Tokenize the source code described by the given range into raw lexical tokens. More...
CINDEX_LINKAGE void | clang_annotateTokens (CXTranslationUnit TU, CXToken *Tokens, unsigned NumTokens, CXCursor *Cursors)
| Annotate the given set of tokens by providing cursors for each token that can be mapped to a specific entity within the abstract syntax tree. More...
CINDEX_LINKAGE void | clang_disposeTokens (CXTranslationUnit TU, CXToken *Tokens, unsigned NumTokens)
| Free the given set of tokens. More...

## Detailed Description

The routines in this group provide access to the tokens within a translation unit, along with a semantic mapping of those tokens to their corresponding cursors.

## Typedef Documentation

## ◆ CXTokenKind

typedef enum CXTokenKind CXTokenKind

Describes a kind of token.

## Enumeration Type Documentation

## ◆ CXTokenKind

enum CXTokenKind

Describes a kind of token.

Enumerator
CXToken_Punctuation | A token that contains some kind of punctuation.
CXToken_Keyword | A language keyword.
CXToken_Identifier | An identifier (that is not a keyword).
CXToken_Literal | A numeric, string, or character literal.
CXToken_Comment | A comment.

Definition at line 4850 of file Index.h.

## Function Documentation

## ◆ clang_annotateTokens()

CINDEX_LINKAGE void clang_annotateTokens | ( | CXTranslationUnit | TU,
|  | CXToken * | Tokens,
|  | unsigned | NumTokens,
|  | CXCursor * | Cursors
| ) |  |

Annotate the given set of tokens by providing cursors for each token that can be mapped to a specific entity within the abstract syntax tree.

This token-annotation routine is equivalent to invoking clang_getCursor() for the source locations of each of the tokens. The cursors provided are filtered, so that only those cursors that have a direct correspondence to the token are accepted. For example, given a function call `f(x)`, clang_getCursor() would provide the following cursors:

- when the cursor is over the 'f', a DeclRefExpr cursor referring to 'f'.
- when the cursor is over the '(' or the ')', a CallExpr referring to 'f'.
- when the cursor is over the 'x', a DeclRefExpr cursor referring to 'x'.

Only the first and last of these cursors will occur within the annotate, since the tokens "f" and "x' directly refer to a function and a variable, respectively, but the parentheses are just a small part of the full syntax of the function call expression, which is not provided as an annotation.

Parameters

  TU | the translation unit that owns the given tokens.
  Tokens | the set of tokens to annotate.
  NumTokens | the number of tokens in `Tokens`.
  Cursors | an array of `NumTokens` cursors, whose contents will be replaced with the cursors corresponding to each token.

## ◆ clang_disposeTokens()

CINDEX_LINKAGE void clang_disposeTokens | ( | CXTranslationUnit | TU,
|  | CXToken * | Tokens,
|  | unsigned | NumTokens
| ) |  |

Free the given set of tokens.

## ◆ clang_getToken()

CINDEX_LINKAGE CXToken* clang_getToken | ( | CXTranslationUnit | TU,
|  | CXSourceLocation | Location
| ) |  |

Get the raw lexical token starting with the given location.

Parameters

  TU | the translation unit whose text is being tokenized.
  Location | the source location with which the token starts.

Returns
  The token starting with the given location or NULL if no such token exist. The returned pointer must be freed with clang_disposeTokens before the translation unit is destroyed.

## ◆ clang_getTokenExtent()

CINDEX_LINKAGE CXSourceRange clang_getTokenExtent | ( | CXTranslationUnit | ,
|  | CXToken |
| ) |  |

Retrieve a source range that covers the given token.

## ◆ clang_getTokenKind()

CINDEX_LINKAGE CXTokenKind clang_getTokenKind | ( | CXToken |  | ) |

Determine the kind of the given token.

## ◆ clang_getTokenLocation()

CINDEX_LINKAGE CXSourceLocation clang_getTokenLocation | ( | CXTranslationUnit | ,
|  | CXToken |
| ) |  |

Retrieve the source location of the given token.

## ◆ clang_getTokenSpelling()

CINDEX_LINKAGE CXString clang_getTokenSpelling | ( | CXTranslationUnit | ,
|  | CXToken |
| ) |  |

Determine the spelling of the given token.

The spelling of a token is the textual representation of that token, e.g., the text of an identifier or keyword.

## ◆ clang_tokenize()

CINDEX_LINKAGE void clang_tokenize | ( | CXTranslationUnit | TU,
|  | CXSourceRange | Range,
|  | CXToken ** | Tokens,
|  | unsigned * | NumTokens
| ) |  |

Tokenize the source code described by the given range into raw lexical tokens.

Parameters

  TU | the translation unit whose text is being tokenized.
  Range | the source range in which text should be tokenized. All of the tokens produced by tokenization will fall within this source range,
  Tokens | this pointer will be set to point to the array of tokens that occur within the given source range. The returned pointer must be freed with clang_disposeTokens() before the translation unit is destroyed.
  NumTokens | will be set to the number of tokens in the `*Tokens` array.