Debugging C++ Coroutines¶
Introduction¶
For performance and other architectural reasons, the C++ Coroutines feature in the Clang compiler is implemented in two parts of the compiler. Semantic analysis is performed in Clang, and Coroutine construction and optimization takes place in the LLVM middle-end.
However, this design forces us to generate insufficient debugging information. Typically, the compiler generates debug information in the Clang frontend, as debug information is highly language specific. However, this is not possible for Coroutine frames because the frames are constructed in the LLVM middle-end.
To mitigate this problem, the LLVM middle end attempts to generate some debug information, which is unfortunately incomplete, since much of the language specific information is missing in the middle end.
This document describes how to use this debug information to better debug coroutines.
Terminology¶
Due to the recent nature of C++20 Coroutines, the terminology used to describe the concepts of Coroutines is not settled. This section defines a common, understandable terminology to be used consistently throughout this document.
coroutine type¶
A coroutine function is any function that contains any of the Coroutine Keywords co_await, co_yield, or co_return. A coroutine type is a possible return type of one of these coroutine functions. Task and Generator are commonly referred to coroutine types.
coroutine¶
By technical definition, a coroutine is a suspendable function. However, programmers typically use coroutine to refer to an individual instance. For example:
std::vector<Task> Coros; // Task is a coroutine type.
for (int i = 0; i < 3; i++)
Coros.push_back(CoroTask()); // CoroTask is a coroutine function, which
// would return a coroutine type 'Task'.
In practice, we typically say “Coros contains 3 coroutines” in the above example, though this is not strictly correct. More technically, this should say “Coros contains 3 coroutine instances” or “Coros contains 3 coroutine objects.”
In this document, we follow the common practice of using coroutine to refer to an individual coroutine instance, since the terms coroutine instance and coroutine object aren’t sufficiently defined in this case.
coroutine frame¶
The C++ Standard uses coroutine state to describe the allocated storage. In the compiler, we use coroutine frame to describe the generated data structure that contains the necessary information.
The structure of coroutine frames¶
The structure of coroutine frames is defined as:
struct {
void (*__r)(); // function pointer to the `resume` function
void (*__d)(); // function pointer to the `destroy` function
promise_type; // the corresponding `promise_type`
... // Any other needed information
}
In the debugger, the function’s name is obtainable from the address of the function. And the name of resume function is equal to the name of the coroutine function. So the name of the coroutine is obtainable once the address of the coroutine is known.
Print promise_type¶
Every coroutine has a promise_type, which defines the behavior for the corresponding coroutine. In other words, if two coroutines have the same promise_type, they should behave in the same way. To print a promise_type in a debugger when stopped at a breakpoint inside a coroutine, printing the promise_type can be done by:
print __promise
It is also possible to print the promise_type of a coroutine from the address of the coroutine frame. For example, if the address of a coroutine frame is 0x416eb0, and the type of the promise_type is task::promise_type, printing the promise_type can be done by:
print (task::promise_type)*(0x416eb0+0x10)
This is possible because the promise_type is guaranteed by the ABI to be at a 16 bit offset from the coroutine frame.
Note that there is also an ABI independent method:
print std::coroutine_handle<task::promise_type>::from_address((void*)0x416eb0).promise()
The functions from_address(void*) and promise() are often small enough to be removed during optimization, so this method may not be possible.
Print coroutine frames¶
LLVM generates the debug information for the coroutine frame in the LLVM middle end, which permits printing of the coroutine frame in the debugger. Much like the promise_type, when stopped at a breakpoint inside a coroutine we can print the coroutine frame by:
print __coro_frame
Just as printing the promise_type is possible from the coroutine address, printing the details of the coroutine frame from an address is also possible:
(gdb) # Get the address of coroutine frame
(gdb) print/x *0x418eb0
$1 = 0x4019e0
(gdb) # Get the linkage name for the coroutine
(gdb) x 0x4019e0
0x4019e0 <_ZL9coro_taski>: 0xe5894855
(gdb) # The coroutine frame type is 'linkage_name.coro_frame_ty'
(gdb) print (_ZL9coro_taski.coro_frame_ty)*(0x418eb0)
$2 = {__resume_fn = 0x4019e0 <coro_task(int)>, __destroy_fn = 0x402000 <coro_task(int)>, __promise = {...}, ...}
The above is possible because:
(1) The name of the debug type of the coroutine frame is the linkage_name, plus the .coro_frame_ty suffix because each coroutine function shares the same coroutine type.
(2) The coroutine function name is accessible from the address of the coroutine frame.
The above commands can be simplified by placing them in debug scripts.
Examples to print coroutine frames¶
The print examples below use the following definition:
#include <coroutine>
#include <iostream>
struct task{
struct promise_type {
task get_return_object() { return std::coroutine_handle<promise_type>::from_promise(*this); }
std::suspend_always initial_suspend() { return {}; }
std::suspend_always final_suspend() noexcept { return {}; }
void return_void() noexcept {}
void unhandled_exception() noexcept {}
int count = 0;
};
void resume() noexcept {
handle.resume();
}
task(std::coroutine_handle<promise_type> hdl) : handle(hdl) {}
~task() {
if (handle)
handle.destroy();
}
std::coroutine_handle<> handle;
};
class await_counter : public std::suspend_always {
public:
template<class PromiseType>
void await_suspend(std::coroutine_handle<PromiseType> handle) noexcept {
handle.promise().count++;
}
};
static task coro_task(int v) {
int a = v;
co_await await_counter{};
a++;
std::cout << a << "\n";
a++;
std::cout << a << "\n";
a++;
std::cout << a << "\n";
co_await await_counter{};
a++;
std::cout << a << "\n";
a++;
std::cout << a << "\n";
}
int main() {
task t = coro_task(43);
t.resume();
t.resume();
t.resume();
return 0;
}
In debug mode (O0 + g), the printing result would be:
{__resume_fn = 0x4019e0 <coro_task(int)>, __destroy_fn = 0x402000 <coro_task(int)>, __promise = {count = 1}, v = 43, a = 45, __coro_index = 1 '001', struct_std__suspend_always_0 = {__int_8 = 0 '000'}, class_await_counter_1 = {__int_8 = 0 '000'}, class_await_counter_2 = {__int_8 = 0 '000'}, struct_std__suspend_always_3 = {__int_8 = 0 '000'}}
In the above, the values of v and a are clearly expressed, as are the temporary values for await_counter (class_await_counter_1 and class_await_counter_2) and std::suspend_always ( struct_std__suspend_always_0 and struct_std__suspend_always_3). The index of the current suspension point of the coroutine is emitted as __coro_index. In the above example, the __coro_index value of 1 means the coroutine stopped at the second suspend point (Note that __coro_index is zero indexed) which is the first co_await await_counter{}; in coro_task. Note that the first initial suspend point is the compiler generated co_await promise_type::initial_suspend().
However, when optimizations are enabled, the printed result changes drastically:
{__resume_fn = 0x401280 <coro_task(int)>, __destroy_fn = 0x401390 <coro_task(int)>, __promise = {count = 1}, __int_32_0 = 43, __coro_index = 1 '001'}
Unused values are optimized out, as well as the name of the local variable a. The only information remained is the value of a 32 bit integer. In this simple case, it seems to be pretty clear that __int_32_0 represents a. However, it is not true.
An important note with optimization is that the value of a variable may not properly express the intended value in the source code. For example:
static task coro_task(int v) {
int a = v;
co_await await_counter{};
a++; // __int_32_0 is 43 here
std::cout << a << "\n";
a++; // __int_32_0 is still 43 here
std::cout << a << "\n";
a++; // __int_32_0 is still 43 here!
std::cout << a << "\n";
co_await await_counter{};
a++; // __int_32_0 is still 43 here!!
std::cout << a << "\n";
a++; // Why is __int_32_0 still 43 here?
std::cout << a << "\n";
}
When debugging step-by-step, the value of __int_32_0 seemingly does not change, despite being frequently incremented, and instead is always 43. While this might be surprising, this is a result of the optimizer recognizing that it can eliminate most of the load/store operations. The above code gets optimized to the equivalent of:
static task coro_task(int v) {
store v to __int_32_0 in the frame
co_await await_counter{};
a = load __int_32_0
std::cout << a+1 << "\n";
std::cout << a+2 << "\n";
std::cout << a+3 << "\n";
co_await await_counter{};
a = load __int_32_0
std::cout << a+4 << "\n";
std::cout << a+5 << "\n";
}
It should now be obvious why the value of __int_32_0 remains unchanged throughout the function. It is important to recognize that __int_32_0 does not directly correspond to a, but is instead a variable generated to assist the compiler in code generation. The variables in an optimized coroutine frame should not be thought of as directly representing the variables in the C++ source.
Get the suspended points¶
An important requirement for debugging coroutines is to understand suspended points, which are where the coroutine is currently suspended and awaiting.
For simple cases like the above, inspecting the value of the __coro_index variable in the coroutine frame works well.
However, it is not quite so simple in really complex situations. In these cases, it is necessary to use the coroutine libraries to insert the line-number.
For example:
// For all the promise_type we want:
class promise_type {
...
+ unsigned line_number = 0xffffffff;
};
#include <source_location>
// For all the awaiter types we need:
class awaiter {
...
template <typename Promise>
void await_suspend(std::coroutine_handle<Promise> handle,
std::source_location sl = std::source_location::current()) {
...
handle.promise().line_number = sl.line();
}
};
In this case, we use std::source_location to store the line number of the await inside the promise_type. Since we can locate the coroutine function from the address of the coroutine, we can identify suspended points this way as well.
The downside here is that this comes at the price of additional runtime cost. This is consistent with the C++ philosophy of “Pay for what you use”.
Get the asynchronous stack¶
Another important requirement to debug a coroutine is to print the asynchronous stack to identify the asynchronous caller of the coroutine. As many implementations of coroutine types store std::coroutine_handle<> continuation in the promise type, identifying the caller should be trivial. The continuation is typically the awaiting coroutine for the current coroutine. That is, the asynchronous parent.
Since the promise_type is obtainable from the address of a coroutine and contains the corresponding continuation (which itself is a coroutine with a promise_type), it should be trivial to print the entire asynchronous stack.
This logic should be quite easily captured in a debugger script.
Get the living coroutines¶
Another useful task when debugging coroutines is to enumerate the list of living coroutines, which is often done with threads. While technically possible, this task is not recommended in production code as it is costly at runtime. One such solution is to store the list of currently running coroutines in a collection:
inline std::unordered_set<void*> lived_coroutines;
// For all promise_type we want to record
class promise_type {
public:
promise_type() {
// Note to avoid data races
lived_coroutines.insert(std::coroutine_handle<promise_type>::from_promise(*this).address());
}
~promise_type() {
// Note to avoid data races
lived_coroutines.erase(std::coroutine_handle<promise_type>::from_promise(*this).address());
}
};
In the above code snippet, we save the address of every lived coroutine in the lived_coroutines unordered_set. As before, once we know the address of the coroutine we can derive the function, promise_type, and other members of the frame. Thus, we could print the list of lived coroutines from that collection.
Please note that the above is expensive from a storage perspective, and requires some level of locking (not pictured) on the collection to prevent data races.