Fix compiler crash rewriting return function pointer type. #483
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The compiler hit an assert while rewrting types with bounds-safe interfaces to make them checked types. This problem was found by trying to call the signal functionin a checked scope. The problem occurred when a function return type was a function pointer type with a bounds-safe interface.
The construction of new type location information asserted about a mismatch in the expected size of the type location buffer.
The problem was that we were rewriting parameter and return types and then modifying types using interop type information. We need to do this in the opposite order to construct new type location information properly.
Detailed explanation:
Clang has a binary serialization of type locations, where it expects source locations for types that are components of a type to be serialized as part of the type location information for the enclosing type. For types with type location information, the representation divides data into "local data" - data specific to the type and "child data" (type location information for the child). The local data is followed by the child data when laying out data. The local data may be variable length. The code is in include\clang\ast\TypeLoc.h. which has poor commenting and relies on obscure template meta-programming. See getInnerTypeLoc() for the messy details of how they decided to do this.
During tree transforms, if types are rewritten, the transform maintains this serialization. That takes some work because the tree transform typically operates bottom-up: transform child nodes of an AST node and then transform the AST node. This means that the type location information for a child type node is rewritten before the parent type node information (which is variable length and may change the position of the child node's type location). The solution is to use a type location builder: the data for a child node is accumulated into the type location builder. For a parent node, the type location build buffer is expanded if necessary. The data for the child node is moved to the back of the buffer.
The problem was that the type location information for a child type was out-of-sync with what was expected, if we replaced the child type with a new type. based on bounds-safe information. The specific child node in question was the return type. The solution is to choose return type, generate the type location information by recursively transforming the return type, and then filling in the data for the parent (the function prototype).
Testing:
return pointer types with bounds-safe interfaces.