This adds a new semantic token type, PreprocessorMacro.
Calls to preprocessor macros will now be highlighted when semantic
highlighting is enabled in Hack Studio.
This helps make the overall codebase consistent. `class_name()` in
`Kernel` is always `StringView`, but not elsewhere.
Additionally, this results in the `strlen` (which needs to be done
when printing or other operations) always being computed at
compile-time.
The type of a function parameter can be null if we failed to parse it.
In such a case, calling to_string() on a FunctionType node used to cause
a null dereference.
This caused the language server to crash when processing
AK/StdLibExtraDetails.h
Previously we didn't set the end position for the return type node of
function FunctionType nodes.
This caused a VERIFY failure crash when dumping an AST that contains
such nodes.
Previously, the names of declarations where stored as a simple
StringView.
Because of that, we couldn't parse out-of-line function definitions,
which have qualified names.
For example, we couldn't parse the following snippet:
```
void MyClass::foo(){}
```
To fix this, we now store the name of a declaration with a
ASTNode::Name node, which represents a qualified named.
The SemanticSyntaxHighlighter uses TokenInfo results from the
language server to provide 'semantic' syntax highlighting, which
provides more fin-grained text spans results.
For example, the SemanticSyntaxHighlighter can color function calls,
member fields references and user-defined types in different colors.
With the simple lexer-only syntax highlighter, all of these tokens were
given the same text highlighting span type.
Since we have to provide immediate highlighting feedback to the user
after each edit and before we get the result for the language server,
we use a heuristic which computes the diff between the current tokens
and the last known tokens with compete semantic information
(We use LibDiff for this).
This heuristic is not very performant, and starts feeling sluggish with
bigger (~200 LOC) files.
A possible future improvement would be only computing the diff for
tokens in text ranges that have changes since the last commit.
Previously, the end position of the Type and Name nodes was incorrect.
It was incorrectly set to the start position of the next unconsumed
token.
This commit adds a previous_token_end() method and uses it to correctly
get the end position for these node types.
Previously, the parent of a parameter's Type node was incorrectly set
to the parent of the Parameter node.
We now set the parent of the parameter's Type node to the Parameter
node itself.
Previously, Declaration::is_member() was just a stub that always
returned false.
It now works by checking whether the parent ASTNode is a declaration
of a struct or a class type.
This was causing some macro redefinition errors after the headers ended
up in the same file through some includes. The simple fix is to undefine
the macro after use.
Before this commit all consume_until overloads aside from the Predicate
one would consume (and ignore) the stop char/string, while the
Predicate overload would not, in order to keep behaviour consistent,
the other overloads no longer consume the stop char/string as well.
When we run the Preprocessor from the CppComprehensionEngine of
the language server, we don't want the preprocessor to crash if it
encounters an invalid preprocessor statement (for example, an #endif
statement without an accompanying previous #if statement).
To achieve this, this commit adds an "ignore_invalid_statements" flag
to the preprocessor which is set by the CppComprehensionEngine.
Fixes#11064.
SonarCloud flagged this 'Identical sub-expressions on both sides of
operator "||"'. When looking at the git history it looks like it was
just a copy / paste mistake that happened when Token::Type::Arrow
support was added.
Previously, the preprocessor first split the source into lines, and then
processed and lexed each line separately.
This patch makes the preprocessor first lex the source, and then do the
processing on the tokenized representation.
This generally simplifies the code, and also fixes an issue we
previously had with multiline comments (we did not recognize them
correctly when processing each line separately).
Similarly to the LibCpp parser regression tests, these tests run the
preprocessor on the .cpp test files under
Userland/LibCpp/Tests/preprocessor, and compare the output with existing
.txt ground truth files.
The preprocessor now understands when a function-like macro is defined,
and can also parse calls to such macros.
The actual evaluation of function-like macros will be done in a
separate commit.
When the preprocessor encounters an #include statement it now adds
the preprocessor definitions that exist in the included header to its
own set of definitions.
We previously only aggregated the definitions from headers after
processing the source, which was less correct. (For example, there
could be an #ifdef that depends on a definition from another header).
We now call Preprocessor::process_and_lex() and pass the result to the
parser.
Doing the lexing in the preprocessor will allow us to maintain the
original position information of tokens after substituting definitions.
