beenull/ladybird
1
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2024-11-22 07:30:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6
ladybird/Userland/Libraries/LibJS/Parser.cpp

4820 lines
207 KiB
C++
Raw Normal View History

LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
/*
Everywhere: Use s.unverwerth@serenityos.org :^)
2021-05-29 10:38:28 +00:00
* Copyright (c) 2020, Stephan Unverwerth <s.unverwerth@serenityos.org>
LibJS: Update a couple of outdated spec comments These are editorial changes in the ECMA-262 spec. See: - https://github.com/tc39/ecma262/commit/e080a7f - https://github.com/tc39/ecma262/commit/c5a9094 - https://github.com/tc39/ecma262/commit/5091520 - https://github.com/tc39/ecma262/commit/1c6564b - https://github.com/tc39/ecma262/commit/e06c80c
2022-04-30 20:22:52 +00:00
* Copyright (c) 2020-2022, Linus Groh <linusg@serenityos.org>
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
* Copyright (c) 2021-2022, David Tuin <davidot@serenityos.org>
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
* Copyright (c) 2021, Ali Mohammad Pur <mpfard@serenityos.org>
LibJS: Add support for await expressions
2021-11-09 20:52:21 +00:00
* Copyright (c) 2021, Idan Horowitz <idan.horowitz@serenityos.org>
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 08:24:48 +00:00
* SPDX-License-Identifier: BSD-2-Clause
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
*/
#include "Parser.h"
LibJS: Throw a syntax error when an identifier is a reserved word From the specification: It is a Syntax Error if StringValue of IdentifierName is the same String value as the StringValue of any ReservedWord except for yield or await. It is a Syntax Error if this phrase is contained in strict mode code and the StringValue of IdentifierName is: "implements", "interface", "let", "package", "private", "protected", "public", "static", or "yield".
2021-06-16 22:57:01 +00:00
#include <AK/Array.h>
LibJS: Use the new is_ascii_foo() helpers from AK These constexpr helpers generate nicer code than the LibC ctype.h variants, so let's make use of them. :^)
2021-06-13 08:47:09 +00:00
#include <AK/CharacterTypes.h>
LibJS: Make invalid RegExp flags a SyntaxError at parse time This patch changes the validation of RegExp flags (checking for invalid and duplicate values) from a SyntaxError at runtime to a SyntaxError at parse time - it's not something that's supposed to be catchable. As a nice side effect, this simplifies the RegExpObject constructor a bit, as it can no longer throw an exception and doesn't have to validate the flags itself.
2021-05-10 11:01:38 +00:00
#include <AK/HashTable.h>
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
#include <AK/ScopeGuard.h>
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
#include <AK/StdLibExtras.h>
Everywhere: Add missing <AK/TemporaryChange.h> includes Don't rely on HashTable.h pulling this in.
2020-10-15 18:46:52 +00:00
#include <AK/TemporaryChange.h>
LibJS: Parse RegExp literals at AST creation time, not execution time The spec requires that invalid RegExp literals must cause a Syntax Error before the JavaScript is executed. See: https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors This is explicitly tested in the RegExp/property-escapes test262 tests. For example, see unsupported-property-Line_Break.js: $DONOTEVALUATE(); /\p{Line_Break}/u; That RegExp literal is invalid because Line_Break is not a supported Unicode property. $DONOTEVALUATE() just throws an exception when it is executed. The test expects that this file will fail to be parsed. Note that RegExp patterns can still be parsed at execution time by way of "new RegExp(...)".
2021-07-29 14:34:37 +00:00
#include <LibJS/Runtime/RegExpObject.h>
#include <LibRegex/Regex.h>
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
namespace JS {
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
class ScopePusher {
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
// NOTE: We really only need ModuleTopLevel and NotModuleTopLevel as the only
// difference seems to be in https://tc39.es/ecma262/#sec-static-semantics-varscopeddeclarations
// where ModuleItemList only does the VarScopedDeclaration and not the
// TopLevelVarScopedDeclarations.
enum class ScopeLevel {
NotTopLevel,
ScriptTopLevel,
ModuleTopLevel,
FunctionTopLevel,
StaticInitTopLevel
};
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
private:
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
ScopePusher(Parser& parser, ScopeNode* node, ScopeLevel scope_level)
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
: m_parser(parser)
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
, m_scope_level(scope_level)
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
{
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
m_parent_scope = exchange(m_parser.m_state.current_scope_pusher, this);
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
VERIFY(node || (m_parent_scope && scope_level == ScopeLevel::NotTopLevel));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (!node)
m_node = m_parent_scope->m_node;
else
m_node = node;
VERIFY(m_node);
LibJS: Improve function hoisting across blocks The parser now keeps track of a scope chain so that it can hoist function declarations to the closest function scope.
2021-07-04 01:15:52 +00:00
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
if (!is_top_level())
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
m_top_level_scope = m_parent_scope->m_top_level_scope;
else
m_top_level_scope = this;
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
}
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
bool is_top_level()
{
return m_scope_level != ScopeLevel::NotTopLevel;
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
public:
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
static ScopePusher function_scope(Parser& parser, FunctionBody& function_body, Vector<FunctionParameter> const& parameters)
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
{
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
ScopePusher scope_pusher(parser, &function_body, ScopeLevel::FunctionTopLevel);
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
scope_pusher.m_function_parameters = parameters;
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
for (auto& parameter : parameters) {
parameter.binding.visit(
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
[&](DeprecatedFlyString const& name) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
scope_pusher.m_forbidden_lexical_names.set(name);
},
[&](NonnullRefPtr<BindingPattern> const& binding_pattern) {
binding_pattern->for_each_bound_name([&](auto const& name) {
scope_pusher.m_forbidden_lexical_names.set(name);
});
});
}
return scope_pusher;
}
LibJS: Improve function hoisting across blocks The parser now keeps track of a scope chain so that it can hoist function declarations to the closest function scope.
2021-07-04 01:15:52 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
static ScopePusher program_scope(Parser& parser, Program& program)
{
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
return ScopePusher(parser, &program, program.type() == Program::Type::Script ? ScopeLevel::ScriptTopLevel : ScopeLevel::ModuleTopLevel);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
}
LibJS: Don't hoist functions under certain circumstances When a lexical declaration with the same name as a function exists, the function is not hoisted (annex B).
2021-07-05 19:45:34 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
static ScopePusher block_scope(Parser& parser, ScopeNode& node)
{
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
return ScopePusher(parser, &node, ScopeLevel::NotTopLevel);
LibJS: Improve function hoisting across blocks The parser now keeps track of a scope chain so that it can hoist function declarations to the closest function scope.
2021-07-04 01:15:52 +00:00
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
static ScopePusher for_loop_scope(Parser& parser, RefPtr<ASTNode> const& init)
LibJS: Improve function hoisting across blocks The parser now keeps track of a scope chain so that it can hoist function declarations to the closest function scope.
2021-07-04 01:15:52 +00:00
{
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
ScopePusher scope_pusher(parser, nullptr, ScopeLevel::NotTopLevel);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (init && is<VariableDeclaration>(*init)) {
auto& variable_declaration = static_cast<VariableDeclaration const&>(*init);
if (variable_declaration.declaration_kind() != DeclarationKind::Var) {
variable_declaration.for_each_bound_name([&](auto const& name) {
scope_pusher.m_forbidden_var_names.set(name);
});
}
}
LibJS: Improve function hoisting across blocks The parser now keeps track of a scope chain so that it can hoist function declarations to the closest function scope.
2021-07-04 01:15:52 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
return scope_pusher;
}
LibJS: Don't hoist functions under certain circumstances When a lexical declaration with the same name as a function exists, the function is not hoisted (annex B).
2021-07-05 19:45:34 +00:00
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
static ScopePusher catch_scope(Parser& parser, RefPtr<BindingPattern> const& pattern, DeprecatedFlyString const& parameter)
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
{
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
ScopePusher scope_pusher(parser, nullptr, ScopeLevel::NotTopLevel);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (pattern) {
pattern->for_each_bound_name([&](auto const& name) {
scope_pusher.m_forbidden_var_names.set(name);
});
LibJS: Treat the Catch binding identifier as a var binding
2021-10-08 10:04:13 +00:00
} else if (!parameter.is_empty()) {
scope_pusher.m_var_names.set(parameter);
LibJS: Don't hoist functions under certain circumstances When a lexical declaration with the same name as a function exists, the function is not hoisted (annex B).
2021-07-05 19:45:34 +00:00
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
return scope_pusher;
LibJS: Don't hoist functions under certain circumstances When a lexical declaration with the same name as a function exists, the function is not hoisted (annex B).
2021-07-05 19:45:34 +00:00
}
LibJS: Add static initializers to classes
2021-10-20 19:29:47 +00:00
static ScopePusher static_init_block_scope(Parser& parser, ScopeNode& node)
{
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
return ScopePusher(parser, &node, ScopeLevel::StaticInitTopLevel);
LibJS: Add static initializers to classes
2021-10-20 19:29:47 +00:00
}
LibJS: Use ClassFieldInitializerStatement for class fields This is necessary as we might have to perform named evaluation with the field name. Ideally we would also skip some setup parts of the function like function_declaration_instantiation however this would require bigger changes to ECMAScriptFunctionObject.
2021-10-13 17:59:38 +00:00
static ScopePusher class_field_scope(Parser& parser)
{
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
return ScopePusher(parser, nullptr, ScopeLevel::NotTopLevel);
LibJS: Use ClassFieldInitializerStatement for class fields This is necessary as we might have to perform named evaluation with the field name. Ideally we would also skip some setup parts of the function like function_declaration_instantiation however this would require bigger changes to ECMAScriptFunctionObject.
2021-10-13 17:59:38 +00:00
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
void add_declaration(NonnullRefPtr<Declaration> declaration)
LibJS: Don't hoist functions under certain circumstances When a lexical declaration with the same name as a function exists, the function is not hoisted (annex B).
2021-07-05 19:45:34 +00:00
{
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (declaration->is_lexical_declaration()) {
declaration->for_each_bound_name([&](auto const& name) {
if (m_var_names.contains(name) || m_forbidden_lexical_names.contains(name) || m_function_names.contains(name))
throw_identifier_declared(name, declaration);
if (m_lexical_names.set(name) != AK::HashSetResult::InsertedNewEntry)
throw_identifier_declared(name, declaration);
});
m_node->add_lexical_declaration(move(declaration));
} else if (!declaration->is_function_declaration()) {
declaration->for_each_bound_name([&](auto const& name) {
ScopePusher* pusher = this;
while (true) {
if (pusher->m_lexical_names.contains(name)
|| pusher->m_function_names.contains(name)
|| pusher->m_forbidden_var_names.contains(name))
throw_identifier_declared(name, declaration);
pusher->m_var_names.set(name);
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
if (pusher->is_top_level())
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
break;
VERIFY(pusher->m_parent_scope != nullptr);
pusher = pusher->m_parent_scope;
}
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
VERIFY(pusher->is_top_level() && pusher->m_node);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
pusher->m_node->add_var_scoped_declaration(declaration);
});
VERIFY(m_top_level_scope);
m_top_level_scope->m_node->add_var_scoped_declaration(move(declaration));
} else {
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
if (m_scope_level != ScopeLevel::NotTopLevel && m_scope_level != ScopeLevel::ModuleTopLevel) {
// Only non-top levels and Module don't var declare the top functions
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
declaration->for_each_bound_name([&](auto const& name) {
m_var_names.set(name);
});
m_node->add_var_scoped_declaration(move(declaration));
} else {
VERIFY(is<FunctionDeclaration>(*declaration));
auto& function_declaration = static_cast<FunctionDeclaration const&>(*declaration);
auto& function_name = function_declaration.name();
if (m_var_names.contains(function_name) || m_lexical_names.contains(function_name))
throw_identifier_declared(function_name, declaration);
LibJS: Rename FunctionKind::{Regular => Normal} This is what CreateDynamicFunction calls it.
2022-01-14 23:30:02 +00:00
if (function_declaration.kind() != FunctionKind::Normal || m_parser.m_state.strict_mode) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (m_function_names.contains(function_name))
throw_identifier_declared(function_name, declaration);
m_lexical_names.set(function_name);
m_node->add_lexical_declaration(move(declaration));
return;
}
m_function_names.set(function_name);
if (!m_lexical_names.contains(function_name))
m_functions_to_hoist.append(static_ptr_cast<FunctionDeclaration>(declaration));
m_node->add_lexical_declaration(move(declaration));
LibJS: Don't hoist functions under certain circumstances When a lexical declaration with the same name as a function exists, the function is not hoisted (annex B).
2021-07-05 19:45:34 +00:00
}
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
}
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
ScopePusher const* last_function_scope() const
{
for (auto scope_ptr = this; scope_ptr; scope_ptr = scope_ptr->m_parent_scope) {
if (scope_ptr->m_function_parameters.has_value())
return scope_ptr;
}
return nullptr;
}
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
Vector<FunctionParameter> const& function_parameters() const
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
{
return *m_function_parameters;
}
ScopePusher* parent_scope() { return m_parent_scope; }
ScopePusher const* parent_scope() const { return m_parent_scope; }
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
[[nodiscard]] bool has_declaration(DeprecatedFlyString const& name) const
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
{
return m_lexical_names.contains(name) || m_var_names.contains(name) || !m_functions_to_hoist.find_if([&name](auto& function) { return function->name() == name; }).is_end();
}
bool contains_direct_call_to_eval() const { return m_contains_direct_call_to_eval; }
bool contains_access_to_arguments_object() const { return m_contains_access_to_arguments_object; }
void set_contains_direct_call_to_eval() { m_contains_direct_call_to_eval = true; }
void set_contains_access_to_arguments_object() { m_contains_access_to_arguments_object = true; }
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
~ScopePusher()
{
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
VERIFY(is_top_level() || m_parent_scope);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
for (size_t i = 0; i < m_functions_to_hoist.size(); i++) {
auto const& function_declaration = m_functions_to_hoist[i];
if (m_lexical_names.contains(function_declaration.name()) || m_forbidden_var_names.contains(function_declaration.name()))
continue;
LibJS: No longer hoist if parent scope has a function with the same name
2022-11-16 23:52:16 +00:00
if (is_top_level()) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
m_node->add_hoisted_function(move(m_functions_to_hoist[i]));
LibJS: No longer hoist if parent scope has a function with the same name
2022-11-16 23:52:16 +00:00
} else {
if (!m_parent_scope->m_lexical_names.contains(function_declaration.name()) && !m_parent_scope->m_function_names.contains(function_declaration.name()))
m_parent_scope->m_functions_to_hoist.append(move(m_functions_to_hoist[i]));
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
}
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
if (m_parent_scope && !m_function_parameters.has_value()) {
m_parent_scope->m_contains_access_to_arguments_object |= m_contains_access_to_arguments_object;
m_parent_scope->m_contains_direct_call_to_eval |= m_contains_direct_call_to_eval;
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
VERIFY(m_parser.m_state.current_scope_pusher == this);
m_parser.m_state.current_scope_pusher = m_parent_scope;
}
LibJS: Track whether a program has a top level await statement
2022-01-18 17:55:19 +00:00
void set_contains_await_expression()
{
m_contains_await_expression = true;
}
bool contains_await_expression() const
{
return m_contains_await_expression;
}
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
bool can_have_using_declaration() const
{
return m_scope_level != ScopeLevel::ScriptTopLevel;
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
private:
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
void throw_identifier_declared(DeprecatedFlyString const& name, NonnullRefPtr<Declaration> const& declaration)
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
{
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
m_parser.syntax_error(DeprecatedString::formatted("Identifier '{}' already declared", name), declaration->source_range().start);
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
}
Parser& m_parser;
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopeNode* m_node { nullptr };
LibJS: Fix that functions in module did not look for var declarations
2022-11-15 00:39:07 +00:00
ScopeLevel m_scope_level { ScopeLevel::NotTopLevel };
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopePusher* m_parent_scope { nullptr };
ScopePusher* m_top_level_scope { nullptr };
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
HashTable<DeprecatedFlyString> m_lexical_names;
HashTable<DeprecatedFlyString> m_var_names;
HashTable<DeprecatedFlyString> m_function_names;
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
HashTable<DeprecatedFlyString> m_forbidden_lexical_names;
HashTable<DeprecatedFlyString> m_forbidden_var_names;
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
NonnullRefPtrVector<FunctionDeclaration> m_functions_to_hoist;
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
Optional<Vector<FunctionParameter>> m_function_parameters;
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
bool m_contains_access_to_arguments_object { false };
bool m_contains_direct_call_to_eval { false };
LibJS: Track whether a program has a top level await statement
2022-01-18 17:55:19 +00:00
bool m_contains_await_expression { false };
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
};
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
class OperatorPrecedenceTable {
public:
constexpr OperatorPrecedenceTable()
: m_token_precedence()
{
for (size_t i = 0; i < array_size(m_operator_precedence); ++i) {
auto& op = m_operator_precedence[i];
m_token_precedence[static_cast<size_t>(op.token)] = op.precedence;
}
}
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 13:24:43 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
constexpr int get(TokenType token) const
{
int p = m_token_precedence[static_cast<size_t>(token)];
if (p == 0) {
LibJS: Use new format functions everywhere This changes the remaining uses of the following functions across LibJS: - String::format() => String::formatted() - dbg() => dbgln() - printf() => out(), outln() - fprintf() => warnln() I also removed the relevant 'LogStream& operator<<' overloads as they're not needed anymore.
2020-12-06 16:55:19 +00:00
warnln("Internal Error: No precedence for operator {}", Token::name(token));
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 19:42:32 +00:00
VERIFY_NOT_REACHED();
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
return -1;
}
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
return p;
}
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
private:
int m_token_precedence[cs_num_of_js_tokens];
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
struct OperatorPrecedence {
TokenType token;
int precedence;
};
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence
static constexpr const OperatorPrecedence m_operator_precedence[] = {
{ TokenType::Period, 20 },
{ TokenType::BracketOpen, 20 },
{ TokenType::ParenOpen, 20 },
{ TokenType::QuestionMarkPeriod, 20 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::New, 19 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::PlusPlus, 18 },
{ TokenType::MinusMinus, 18 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::ExclamationMark, 17 },
{ TokenType::Tilde, 17 },
{ TokenType::Typeof, 17 },
{ TokenType::Void, 17 },
{ TokenType::Delete, 17 },
{ TokenType::Await, 17 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::DoubleAsterisk, 16 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::Asterisk, 15 },
{ TokenType::Slash, 15 },
{ TokenType::Percent, 15 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::Plus, 14 },
{ TokenType::Minus, 14 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::ShiftLeft, 13 },
{ TokenType::ShiftRight, 13 },
{ TokenType::UnsignedShiftRight, 13 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::LessThan, 12 },
{ TokenType::LessThanEquals, 12 },
{ TokenType::GreaterThan, 12 },
{ TokenType::GreaterThanEquals, 12 },
{ TokenType::In, 12 },
{ TokenType::Instanceof, 12 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::EqualsEquals, 11 },
{ TokenType::ExclamationMarkEquals, 11 },
{ TokenType::EqualsEqualsEquals, 11 },
{ TokenType::ExclamationMarkEqualsEquals, 11 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::Ampersand, 10 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::Caret, 9 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::Pipe, 8 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::DoubleQuestionMark, 7 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::DoubleAmpersand, 6 },
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::DoublePipe, 5 },
{ TokenType::QuestionMark, 4 },
{ TokenType::Equals, 3 },
{ TokenType::PlusEquals, 3 },
{ TokenType::MinusEquals, 3 },
{ TokenType::DoubleAsteriskEquals, 3 },
{ TokenType::AsteriskEquals, 3 },
{ TokenType::SlashEquals, 3 },
{ TokenType::PercentEquals, 3 },
{ TokenType::ShiftLeftEquals, 3 },
{ TokenType::ShiftRightEquals, 3 },
{ TokenType::UnsignedShiftRightEquals, 3 },
{ TokenType::AmpersandEquals, 3 },
{ TokenType::CaretEquals, 3 },
LibJS: Implement logical assignment operators (&&=, ||=, ??=) TC39 proposal, stage 4 as of 2020-07. https://tc39.es/proposal-logical-assignment/
2020-10-05 15:49:43 +00:00
{ TokenType::PipeEquals, 3 },
{ TokenType::DoubleAmpersandEquals, 3 },
{ TokenType::DoublePipeEquals, 3 },
{ TokenType::DoubleQuestionMarkEquals, 3 },
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{ TokenType::Yield, 2 },
{ TokenType::Comma, 1 },
};
};
constexpr OperatorPrecedenceTable g_operator_precedence;
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Add a mode to parse JS as a module In a module strict mode should be enabled at the start of parsing and we allow import and export statements.
2021-08-14 15:30:37 +00:00
Parser::ParserState::ParserState(Lexer l, Program::Type program_type)
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
: lexer(move(l))
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
{
LibJS: Add a mode to parse JS as a module In a module strict mode should be enabled at the start of parsing and we allow import and export statements.
2021-08-14 15:30:37 +00:00
if (program_type == Program::Type::Module)
lexer.disallow_html_comments();
current_token = lexer.next();
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
}
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Add missing steps and spec comments to PerformEval While adding spec comments to PerformEval, I noticed we were missing multiple steps. Namely, these were: - Checking if the host will allow us to compile the string (allowing LibWeb to perform CSP for eval) - The parser's initial state depending on the environment around us on direct eval: - Allowing new.target via eval in functions - Allowing super calls and super properties via eval in classes - Disallowing the use of the arguments object in class field initializers at eval's parse time - Setting ScriptOrModule of eval's execution context The spec allows us to apply the additional parsing steps in any order. The method I have gone with is passing in a struct to the parser's constructor, which overrides the parser's initial state to (dis)allow the things stated above from the get-go.
2022-04-09 23:55:45 +00:00
Parser::Parser(Lexer lexer, Program::Type program_type, Optional<EvalInitialState> initial_state_for_eval)
LibJS: Remove some usage of DeprecatedString usage from Lexer This changes the filename member from DeprecatedString to String. Parser has also been updated to meet the updated Lexer interface.
2023-01-26 13:40:21 +00:00
: m_source_code(SourceCode::create(lexer.filename(), String::from_deprecated_string(lexer.source()).release_value_but_fixme_should_propagate_errors()))
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
, m_state(move(lexer), program_type)
LibJS: Add a mode to parse JS as a module In a module strict mode should be enabled at the start of parsing and we allow import and export statements.
2021-08-14 15:30:37 +00:00
, m_program_type(program_type)
LibJS: Parser refactored to use constexpr precedence table Replaced implementation dependent on HashMap with a constexpr PrecedenceTable based on array lookup.
2020-08-18 16:46:36 +00:00
{
LibJS: Add missing steps and spec comments to PerformEval While adding spec comments to PerformEval, I noticed we were missing multiple steps. Namely, these were: - Checking if the host will allow us to compile the string (allowing LibWeb to perform CSP for eval) - The parser's initial state depending on the environment around us on direct eval: - Allowing new.target via eval in functions - Allowing super calls and super properties via eval in classes - Disallowing the use of the arguments object in class field initializers at eval's parse time - Setting ScriptOrModule of eval's execution context The spec allows us to apply the additional parsing steps in any order. The method I have gone with is passing in a struct to the parser's constructor, which overrides the parser's initial state to (dis)allow the things stated above from the get-go.
2022-04-09 23:55:45 +00:00
if (initial_state_for_eval.has_value()) {
m_state.in_eval_function_context = initial_state_for_eval->in_eval_function_context;
m_state.allow_super_property_lookup = initial_state_for_eval->allow_super_property_lookup;
m_state.allow_super_constructor_call = initial_state_for_eval->allow_super_constructor_call;
m_state.in_class_field_initializer = initial_state_for_eval->in_class_field_initializer;
}
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
}
Associativity Parser::operator_associativity(TokenType type) const
{
switch (type) {
case TokenType::Period:
case TokenType::BracketOpen:
case TokenType::ParenOpen:
case TokenType::QuestionMarkPeriod:
case TokenType::Asterisk:
case TokenType::Slash:
case TokenType::Percent:
case TokenType::Plus:
case TokenType::Minus:
case TokenType::ShiftLeft:
case TokenType::ShiftRight:
case TokenType::UnsignedShiftRight:
case TokenType::LessThan:
case TokenType::LessThanEquals:
case TokenType::GreaterThan:
case TokenType::GreaterThanEquals:
case TokenType::In:
case TokenType::Instanceof:
case TokenType::EqualsEquals:
case TokenType::ExclamationMarkEquals:
case TokenType::EqualsEqualsEquals:
case TokenType::ExclamationMarkEqualsEquals:
LibJS: Fix broken parsing of `!o.a` Unary expressions parsing now respects precedence and associativity of operators. This patch also makes `typeof` left-associative which was an oversight. Thanks to Conrad for helping me work this out. :^)
2020-03-28 10:48:52 +00:00
case TokenType::Typeof:
LibJS: Implement void operator
2020-04-15 16:55:03 +00:00
case TokenType::Void:
LibJS: Implement basic support for the "delete" operator It turns out "delete" is actually a unary op :) This patch implements deletion of object properties, it doesn't yet work for casually deleting properties from the global object. When deleting a property from an object, we switch that object to having a unique shape, no longer sharing shapes with others. Once an object has a unique shape, it no longer needs to care about shape transitions.
2020-04-26 11:53:40 +00:00
case TokenType::Delete:
LibJS: Add support for await expressions
2021-11-09 20:52:21 +00:00
case TokenType::Await:
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
case TokenType::Ampersand:
case TokenType::Caret:
case TokenType::Pipe:
case TokenType::DoubleQuestionMark:
case TokenType::DoubleAmpersand:
case TokenType::DoublePipe:
case TokenType::Comma:
return Associativity::Left;
default:
return Associativity::Right;
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
bool Parser::parse_directive(ScopeNode& body)
{
bool found_use_strict = false;
while (!done() && match(TokenType::StringLiteral)) {
LibJS: Remove is_use_strict_directive for all StringLiterals This value was only used in the parser so no need to have this in every string literal in the ast.
2022-11-27 01:21:25 +00:00
auto raw_value = m_state.current_token.original_value();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
// It cannot be a labelled function since we hit a string literal.
auto statement = parse_statement(AllowLabelledFunction::No);
body.append(statement);
VERIFY(is<ExpressionStatement>(*statement));
auto& expression = static_cast<ExpressionStatement const&>(*statement).expression();
if (!is<StringLiteral>(expression))
break;
LibJS: Remove is_use_strict_directive for all StringLiterals This value was only used in the parser so no need to have this in every string literal in the ast.
2022-11-27 01:21:25 +00:00
if (raw_value.is_one_of("'use strict'"sv, "\"use strict\"")) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
found_use_strict = true;
if (m_state.string_legacy_octal_escape_sequence_in_scope)
syntax_error("Octal escape sequence in string literal not allowed in strict mode");
break;
}
}
m_state.string_legacy_octal_escape_sequence_in_scope = false;
return found_use_strict;
}
LibJS: Correct behaviour of direct vs. indirect eval eval only has direct access to the local scope when accessed through the name eval. This includes locals named eval, because of course it does.
2021-06-20 03:13:53 +00:00
NonnullRefPtr<Program> Parser::parse_program(bool starts_in_strict_mode)
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto program = adopt_ref(*new Program({ m_source_code, rule_start.position(), position() }, m_program_type));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopePusher program_scope = ScopePusher::program_scope(*this, *program);
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
if (m_program_type == Program::Type::Script)
parse_script(program, starts_in_strict_mode);
else
parse_module(program);
LibJS: Avoid unnecessary SourceRange construction in parse_program() This takes `test-js` runtime from 4.3 sec to 4.1 sec on my machine.
2022-11-24 14:38:52 +00:00
program->set_end_offset({}, position().offset);
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
return program;
}
void Parser::parse_script(Program& program, bool starts_in_strict_mode)
{
bool strict_before = m_state.strict_mode;
if (starts_in_strict_mode)
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
m_state.strict_mode = true;
bool has_use_strict = parse_directive(program);
if (m_state.strict_mode || has_use_strict) {
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
program.set_strict_mode();
LibJS: Correct behaviour of direct vs. indirect eval eval only has direct access to the local scope when accessed through the name eval. This includes locals named eval, because of course it does.
2021-06-20 03:13:53 +00:00
m_state.strict_mode = true;
}
LibJS: Add strict mode Adds the ability for a scope (either a function or the entire program) to be in strict mode. Scopes default to non-strict mode. There are two ways to determine the strict-ness of the JS engine: 1. In the parser, this can be accessed with the parser_state variable m_is_strict_mode boolean. If true, the Parser is currently parsing in strict mode. This is done so that the Parser can generate syntax errors at parse time, which is required in some cases. 2. With Interpreter.is_strict_mode(). This allows strict mode checking at runtime as opposed to compile time. Additionally, in order to test this, a global isStrictMode() function has been added to the JS ReplObject under the test-mode flag.
2020-05-28 05:22:08 +00:00
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
parse_statement_list(program, AllowLabelledFunction::Yes);
if (!done()) {
expected("statement or declaration");
consume();
}
m_state.strict_mode = strict_before;
}
void Parser::parse_module(Program& program)
{
TemporaryChange strict_mode_rollback(m_state.strict_mode, true);
TemporaryChange await_expression_valid_rollback(m_state.await_expression_is_valid, true);
// Since strict mode is already enabled we skip any directive parsing.
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
while (!done()) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
parse_statement_list(program, AllowLabelledFunction::Yes);
LibJS: Be less strict about 'use strict' It is allowed to have (bogus) directives e.g. 'does nothing'; 'same'; 'use strict'; Should still trigger strict mode.
2021-07-11 23:25:32 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (done())
break;
LibJS: Be less strict about 'use strict' It is allowed to have (bogus) directives e.g. 'does nothing'; 'same'; 'use strict'; Should still trigger strict mode.
2021-07-11 23:25:32 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (match_export_or_import()) {
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
VERIFY(m_state.current_token.type() == TokenType::Export || m_state.current_token.type() == TokenType::Import);
if (m_state.current_token.type() == TokenType::Export)
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
program.append_export(parse_export_statement(program));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
else
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
program.append_import(parse_import_statement(program));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
} else {
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
expected("statement or declaration");
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume();
}
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
LibJS: Track whether a program has a top level await statement
2022-01-18 17:55:19 +00:00
VERIFY(m_state.current_scope_pusher);
if (m_state.current_scope_pusher->contains_await_expression())
program.set_has_top_level_await();
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
for (auto& export_statement : program.exports()) {
if (export_statement.has_statement())
continue;
for (auto& entry : export_statement.entries()) {
LibJS: Move {Import,Export}Entry out of {Import,Export}Statement By making these be standalone instead of nested structs, we can forward declare them. This will allow us to stop including AST.h in some places.
2022-11-23 11:16:51 +00:00
if (entry.is_module_request() || entry.kind == ExportEntry::Kind::EmptyNamedExport)
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
return;
auto const& exported_name = entry.local_or_import_name;
bool found = false;
program.for_each_lexically_declared_name([&](auto const& name) {
LibJS: Convert ScopeNode declaration functions to ThrowCompletionOr This removes a number of vm.exception() checks which are now caught directly by TRY. Make use of these checks in {Global, Eval}DeclarationInstantiation and while we're here add spec comments.
2022-02-07 12:31:01 +00:00
if (name == exported_name)
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
found = true;
});
if (found)
continue;
program.for_each_var_declared_name([&](auto const& name) {
LibJS: Convert ScopeNode declaration functions to ThrowCompletionOr This removes a number of vm.exception() checks which are now caught directly by TRY. Make use of these checks in {Global, Eval}DeclarationInstantiation and while we're here add spec comments.
2022-02-07 12:31:01 +00:00
if (name == exported_name)
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
found = true;
});
LibJS: Allow exporting any imported bindings
2022-09-01 21:13:45 +00:00
for (auto& import : program.imports()) {
if (import.has_bound_name(exported_name)) {
found = true;
break;
}
}
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
if (!found)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("'{}' in export is not declared", exported_name), export_statement.source_range().start);
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
}
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
NonnullRefPtr<Declaration> Parser::parse_declaration()
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (m_state.current_token.type() == TokenType::Async && next_token().type() == TokenType::Function)
return parse_function_node<FunctionDeclaration>();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
switch (m_state.current_token.type()) {
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
case TokenType::Class:
LibJS: Avoid unnecessary lambda Especially when it's evaluated immediately and unconditionally.
2020-08-29 11:25:37 +00:00
return parse_class_declaration();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
case TokenType::Function:
return parse_function_node<FunctionDeclaration>();
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
case TokenType::Let:
case TokenType::Const:
return parse_variable_declaration();
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
case TokenType::Identifier:
if (m_state.current_token.original_value() == "using"sv) {
if (!m_state.current_scope_pusher->can_have_using_declaration())
syntax_error("'using' not allowed outside of block, for loop or function");
return parse_using_declaration();
}
[[fallthrough]];
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
default:
expected("declaration");
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ErrorDeclaration>({ m_source_code, rule_start.position(), position() });
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
}
}
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
NonnullRefPtr<Statement> Parser::parse_statement(AllowLabelledFunction allow_labelled_function)
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
auto type = m_state.current_token.type();
switch (type) {
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
case TokenType::CurlyOpen:
return parse_block_statement();
case TokenType::Return:
return parse_return_statement();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
case TokenType::Var: {
auto declaration = parse_variable_declaration();
m_state.current_scope_pusher->add_declaration(declaration);
return declaration;
}
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
case TokenType::For:
return parse_for_statement();
LibJS: Parse "if" statements This patch implements basic parsing of "if" statements. We don't yet support parsing "else", so I added a FIXME about that.
2020-03-21 17:40:17 +00:00
case TokenType::If:
return parse_if_statement();
LibJS: Implement "throw" You can now throw an expression to the nearest catcher! :^) To support throwing arbitrary values, I added an Exception class that sits as a wrapper around whatever is thrown. In the future it will be a logical place to store a call stack.
2020-03-24 21:03:50 +00:00
case TokenType::Throw:
return parse_throw_statement();
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
case TokenType::Try:
return parse_try_statement();
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
case TokenType::Break:
return parse_break_statement();
LibJS: Add support for "continue" inside "for" statements :^)
2020-04-04 22:22:42 +00:00
case TokenType::Continue:
LibJS: Avoid unnecessary lambda Especially when it's evaluated immediately and unconditionally.
2020-08-29 11:25:37 +00:00
return parse_continue_statement();
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
case TokenType::Switch:
return parse_switch_statement();
LibJS: Add support for do..while statements
2020-04-04 19:29:23 +00:00
case TokenType::Do:
return parse_do_while_statement();
LibJS: Parse while statements
2020-04-21 18:27:57 +00:00
case TokenType::While:
return parse_while_statement();
LibJS: Parse "with" statements :^)
2020-11-28 14:05:57 +00:00
case TokenType::With:
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (m_state.strict_mode)
LibJS: Disallow 'with' statement in strict mode
2020-11-28 19:17:33 +00:00
syntax_error("'with' statement not allowed in strict mode");
LibJS: Parse "with" statements :^)
2020-11-28 14:05:57 +00:00
return parse_with_statement();
LibJS: Implement (no-op) debugger statement
2020-04-30 16:26:27 +00:00
case TokenType::Debugger:
LibJS: Avoid unnecessary lambda Especially when it's evaluated immediately and unconditionally.
2020-08-29 11:25:37 +00:00
return parse_debugger_statement();
LibJS: Support empty statements We already skipped random semicolons in Parser::parse_program(), but now they are properly matched and parsed as empty statements - and thus recognized as a valid body of an if / else / while / ... statement.
2020-05-03 09:59:00 +00:00
case TokenType::Semicolon:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<EmptyStatement>({ m_source_code, rule_start.position(), position() });
LibJS: Force the lexer to parse a regex when expecting a statement
2021-07-29 21:28:28 +00:00
case TokenType::Slash:
case TokenType::SlashEquals:
m_state.current_token = m_state.lexer.force_slash_as_regex();
[[fallthrough]];
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
default:
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
if (match_invalid_escaped_keyword())
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
syntax_error("Keyword must not contain escaped characters");
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
if (match_identifier_name()) {
auto result = try_parse_labelled_statement(allow_labelled_function);
LibJS: Integrate labels into the Interpreter The interpreter now considers a statement or block's label when considering whether or not to break. All statements can be labelled.
2020-05-28 20:36:59 +00:00
if (!result.is_null())
return result.release_nonnull();
}
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
if (match_expression()) {
LibJS: Don't match async \n function as an async function declaration
2021-11-30 14:52:51 +00:00
if (match(TokenType::Async)) {
auto lookahead_token = next_token();
if (lookahead_token.type() == TokenType::Function && !lookahead_token.trivia_contains_line_terminator())
syntax_error("Async function declaration not allowed in single-statement context");
} else if (match(TokenType::Function) || match(TokenType::Class)) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("{} declaration not allowed in single-statement context", m_state.current_token.name()));
LibJS: Don't match async \n function as an async function declaration
2021-11-30 14:52:51 +00:00
} else if (match(TokenType::Let) && next_token().type() == TokenType::BracketOpen) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("let followed by [ is not allowed in single-statement context"));
LibJS: Don't match async \n function as an async function declaration
2021-11-30 14:52:51 +00:00
}
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
auto expr = parse_expression(0);
consume_or_insert_semicolon();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ExpressionStatement>({ m_source_code, rule_start.position(), position() }, move(expr));
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
}
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
expected("statement");
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ErrorStatement>({ m_source_code, rule_start.position(), position() });
LibJS: Avoid unnecessary lambda Especially when it's evaluated immediately and unconditionally.
2020-08-29 11:25:37 +00:00
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
bool Parser::match_invalid_escaped_keyword() const
{
if (m_state.current_token.type() != TokenType::EscapedKeyword)
return false;
auto token_value = m_state.current_token.value();
LibJS: Allow escaped 'async' as identifier Since 'async' is only special if it occurs before a function it can be used as escaped identifier in all cases.
2021-11-26 22:25:10 +00:00
if (token_value == "await"sv)
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
return m_program_type == Program::Type::Module || m_state.await_expression_is_valid;
LibJS: Allow escaped 'async' as identifier Since 'async' is only special if it occurs before a function it can be used as escaped identifier in all cases.
2021-11-26 22:25:10 +00:00
if (token_value == "async"sv)
return false;
if (token_value == "yield"sv)
return m_state.in_generator_function_context;
if (m_state.strict_mode)
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
return true;
LibJS: Allow escaped 'async' as identifier Since 'async' is only special if it occurs before a function it can be used as escaped identifier in all cases.
2021-11-26 22:25:10 +00:00
return token_value != "let"sv;
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
}
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
static constexpr AK::Array<StringView, 9> strict_reserved_words = { "implements"sv, "interface"sv, "let"sv, "package"sv, "private"sv, "protected"sv, "public"sv, "static"sv, "yield"sv };
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
static bool is_strict_reserved_word(StringView str)
{
Everywhere: Pass AK::StringView by value
2021-11-10 23:55:02 +00:00
return any_of(strict_reserved_words, [&str](StringView word) {
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
return word == str;
});
}
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
static bool is_simple_parameter_list(Vector<FunctionParameter> const& parameters)
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
{
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
return all_of(parameters, [](FunctionParameter const& parameter) {
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
return !parameter.is_rest && parameter.default_value.is_null() && parameter.binding.has<DeprecatedFlyString>();
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
});
}
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
RefPtr<FunctionExpression> Parser::try_parse_arrow_function_expression(bool expect_parens, bool is_async)
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
{
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
if (is_async)
VERIFY(match(TokenType::Async));
if (!expect_parens && !is_async) {
LibJS: Add a fast failure path to try_parse_arrow_function_expression() The save/load of parser state performed by lookahead parsing is quite expensive so let's try to avoid it in the most common case. This is a 15-20% speedup on various chunks of JS I've tested. :^)
2021-09-14 00:51:16 +00:00
// NOTE: This is a fast path where we try to fail early in case this can't possibly
// be a match. The idea is to avoid the expensive parser state save/load mechanism.
// The logic is duplicated below in the "real" !expect_parens branch.
if (!match_identifier() && !match(TokenType::Yield) && !match(TokenType::Await))
return nullptr;
LibJS: Do not save state for peeking at the next token from the lexer This saves having to save and load the parser state. This could give an incorrect token in some cases where the parser communicates to the lexer. However this is not applicable in any of the current usages and this would require one to parse the current token as normal which is exactly what you don't want to do in that scenario.
2021-10-07 22:38:24 +00:00
auto token = next_token();
LibJS: Add a fast failure path to try_parse_arrow_function_expression() The save/load of parser state performed by lookahead parsing is quite expensive so let's try to avoid it in the most common case. This is a 15-20% speedup on various chunks of JS I've tested. :^)
2021-09-14 00:51:16 +00:00
if (token.trivia_contains_line_terminator())
return nullptr;
if (token.type() != TokenType::Arrow)
return nullptr;
}
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
save_state();
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto rule_start = (expect_parens && !is_async)
// Someone has consumed the opening parenthesis for us! Start there.
? RulePosition { *this, m_rule_starts.last() }
// We've not encountered one yet, so the rule start is actually here.
: push_start();
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
ArmedScopeGuard state_rollback_guard = [&] {
load_state();
};
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
LibJS: Rename FunctionKind::{Regular => Normal} This is what CreateDynamicFunction calls it.
2022-01-14 23:30:02 +00:00
auto function_kind = FunctionKind::Normal;
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
if (is_async) {
consume(TokenType::Async);
function_kind = FunctionKind::Async;
if (m_state.current_token.trivia_contains_line_terminator())
return nullptr;
// Since we have async it can be followed by paren open in the expect_parens case
// so we also consume that token.
if (expect_parens) {
VERIFY(match(TokenType::ParenOpen));
consume(TokenType::ParenOpen);
}
}
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
Vector<FunctionParameter> parameters;
LibJS: Function.length respects default and rest parameters "[Function.length is] the number of formal parameters. This number excludes the rest parameter and only includes parameters before the first one with a default value." - MDN
2020-05-06 03:02:14 +00:00
i32 function_length = -1;
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
if (expect_parens) {
// We have parens around the function parameters and can re-use the same parsing
// logic used for regular functions: multiple parameters, default values, rest
// parameter, maybe a trailing comma. If we have a new syntax error afterwards we
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
// check if it's about a wrong token (something like duplicate parameter name must
// not abort), know parsing failed and rollback the parser state.
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto previous_syntax_errors = m_state.errors.size();
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
TemporaryChange in_async_context(m_state.await_expression_is_valid, is_async || m_state.await_expression_is_valid);
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
parameters = parse_formal_parameters(function_length, FunctionNodeParseOptions::IsArrowFunction | (is_async ? FunctionNodeParseOptions::IsAsyncFunction : 0));
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
if (m_state.errors.size() > previous_syntax_errors && m_state.errors[previous_syntax_errors].message.starts_with("Unexpected token"sv))
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
return nullptr;
if (!match(TokenType::ParenClose))
return nullptr;
consume();
} else {
// No parens - this must be an identifier followed by arrow. That's it.
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
if (!match_identifier() && !match(TokenType::Yield) && !match(TokenType::Await))
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
return nullptr;
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
auto token = consume_identifier_reference();
if (m_state.strict_mode && token.value().is_one_of("arguments"sv, "eval"sv))
syntax_error("BindingIdentifier may not be 'arguments' or 'eval' in strict mode");
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
if (is_async && token.value() == "await"sv)
syntax_error("'await' is a reserved identifier in async functions");
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
parameters.append({ DeprecatedFlyString { token.value() }, {} });
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
}
LibJS: Don't parse arrow function with newline between ) and => If there's a newline between the closing paren and arrow it's not a valid arrow function, ASI should kick in instead (it'll then fail with "Unexpected token Arrow")
2020-10-18 23:29:17 +00:00
// If there's a newline between the closing paren and arrow it's not a valid arrow function,
// ASI should kick in instead (it'll then fail with "Unexpected token Arrow")
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (m_state.current_token.trivia_contains_line_terminator())
LibJS: Don't parse arrow function with newline between ) and => If there's a newline between the closing paren and arrow it's not a valid arrow function, ASI should kick in instead (it'll then fail with "Unexpected token Arrow")
2020-10-18 23:29:17 +00:00
return nullptr;
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
if (!match(TokenType::Arrow))
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
return nullptr;
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
consume();
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
if (function_length == -1)
LibJS: Function.length respects default and rest parameters "[Function.length is] the number of formal parameters. This number excludes the rest parameter and only includes parameters before the first one with a default value." - MDN
2020-05-06 03:02:14 +00:00
function_length = parameters.size();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto old_labels_in_scope = move(m_state.labels_in_scope);
LibJS: break or continue with nonexistent label is a syntax error
2020-10-08 17:49:08 +00:00
ScopeGuard guard([&]() {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
m_state.labels_in_scope = move(old_labels_in_scope);
LibJS: break or continue with nonexistent label is a syntax error
2020-10-08 17:49:08 +00:00
});
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 10:43:38 +00:00
bool contains_direct_call_to_eval = false;
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
auto function_body_result = [&]() -> RefPtr<FunctionBody> {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
TemporaryChange change(m_state.in_arrow_function_context, true);
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
TemporaryChange async_context_change(m_state.await_expression_is_valid, is_async);
LibJS: Disallow await keywords in static init blocks In static init blocks 'await' cannot be used. Note that this does not cover all the cases since the parser currently cannot distinguish between expressions within parenthesis and direct expressions.
2021-11-26 22:29:05 +00:00
TemporaryChange in_class_static_init_block_change(m_state.in_class_static_init_block, false);
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
if (match(TokenType::CurlyOpen)) {
// Parse a function body with statements
LibJS: Consume curly braces outside of Parser::parse_function_body() The curly braces are not part of the FunctionBody production. This becomes relevant when parsing a standalone function body as per the spec in the CreateDynamicFunction AO.
2022-01-15 16:07:51 +00:00
consume(TokenType::CurlyOpen);
auto body = parse_function_body(parameters, function_kind, contains_direct_call_to_eval);
consume(TokenType::CurlyClose);
return body;
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
}
if (match_expression()) {
// Parse a function body which returns a single expression
// FIXME: We synthesize a block with a return statement
// for arrow function bodies which are a single expression.
// Esprima generates a single "ArrowFunctionExpression"
// with a "body" property.
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto return_block = create_ast_node<FunctionBody>({ m_source_code, rule_start.position(), position() });
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
ScopePusher function_scope = ScopePusher::function_scope(*this, return_block, parameters);
auto return_expression = parse_expression(2);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return_block->append<ReturnStatement>({ m_source_code, rule_start.position(), position() }, move(return_expression));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (m_state.strict_mode)
return_block->set_strict_mode();
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 10:43:38 +00:00
contains_direct_call_to_eval = function_scope.contains_direct_call_to_eval();
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
return return_block;
}
// Invalid arrow function body
return nullptr;
}();
LibJS: Handle strict mode for functions more correctly If a function is strict (has 'use strict' directive) it cannot have bindings, cannot have duplicated parameter names and cannot have some reserved keywords and identifiers as parameter names. The logic partly applies depending on whether we are already in strict mode or the function contains 'use strict';
2021-07-11 23:29:07 +00:00
if (function_body_result.is_null())
return nullptr;
state_rollback_guard.disarm();
discard_saved_state();
auto body = function_body_result.release_nonnull();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (body->in_strict_mode()) {
LibJS: Handle strict mode for functions more correctly If a function is strict (has 'use strict' directive) it cannot have bindings, cannot have duplicated parameter names and cannot have some reserved keywords and identifiers as parameter names. The logic partly applies depending on whether we are already in strict mode or the function contains 'use strict';
2021-07-11 23:29:07 +00:00
for (auto& parameter : parameters) {
parameter.binding.visit(
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
[&](DeprecatedFlyString const& name) {
LibJS: Handle strict mode for functions more correctly If a function is strict (has 'use strict' directive) it cannot have bindings, cannot have duplicated parameter names and cannot have some reserved keywords and identifiers as parameter names. The logic partly applies depending on whether we are already in strict mode or the function contains 'use strict';
2021-07-11 23:29:07 +00:00
check_identifier_name_for_assignment_validity(name, true);
},
[&](auto const&) {});
}
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
}
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto function_start_offset = rule_start.position().offset;
auto function_end_offset = position().offset - m_state.current_token.trivia().length();
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
auto source_text = DeprecatedString { m_state.lexer.source().substring_view(function_start_offset, function_end_offset - function_start_offset) };
LibJS: Handle strict mode for functions more correctly If a function is strict (has 'use strict' directive) it cannot have bindings, cannot have duplicated parameter names and cannot have some reserved keywords and identifiers as parameter names. The logic partly applies depending on whether we are already in strict mode or the function contains 'use strict';
2021-07-11 23:29:07 +00:00
return create_ast_node<FunctionExpression>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() }, "", move(source_text),
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
move(body), move(parameters), function_length, function_kind, body->in_strict_mode(),
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 10:43:38 +00:00
/* might_need_arguments_object */ false, contains_direct_call_to_eval, /* is_arrow_function */ true);
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
}
LibJS: Replace the custom unwind mechanism with completions :^) This includes: - Parsing proper LabelledStatements with try_parse_labelled_statement() - Removing LabelableStatement - Implementing the LoopEvaluation semantics via loop_evaluation() in each IterationStatement subclass; and IterationStatement evaluation via {For,ForIn,ForOf,ForAwaitOf,While,DoWhile}Statement::execute() - Updating ReturnStatement, BreakStatement and ContinueStatement to return the appropriate completion types - Basically reimplementing TryStatement and SwitchStatement according to the spec, using completions - Honoring result completion types in AsyncBlockStart and OrdinaryCallEvaluateBody - Removing any uses of the VM unwind mechanism - most importantly, VM::throw_exception() now exclusively sets an exception and no longer triggers any unwinding mechanism. However, we already did a good job updating all of LibWeb and userland applications to not use it, and the few remaining uses elsewhere don't rely on unwinding AFAICT.
2022-01-05 18:11:16 +00:00
RefPtr<LabelledStatement> Parser::try_parse_labelled_statement(AllowLabelledFunction allow_function)
LibJS: Parse labelled statements All statements now have an optional label string that can be null.
2020-05-28 18:09:19 +00:00
{
LibJS: Add fast failure path to try_parse_labelled_statement() If the next token isn't a TokenType::Colon (:), this can't possibly be a labelled statement, so we can fail before having to save_state(). This improves parsing time on a large chunk of JS by ~12.5%.
2021-09-18 17:39:20 +00:00
{
// NOTE: This is a fast path where we try to fail early to avoid the expensive save_state+load_state.
LibJS: Do not save state for peeking at the next token from the lexer This saves having to save and load the parser state. This could give an incorrect token in some cases where the parser communicates to the lexer. However this is not applicable in any of the current usages and this would require one to parse the current token as normal which is exactly what you don't want to do in that scenario.
2021-10-07 22:38:24 +00:00
if (next_token().type() != TokenType::Colon)
LibJS: Add fast failure path to try_parse_labelled_statement() If the next token isn't a TokenType::Colon (:), this can't possibly be a labelled statement, so we can fail before having to save_state(). This improves parsing time on a large chunk of JS by ~12.5%.
2021-09-18 17:39:20 +00:00
return {};
}
LibJS: Parse labelled statements All statements now have an optional label string that can be null.
2020-05-28 18:09:19 +00:00
save_state();
LibJS: `save_state()' before creating a RulePosition Fixes #4617.
2020-12-29 05:12:02 +00:00
auto rule_start = push_start();
LibJS: Parse labelled statements All statements now have an optional label string that can be null.
2020-05-28 18:09:19 +00:00
ArmedScopeGuard state_rollback_guard = [&] {
load_state();
};
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
if (m_state.current_token.value() == "yield"sv && (m_state.strict_mode || m_state.in_generator_function_context)) {
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
return {};
}
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
if (m_state.current_token.value() == "await"sv && (m_program_type == Program::Type::Module || m_state.await_expression_is_valid || m_state.in_class_static_init_block)) {
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
return {};
}
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
auto identifier = [&] {
if (m_state.current_token.value() == "await"sv) {
return consume().value();
}
return consume_identifier_reference().value();
}();
LibJS: Parse labelled statements All statements now have an optional label string that can be null.
2020-05-28 18:09:19 +00:00
if (!match(TokenType::Colon))
return {};
consume(TokenType::Colon);
if (!match_statement())
return {};
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
state_rollback_guard.disarm();
discard_saved_state();
if (m_state.strict_mode && identifier == "let"sv) {
syntax_error("Strict mode reserved word 'let' is not allowed in label", rule_start.position());
return {};
}
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
if (match(TokenType::Function) && (allow_function == AllowLabelledFunction::No || m_state.strict_mode)) {
syntax_error("Not allowed to declare a function here");
return {};
}
if (m_state.labels_in_scope.contains(identifier))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Label '{}' has already been declared", identifier));
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
LibJS: Replace the custom unwind mechanism with completions :^) This includes: - Parsing proper LabelledStatements with try_parse_labelled_statement() - Removing LabelableStatement - Implementing the LoopEvaluation semantics via loop_evaluation() in each IterationStatement subclass; and IterationStatement evaluation via {For,ForIn,ForOf,ForAwaitOf,While,DoWhile}Statement::execute() - Updating ReturnStatement, BreakStatement and ContinueStatement to return the appropriate completion types - Basically reimplementing TryStatement and SwitchStatement according to the spec, using completions - Honoring result completion types in AsyncBlockStart and OrdinaryCallEvaluateBody - Removing any uses of the VM unwind mechanism - most importantly, VM::throw_exception() now exclusively sets an exception and no longer triggers any unwinding mechanism. However, we already did a good job updating all of LibWeb and userland applications to not use it, and the few remaining uses elsewhere don't rely on unwinding AFAICT.
2022-01-05 18:11:16 +00:00
RefPtr<Statement> labelled_item;
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
auto is_iteration_statement = false;
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
if (match(TokenType::Function)) {
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
m_state.labels_in_scope.set(identifier, {});
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
auto function_declaration = parse_function_node<FunctionDeclaration>();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
VERIFY(m_state.current_scope_pusher);
m_state.current_scope_pusher->add_declaration(function_declaration);
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
if (function_declaration->kind() == FunctionKind::Generator)
syntax_error("Generator functions cannot be defined in labelled statements");
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (function_declaration->kind() == FunctionKind::Async)
syntax_error("Async functions cannot be defined in labelled statements");
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
LibJS: Replace the custom unwind mechanism with completions :^) This includes: - Parsing proper LabelledStatements with try_parse_labelled_statement() - Removing LabelableStatement - Implementing the LoopEvaluation semantics via loop_evaluation() in each IterationStatement subclass; and IterationStatement evaluation via {For,ForIn,ForOf,ForAwaitOf,While,DoWhile}Statement::execute() - Updating ReturnStatement, BreakStatement and ContinueStatement to return the appropriate completion types - Basically reimplementing TryStatement and SwitchStatement according to the spec, using completions - Honoring result completion types in AsyncBlockStart and OrdinaryCallEvaluateBody - Removing any uses of the VM unwind mechanism - most importantly, VM::throw_exception() now exclusively sets an exception and no longer triggers any unwinding mechanism. However, we already did a good job updating all of LibWeb and userland applications to not use it, and the few remaining uses elsewhere don't rely on unwinding AFAICT.
2022-01-05 18:11:16 +00:00
labelled_item = move(function_declaration);
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
} else {
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
m_state.labels_in_scope.set(identifier, {});
LibJS: Replace the custom unwind mechanism with completions :^) This includes: - Parsing proper LabelledStatements with try_parse_labelled_statement() - Removing LabelableStatement - Implementing the LoopEvaluation semantics via loop_evaluation() in each IterationStatement subclass; and IterationStatement evaluation via {For,ForIn,ForOf,ForAwaitOf,While,DoWhile}Statement::execute() - Updating ReturnStatement, BreakStatement and ContinueStatement to return the appropriate completion types - Basically reimplementing TryStatement and SwitchStatement according to the spec, using completions - Honoring result completion types in AsyncBlockStart and OrdinaryCallEvaluateBody - Removing any uses of the VM unwind mechanism - most importantly, VM::throw_exception() now exclusively sets an exception and no longer triggers any unwinding mechanism. However, we already did a good job updating all of LibWeb and userland applications to not use it, and the few remaining uses elsewhere don't rely on unwinding AFAICT.
2022-01-05 18:11:16 +00:00
labelled_item = parse_statement(allow_function);
// Extract the innermost statement from a potentially nested chain of LabelledStatements.
auto statement = labelled_item;
while (is<LabelledStatement>(*statement))
statement = static_cast<LabelledStatement&>(*statement).labelled_item();
if (is<IterationStatement>(*statement))
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
is_iteration_statement = true;
}
if (!is_iteration_statement) {
if (auto entry = m_state.labels_in_scope.find(identifier); entry != m_state.labels_in_scope.end() && entry->value.has_value())
syntax_error("labelled continue statement cannot use non iterating statement", m_state.labels_in_scope.get(identifier).value());
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-24 23:01:22 +00:00
}
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
m_state.labels_in_scope.remove(identifier);
LibJS: Parse labelled statements All statements now have an optional label string that can be null.
2020-05-28 18:09:19 +00:00
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<LabelledStatement>({ m_source_code, rule_start.position(), position() }, identifier, labelled_item.release_nonnull());
LibJS: Parse labelled statements All statements now have an optional label string that can be null.
2020-05-28 18:09:19 +00:00
}
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
RefPtr<MetaProperty> Parser::try_parse_new_target_expression()
{
LibJS: Lookahead for a period when parsing new.target This allows us to skip saving and loading the state whenever we parse 'new'.
2021-11-26 22:41:59 +00:00
// Optimization which skips the save/load state.
if (next_token().type() != TokenType::Period)
return {};
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
save_state();
LibJS: `save_state()' before creating a RulePosition Fixes #4617.
2020-12-29 05:12:02 +00:00
auto rule_start = push_start();
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
ArmedScopeGuard state_rollback_guard = [&] {
load_state();
};
consume(TokenType::New);
LibJS: Lookahead for a period when parsing new.target This allows us to skip saving and loading the state whenever we parse 'new'.
2021-11-26 22:41:59 +00:00
consume(TokenType::Period);
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
if (!match(TokenType::Identifier))
return {};
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
// The string 'target' cannot have escapes so we check original value.
if (consume().original_value() != "target"sv)
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
return {};
state_rollback_guard.disarm();
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 13:17:39 +00:00
discard_saved_state();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<MetaProperty>({ m_source_code, rule_start.position(), position() }, MetaProperty::Type::NewTarget);
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
}
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
RefPtr<MetaProperty> Parser::try_parse_import_meta_expression()
{
// Optimization which skips the save/load state.
if (next_token().type() != TokenType::Period)
return {};
save_state();
auto rule_start = push_start();
ArmedScopeGuard state_rollback_guard = [&] {
load_state();
};
consume(TokenType::Import);
consume(TokenType::Period);
if (!match(TokenType::Identifier))
return {};
// The string 'meta' cannot have escapes so we check original value.
if (consume().original_value() != "meta"sv)
return {};
state_rollback_guard.disarm();
discard_saved_state();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<MetaProperty>({ m_source_code, rule_start.position(), position() }, MetaProperty::Type::ImportMeta);
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
}
NonnullRefPtr<ImportCall> Parser::parse_import_call()
{
auto rule_start = push_start();
// We use the extended definition:
// ImportCall[Yield, Await]:
// import(AssignmentExpression[+In, ?Yield, ?Await] ,opt)
// import(AssignmentExpression[+In, ?Yield, ?Await] ,AssignmentExpression[+In, ?Yield, ?Await] ,opt)
// From https://tc39.es/proposal-import-assertions/#sec-evaluate-import-call
consume(TokenType::Import);
consume(TokenType::ParenOpen);
auto argument = parse_expression(2);
RefPtr<Expression> options;
if (match(TokenType::Comma)) {
consume(TokenType::Comma);
if (!match(TokenType::ParenClose)) {
options = parse_expression(2);
// Second optional comma
if (match(TokenType::Comma))
consume(TokenType::Comma);
}
}
consume(TokenType::ParenClose);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ImportCall>({ m_source_code, rule_start.position(), position() }, move(argument), move(options));
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
}
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
NonnullRefPtr<ClassDeclaration> Parser::parse_class_declaration()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ClassDeclaration>({ m_source_code, rule_start.position(), position() }, parse_class_expression(true));
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
}
NonnullRefPtr<ClassExpression> Parser::parse_class_expression(bool expect_class_name)
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
// Classes are always in strict mode.
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
TemporaryChange strict_mode_rollback(m_state.strict_mode, true);
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
consume(TokenType::Class);
LibJS: Make class definition evaluation more spec like in ordering
2021-10-06 23:09:04 +00:00
NonnullRefPtrVector<ClassElement> elements;
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
RefPtr<Expression> super_class;
RefPtr<FunctionExpression> constructor;
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
HashTable<DeprecatedFlyString> found_private_names;
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString class_name = expect_class_name || match_identifier() || match(TokenType::Yield) || match(TokenType::Await)
? consume_identifier_reference().DeprecatedFlyString_value()
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
: "";
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
check_identifier_name_for_assignment_validity(class_name, true);
LibJS: Disallow await keywords in static init blocks In static init blocks 'await' cannot be used. Note that this does not cover all the cases since the parser currently cannot distinguish between expressions within parenthesis and direct expressions.
2021-11-26 22:29:05 +00:00
if (m_state.in_class_static_init_block && class_name == "await"sv)
syntax_error("Identifier must not be a reserved word in modules ('await')");
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
if (match(TokenType::Extends)) {
consume();
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
auto [expression, should_continue_parsing] = parse_primary_expression();
LibJS: Extend class 'extends' RHS expression parsing Instead of only parsing a primary expression, we should also allow member expressions, call expressions, and tagged template literals (and optional chains, which we don't have yet). In the spec, all of this is covered by `LeftHandSideExpression` (https://tc39.es/ecma262/#prod-LeftHandSideExpression).
2021-07-17 22:19:03 +00:00
// Basically a (much) simplified parse_secondary_expression().
for (;;) {
if (match(TokenType::TemplateLiteralStart)) {
auto template_literal = parse_template_literal(true);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
expression = create_ast_node<TaggedTemplateLiteral>({ m_source_code, rule_start.position(), position() }, move(expression), move(template_literal));
LibJS: Extend class 'extends' RHS expression parsing Instead of only parsing a primary expression, we should also allow member expressions, call expressions, and tagged template literals (and optional chains, which we don't have yet). In the spec, all of this is covered by `LeftHandSideExpression` (https://tc39.es/ecma262/#prod-LeftHandSideExpression).
2021-07-17 22:19:03 +00:00
continue;
}
if (match(TokenType::BracketOpen) || match(TokenType::Period) || match(TokenType::ParenOpen)) {
auto precedence = g_operator_precedence.get(m_state.current_token.type());
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
expression = parse_secondary_expression(move(expression), precedence).expression;
LibJS: Extend class 'extends' RHS expression parsing Instead of only parsing a primary expression, we should also allow member expressions, call expressions, and tagged template literals (and optional chains, which we don't have yet). In the spec, all of this is covered by `LeftHandSideExpression` (https://tc39.es/ecma262/#prod-LeftHandSideExpression).
2021-07-17 22:19:03 +00:00
continue;
}
break;
}
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
super_class = move(expression);
(void)should_continue_parsing;
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
}
consume(TokenType::CurlyOpen);
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
HashTable<StringView> referenced_private_names;
HashTable<StringView>* outer_referenced_private_names = m_state.referenced_private_names;
m_state.referenced_private_names = &referenced_private_names;
ScopeGuard restore_private_name_table = [&] {
m_state.referenced_private_names = outer_referenced_private_names;
};
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
while (!done() && !match(TokenType::CurlyClose)) {
RefPtr<Expression> property_key;
bool is_static = false;
bool is_constructor = false;
LibJS: Parse generator functions in class expressions too
2021-07-02 10:07:00 +00:00
bool is_generator = false;
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
bool is_async = false;
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
auto method_kind = ClassMethod::Kind::Method;
if (match(TokenType::Semicolon)) {
consume();
continue;
}
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto function_start = position();
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (match(TokenType::Async)) {
LibJS: Allow defining class fields with "keyword" names
2021-11-26 22:30:29 +00:00
auto lookahead_token = next_token();
LibJS: Allow methods in classes named 'async' Also add tests for all static, setter and getter cases.
2022-02-17 17:03:41 +00:00
// If async is followed by a Semicolon or CurlyClose it is a field (CurlyClose indicates end of class)
// Otherwise if it is followed by a ParenOpen it is a function named async
if (lookahead_token.type() != TokenType::Semicolon && lookahead_token.type() != TokenType::CurlyClose && lookahead_token.type() != TokenType::ParenOpen
LibJS: Allow defining class fields with "keyword" names
2021-11-26 22:30:29 +00:00
&& !lookahead_token.trivia_contains_line_terminator()) {
consume();
is_async = true;
}
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
}
LibJS: Parse generator functions in class expressions too
2021-07-02 10:07:00 +00:00
if (match(TokenType::Asterisk)) {
consume();
is_generator = true;
}
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
StringView name;
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
if (match_property_key() || match(TokenType::PrivateIdentifier)) {
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (!is_generator && !is_async && m_state.current_token.original_value() == "static"sv) {
LibJS: Parse generator functions in class expressions too
2021-07-02 10:07:00 +00:00
if (match(TokenType::Identifier)) {
consume();
is_static = true;
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
function_start = position();
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (match(TokenType::Async)) {
consume();
is_async = true;
}
LibJS: Parse generator functions in class expressions too
2021-07-02 10:07:00 +00:00
if (match(TokenType::Asterisk)) {
consume();
is_generator = true;
}
}
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
}
if (match(TokenType::Identifier)) {
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
auto identifier_name = m_state.current_token.original_value();
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
if (identifier_name == "get"sv) {
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
method_kind = ClassMethod::Kind::Getter;
consume();
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
} else if (identifier_name == "set"sv) {
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
method_kind = ClassMethod::Kind::Setter;
consume();
}
}
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
if (match_property_key() || match(TokenType::PrivateIdentifier)) {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
switch (m_state.current_token.type()) {
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
case TokenType::Identifier:
name = consume().value();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
property_key = create_ast_node<StringLiteral>({ m_source_code, rule_start.position(), position() }, name);
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
break;
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
case TokenType::PrivateIdentifier:
name = consume().value();
if (name == "#constructor")
syntax_error("Private property with name '#constructor' is not allowed");
if (method_kind != ClassMethod::Kind::Method) {
// It is a Syntax Error if PrivateBoundIdentifiers of ClassElementList contains any duplicate entries,
// unless the name is used once for a getter and once for a setter and in no other entries,
// and the getter and setter are either both static or both non-static.
for (auto& element : elements) {
auto private_name = element.private_bound_identifier();
if (!private_name.has_value() || private_name.value() != name)
continue;
if (element.class_element_kind() != ClassElement::ElementKind::Method
|| element.is_static() != is_static) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Duplicate private field or method named '{}'", name));
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
break;
}
VERIFY(is<ClassMethod>(element));
auto& class_method_element = static_cast<ClassMethod const&>(element);
if (class_method_element.kind() == ClassMethod::Kind::Method || class_method_element.kind() == method_kind) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Duplicate private field or method named '{}'", name));
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
break;
}
}
found_private_names.set(name);
} else if (found_private_names.set(name) != AK::HashSetResult::InsertedNewEntry) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Duplicate private field or method named '{}'", name));
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
property_key = create_ast_node<PrivateIdentifier>({ m_source_code, rule_start.position(), position() }, name);
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
break;
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
case TokenType::StringLiteral: {
auto string_literal = parse_string_literal(consume());
name = string_literal->value();
property_key = move(string_literal);
break;
}
default:
property_key = parse_property_key();
break;
}
LibJS: Add an optimization to avoid needless arguments object creation This gives FunctionNode a "might need arguments object" boolean flag and sets it based on the simplest possible heuristic for this: if we encounter an identifier called "arguments" or "eval" up to the next (nested) function declaration or expression, we won't need an arguments object. Otherwise, we *might* need one - the final decision is made in the FunctionDeclarationInstantiation AO. Now, this is obviously not perfect. Even if you avoid eval, something like `foo.arguments` will still trigger a false positive - but it's a start and already massively cuts down on needlessly allocated objects, especially in real-world code that is often minified, and so a full "arguments" identifier will be an actual arguments object more often than not. To illustrate the actual impact of this change, here's the number of allocated arguments objects during a full test-js run: Before: - Unmapped arguments objects: 78765 - Mapped arguments objects: 2455 After: - Unmapped arguments objects: 18 - Mapped arguments objects: 37 This results in a ~5% speedup of test-js on my Linux host machine, and about 3.5% on i686 Serenity in QEMU (warm runs, average of 5). The following microbenchmark (calling an empty function 1M times) runs 25% faster on Linux and 45% on Serenity: function foo() {} for (var i = 0; i < 1_000_000; ++i) foo(); test262 reports no changes in either direction, apart from a speedup :^)
2021-10-05 07:44:58 +00:00
// https://tc39.es/ecma262/#sec-class-definitions-static-semantics-early-errors
LibJS: Disallow static methods named prototype in classes
2021-07-11 23:30:04 +00:00
// ClassElement : static MethodDefinition
// It is a Syntax Error if PropName of MethodDefinition is "prototype".
if (is_static && name == "prototype"sv)
syntax_error("Classes may not have a static property named 'prototype'");
LibJS: Allow defining class fields with "keyword" names
2021-11-26 22:30:29 +00:00
} else if ((match(TokenType::ParenOpen) || match(TokenType::Equals) || match(TokenType::Semicolon) || match(TokenType::CurlyClose)) && (is_static || is_async || method_kind != ClassMethod::Kind::Method)) {
LibJS: Allow class methods named "get", "set" or "static"
2021-07-25 09:50:12 +00:00
switch (method_kind) {
case ClassMethod::Kind::Method:
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (is_async) {
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
name = "async"sv;
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
is_async = false;
} else {
VERIFY(is_static);
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
name = "static"sv;
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
is_static = false;
}
LibJS: Allow class methods named "get", "set" or "static"
2021-07-25 09:50:12 +00:00
break;
case ClassMethod::Kind::Getter:
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
name = "get"sv;
LibJS: Allow class methods named "get", "set" or "static"
2021-07-25 09:50:12 +00:00
method_kind = ClassMethod::Kind::Method;
break;
case ClassMethod::Kind::Setter:
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
name = "set"sv;
LibJS: Allow class methods named "get", "set" or "static"
2021-07-25 09:50:12 +00:00
method_kind = ClassMethod::Kind::Method;
break;
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
property_key = create_ast_node<StringLiteral>({ m_source_code, rule_start.position(), position() }, name);
LibJS: Add static initializers to classes
2021-10-20 19:29:47 +00:00
} else if (match(TokenType::CurlyOpen) && is_static) {
auto static_start = push_start();
consume(TokenType::CurlyOpen);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto static_init_block = create_ast_node<FunctionBody>({ m_source_code, rule_start.position(), position() });
LibJS: Add static initializers to classes
2021-10-20 19:29:47 +00:00
TemporaryChange break_context_rollback(m_state.in_break_context, false);
TemporaryChange continue_context_rollback(m_state.in_continue_context, false);
TemporaryChange function_context_rollback(m_state.in_function_context, false);
TemporaryChange generator_function_context_rollback(m_state.in_generator_function_context, false);
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
TemporaryChange async_function_context_rollback(m_state.await_expression_is_valid, false);
LibJS: Add support for await expressions
2021-11-09 20:52:21 +00:00
TemporaryChange class_field_initializer_rollback(m_state.in_class_field_initializer, true);
TemporaryChange class_static_init_block_rollback(m_state.in_class_static_init_block, true);
LibJS: Allow super property lookup and new.target in static init blocks
2021-12-19 01:06:22 +00:00
TemporaryChange super_property_access_rollback(m_state.allow_super_property_lookup, true);
LibJS: Add static initializers to classes
2021-10-20 19:29:47 +00:00
ScopePusher static_init_scope = ScopePusher::static_init_block_scope(*this, *static_init_block);
parse_statement_list(static_init_block);
consume(TokenType::CurlyClose);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
elements.append(create_ast_node<StaticInitializer>({ m_source_code, static_start.position(), position() }, move(static_init_block), static_init_scope.contains_direct_call_to_eval()));
LibJS: Add static initializers to classes
2021-10-20 19:29:47 +00:00
continue;
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
} else {
expected("property key");
}
// Constructor may be a StringLiteral or an Identifier.
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
if (!is_static && name == "constructor"sv) {
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
if (method_kind != ClassMethod::Kind::Method)
syntax_error("Class constructor may not be an accessor");
if (!constructor.is_null())
syntax_error("Classes may not have more than one constructor");
LibJS: Parse generator functions in class expressions too
2021-07-02 10:07:00 +00:00
if (is_generator)
syntax_error("Class constructor may not be a generator");
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (is_async)
syntax_error("Class constructor may not be async");
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
is_constructor = true;
}
}
if (match(TokenType::ParenOpen)) {
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 16:56:49 +00:00
u8 parse_options = FunctionNodeParseOptions::AllowSuperPropertyLookup;
LibJS: Allow class methods named "get", "set" or "static"
2021-07-25 09:50:12 +00:00
if (!super_class.is_null() && !is_static && is_constructor)
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 16:56:49 +00:00
parse_options |= FunctionNodeParseOptions::AllowSuperConstructorCall;
LibJS: Move checks for invalid getter/setter params to parse_function_node This allows us to provide better error messages as we can point the syntax error location to the exact first invalid parameter instead of always the end of the function within a object literal or class definition. Before this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Object setter property must have one argument (line: 1, column: 28) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Class setter method must have one argument (line: 1, column: 26) After this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 23) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 21) The only possible downside of this change is that class getters/setters and functions in objects are not distinguished in the message anymore - I don't think that's important though, and classes are (mostly) just syntactic sugar anyway.
2020-10-20 17:32:51 +00:00
if (method_kind == ClassMethod::Kind::Getter)
parse_options |= FunctionNodeParseOptions::IsGetterFunction;
if (method_kind == ClassMethod::Kind::Setter)
parse_options |= FunctionNodeParseOptions::IsSetterFunction;
LibJS: Parse generator functions in class expressions too
2021-07-02 10:07:00 +00:00
if (is_generator)
parse_options |= FunctionNodeParseOptions::IsGeneratorFunction;
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (is_async)
parse_options |= FunctionNodeParseOptions::IsAsyncFunction;
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto function = parse_function_node<FunctionExpression>(parse_options, function_start);
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
if (is_constructor) {
constructor = move(function);
} else if (!property_key.is_null()) {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
elements.append(create_ast_node<ClassMethod>({ m_source_code, rule_start.position(), position() }, property_key.release_nonnull(), move(function), method_kind, is_static));
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
} else {
syntax_error("No key for class method");
}
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
} else if (is_generator || is_async) {
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
expected("ParenOpen");
consume();
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
} else if (property_key.is_null()) {
expected("property key");
consume();
} else {
if (name == "constructor"sv)
syntax_error("Class cannot have field named 'constructor'");
RefPtr<Expression> initializer;
LibJS: Use ClassFieldInitializerStatement for class fields This is necessary as we might have to perform named evaluation with the field name. Ideally we would also skip some setup parts of the function like function_declaration_instantiation however this would require bigger changes to ECMAScriptFunctionObject.
2021-10-13 17:59:38 +00:00
bool contains_direct_call_to_eval = false;
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
if (match(TokenType::Equals)) {
consume();
TemporaryChange super_property_access_rollback(m_state.allow_super_property_lookup, true);
TemporaryChange field_initializer_rollback(m_state.in_class_field_initializer, true);
LibJS: Use ClassFieldInitializerStatement for class fields This is necessary as we might have to perform named evaluation with the field name. Ideally we would also skip some setup parts of the function like function_declaration_instantiation however this would require bigger changes to ECMAScriptFunctionObject.
2021-10-13 17:59:38 +00:00
auto class_field_scope = ScopePusher::class_field_scope(*this);
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
initializer = parse_expression(2);
LibJS: Use ClassFieldInitializerStatement for class fields This is necessary as we might have to perform named evaluation with the field name. Ideally we would also skip some setup parts of the function like function_declaration_instantiation however this would require bigger changes to ECMAScriptFunctionObject.
2021-10-13 17:59:38 +00:00
contains_direct_call_to_eval = class_field_scope.contains_direct_call_to_eval();
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
elements.append(create_ast_node<ClassField>({ m_source_code, rule_start.position(), position() }, property_key.release_nonnull(), move(initializer), contains_direct_call_to_eval, is_static));
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
consume_or_insert_semicolon();
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
}
}
consume(TokenType::CurlyClose);
if (constructor.is_null()) {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto constructor_body = create_ast_node<BlockStatement>({ m_source_code, rule_start.position(), position() });
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
if (!super_class.is_null()) {
// Set constructor to the result of parsing the source text
// constructor(... args){ super (...args);}
LibJS: Use a synthetic constructor if class with parent doesn't have one We already did this but it called the @@iterator method of %Array.prototype% visible to the user for example by overriding that method. This should not be visible so we use a special version of SuperCall now.
2022-08-20 15:27:02 +00:00
// However: The most notable distinction is that while the aforementioned ECMAScript
// source text observably calls the @@iterator method on %Array.prototype%,
// this function does not.
// So we use a custom version of SuperCall which doesn't use the @@iterator
// method on %Array.prototype% visibly.
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString argument_name = "args";
LibJS: Make SuperCall a proper AST node and clean up evaluation
2021-07-02 17:30:38 +00:00
auto super_call = create_ast_node<SuperCall>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Use a synthetic constructor if class with parent doesn't have one We already did this but it called the @@iterator method of %Array.prototype% visible to the user for example by overriding that method. This should not be visible so we use a special version of SuperCall now.
2022-08-20 15:27:02 +00:00
SuperCall::IsPartOfSyntheticConstructor::Yes,
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
CallExpression::Argument { create_ast_node<Identifier>({ m_source_code, rule_start.position(), position() }, "args"), true });
LibJS: Replace the custom unwind mechanism with completions :^) This includes: - Parsing proper LabelledStatements with try_parse_labelled_statement() - Removing LabelableStatement - Implementing the LoopEvaluation semantics via loop_evaluation() in each IterationStatement subclass; and IterationStatement evaluation via {For,ForIn,ForOf,ForAwaitOf,While,DoWhile}Statement::execute() - Updating ReturnStatement, BreakStatement and ContinueStatement to return the appropriate completion types - Basically reimplementing TryStatement and SwitchStatement according to the spec, using completions - Honoring result completion types in AsyncBlockStart and OrdinaryCallEvaluateBody - Removing any uses of the VM unwind mechanism - most importantly, VM::throw_exception() now exclusively sets an exception and no longer triggers any unwinding mechanism. However, we already did a good job updating all of LibWeb and userland applications to not use it, and the few remaining uses elsewhere don't rely on unwinding AFAICT.
2022-01-05 18:11:16 +00:00
// NOTE: While the JS approximation above doesn't do `return super(...args)`, the
// abstract closure is expected to capture and return the result, so we do need a
// return statement here to create the correct completion.
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
constructor_body->append(create_ast_node<ReturnStatement>({ m_source_code, rule_start.position(), position() }, move(super_call)));
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
LibJS: Use an enum class instead of 'bool is_generator' This avoid confusion in the order of the multiple boolean parameters that exist.
2021-06-10 23:08:05 +00:00
constructor = create_ast_node<FunctionExpression>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() }, class_name, "",
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
move(constructor_body), Vector { FunctionParameter { move(argument_name), nullptr, true } }, 0, FunctionKind::Normal,
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 10:43:38 +00:00
/* is_strict_mode */ true, /* might_need_arguments_object */ false, /* contains_direct_call_to_eval */ false);
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
} else {
LibJS: Use an enum class instead of 'bool is_generator' This avoid confusion in the order of the multiple boolean parameters that exist.
2021-06-10 23:08:05 +00:00
constructor = create_ast_node<FunctionExpression>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() }, class_name, "",
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
move(constructor_body), Vector<FunctionParameter> {}, 0, FunctionKind::Normal,
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 10:43:38 +00:00
/* is_strict_mode */ true, /* might_need_arguments_object */ false, /* contains_direct_call_to_eval */ false);
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
}
}
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
// We could be in a subclass defined within the main class so must move all non declared private names to outer.
for (auto& private_name : referenced_private_names) {
if (found_private_names.contains(private_name))
continue;
if (outer_referenced_private_names)
outer_referenced_private_names->set(private_name);
else // FIXME: Make these error appear in the appropriate places.
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Reference to undeclared private field or method '{}'", private_name));
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
}
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto function_start_offset = rule_start.position().offset;
auto function_end_offset = position().offset - m_state.current_token.trivia().length();
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
auto source_text = DeprecatedString { m_state.lexer.source().substring_view(function_start_offset, function_end_offset - function_start_offset) };
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ClassExpression>({ m_source_code, rule_start.position(), position() }, move(class_name), move(source_text), move(constructor), move(super_class), move(elements));
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
}
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
Parser::PrimaryExpressionParseResult Parser::parse_primary_expression()
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Fix parsing of `if (typeof "foo" === "string")` Move parsing of unary expressions into the primary expression parsing step so that `typeof` takes precedence over `===` in the above example.
2020-03-25 08:51:54 +00:00
if (match_unary_prefixed_expression())
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_unary_prefixed_expression() };
LibJS: Fix parsing of `if (typeof "foo" === "string")` Move parsing of unary expressions into the primary expression parsing step so that `typeof` takes precedence over `===` in the above example.
2020-03-25 08:51:54 +00:00
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
auto try_arrow_function_parse_or_fail = [this](Position const& position, bool expect_paren, bool is_async = false) -> RefPtr<FunctionExpression> {
if (try_parse_arrow_function_expression_failed_at_position(position))
return nullptr;
auto arrow_function = try_parse_arrow_function_expression(expect_paren, is_async);
if (arrow_function)
return arrow_function;
set_try_parse_arrow_function_expression_failed_at_position(position, true);
return nullptr;
};
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
switch (m_state.current_token.type()) {
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
case TokenType::ParenOpen: {
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 20:41:51 +00:00
auto paren_position = position();
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume(TokenType::ParenOpen);
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
if ((match(TokenType::ParenClose) || match_identifier() || match(TokenType::TripleDot) || match(TokenType::CurlyOpen) || match(TokenType::BracketOpen))) {
if (auto arrow_function_result = try_arrow_function_parse_or_fail(paren_position, true))
LibJS: Treat arrow expression as function and stop parsing after
2021-07-28 15:11:33 +00:00
return { arrow_function_result.release_nonnull(), false };
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
}
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
auto expression = parse_expression(0);
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume(TokenType::ParenClose);
Only apply auto-naming of function expressions based on syntax The auto naming of function expressions is a purely syntactic decision, so shouldn't be decided based on the dynamic type of an assignment. This moves the decision making into the parser. One icky hack is that we add a field to FunctionExpression to indicate whether we can autoname. The real solution is to actually generate a CompoundExpression node so that the parser can make the correct decision, however this would have a potentially significant run time cost. This does not correct the behaviour for class expressions. Patch from Anonymous.
2021-03-22 11:44:07 +00:00
if (is<FunctionExpression>(*expression)) {
LibJS: Treat yield as an identifier in more non-generator contexts And disallow some cases where we are in a generator context.
2021-07-28 23:45:55 +00:00
auto& function = static_cast<FunctionExpression&>(*expression);
if (function.kind() == FunctionKind::Generator && function.name() == "yield"sv)
syntax_error("function is not allowed to be called 'yield' in this context", function.source_range().start);
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (function.kind() == FunctionKind::Async && function.name() == "await"sv)
syntax_error("function is not allowed to be called 'await' in this context", function.source_range().start);
Only apply auto-naming of function expressions based on syntax The auto naming of function expressions is a purely syntactic decision, so shouldn't be decided based on the dynamic type of an assignment. This moves the decision making into the parser. One icky hack is that we add a field to FunctionExpression to indicate whether we can autoname. The real solution is to actually generate a CompoundExpression node so that the parser can make the correct decision, however this would have a potentially significant run time cost. This does not correct the behaviour for class expressions. Patch from Anonymous.
2021-03-22 11:44:07 +00:00
}
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { move(expression) };
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Parse "this" as ThisExpression
2020-04-12 22:42:14 +00:00
case TokenType::This:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return { create_ast_node<ThisExpression>({ m_source_code, rule_start.position(), position() }) };
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
case TokenType::Class:
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_class_expression(false) };
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
case TokenType::Super:
consume();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (!m_state.allow_super_property_lookup)
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
syntax_error("'super' keyword unexpected here");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return { create_ast_node<SuperExpression>({ m_source_code, rule_start.position(), position() }) };
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
case TokenType::EscapedKeyword:
LibJS: Allow escaped 'async' as identifier Since 'async' is only special if it occurs before a function it can be used as escaped identifier in all cases.
2021-11-26 22:25:10 +00:00
if (match_invalid_escaped_keyword())
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
syntax_error("Keyword must not contain escaped characters");
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
[[fallthrough]];
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
case TokenType::Identifier: {
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
read_as_identifier:;
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
if (auto arrow_function_result = try_arrow_function_parse_or_fail(position(), false))
return { arrow_function_result.release_nonnull(), false };
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 20:41:51 +00:00
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
auto string = m_state.current_token.value();
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
// This could be 'eval' or 'arguments' and thus needs a custom check (`eval[1] = true`)
if (m_state.strict_mode && (string == "let" || is_strict_reserved_word(string)))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Identifier must not be a reserved word in strict mode ('{}')", string));
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
return { parse_identifier() };
LibJS: Add support for arrow functions
2020-03-30 13:26:09 +00:00
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
case TokenType::NumericLiteral:
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return { create_ast_node<NumericLiteral>({ m_source_code, rule_start.position(), position() }, consume_and_validate_numeric_literal().double_value()) };
LibJS: Add BigInt
2020-06-06 00:14:10 +00:00
case TokenType::BigIntLiteral:
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return { create_ast_node<BigIntLiteral>({ m_source_code, rule_start.position(), position() }, consume().value()) };
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
case TokenType::BoolLiteral:
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return { create_ast_node<BooleanLiteral>({ m_source_code, rule_start.position(), position() }, consume().bool_value()) };
LibJS: Implement basic lexing + parsing of StringLiteral This still includes the double-quote characters (") but at least the AST comes out right.
2020-03-12 12:05:06 +00:00
case TokenType::StringLiteral:
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_string_literal(consume()) };
LibJS: Implement null and undefined literals
2020-03-15 21:32:34 +00:00
case TokenType::NullLiteral:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return { create_ast_node<NullLiteral>({ m_source_code, rule_start.position(), position() }) };
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
case TokenType::CurlyOpen:
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_object_expression() };
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
case TokenType::Async: {
auto lookahead_token = next_token();
// No valid async function (arrow or not) can have a line terminator after the async since asi would kick in.
if (lookahead_token.trivia_contains_line_terminator())
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
goto read_as_identifier;
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
if (lookahead_token.type() == TokenType::Function)
return { parse_function_node<FunctionExpression>() };
if (lookahead_token.type() == TokenType::ParenOpen) {
if (auto arrow_function_result = try_arrow_function_parse_or_fail(position(), true, true))
return { arrow_function_result.release_nonnull(), false };
} else if (lookahead_token.is_identifier_name()) {
if (auto arrow_function_result = try_arrow_function_parse_or_fail(position(), false, true))
return { arrow_function_result.release_nonnull(), false };
}
goto read_as_identifier;
}
LibJS: Parse FunctionExpressions FunctionExpression is mostly like FunctionDeclaration, except the name is optional. Share the parsing logic in parse_function_node<NodeType>. This allows us to do nice things like: document.addEventListener("DOMContentLoaded", function() { alert("Hello friends!"); });
2020-03-19 10:52:56 +00:00
case TokenType::Function:
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_function_node<FunctionExpression>() };
LibJS: Parse ArrayExpression and start implementing Array objects Note that property lookup is not functional yet.
2020-03-20 19:29:57 +00:00
case TokenType::BracketOpen:
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_array_expression() };
LibJS: Lex and parse regex literals, add RegExp objects This adds regex parsing/lexing, as well as a relatively empty RegExpObject. The purpose of this patch is to allow the engine to not get hung up on parsing regexes. This will aid in finding new syntax errors (say, from google or twitter) without having to replace all of their regexes first!
2020-06-03 23:05:49 +00:00
case TokenType::RegexLiteral:
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_regexp_literal() };
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
case TokenType::TemplateLiteralStart:
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_template_literal(false) };
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
case TokenType::New: {
auto new_start = position();
auto new_target_result = try_parse_new_target_expression();
if (!new_target_result.is_null()) {
LibJS: Add missing steps and spec comments to PerformEval While adding spec comments to PerformEval, I noticed we were missing multiple steps. Namely, these were: - Checking if the host will allow us to compile the string (allowing LibWeb to perform CSP for eval) - The parser's initial state depending on the environment around us on direct eval: - Allowing new.target via eval in functions - Allowing super calls and super properties via eval in classes - Disallowing the use of the arguments object in class field initializers at eval's parse time - Setting ScriptOrModule of eval's execution context The spec allows us to apply the additional parsing steps in any order. The method I have gone with is passing in a struct to the parser's constructor, which overrides the parser's initial state to (dis)allow the things stated above from the get-go.
2022-04-09 23:55:45 +00:00
if (!m_state.in_function_context && !m_state.in_eval_function_context && !m_state.in_class_static_init_block)
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
syntax_error("'new.target' not allowed outside of a function", new_start);
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { new_target_result.release_nonnull() };
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
}
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_new_expression() };
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
}
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
case TokenType::Import: {
auto lookahead_token = next_token();
if (lookahead_token.type() == TokenType::ParenOpen)
return { parse_import_call() };
LibJS: Don't VERIFY that the token after 'import' is one of '.' and '(' Although those are the only valid options parse_primary_expression is sometimes called when only an expression is valid which means it did not check match_expression and might fail the now removed VERIFY.
2021-12-29 11:15:29 +00:00
if (lookahead_token.type() == TokenType::Period) {
if (auto import_meta = try_parse_import_meta_expression()) {
if (m_program_type != Program::Type::Module)
syntax_error("import.meta is only allowed in modules");
return { import_meta.release_nonnull() };
}
} else {
consume();
expected("import.meta or import call");
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
}
break;
}
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
case TokenType::Yield:
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (!m_state.in_generator_function_context)
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
goto read_as_identifier;
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
return { parse_yield_expression(), false };
LibJS: Add support for await expressions
2021-11-09 20:52:21 +00:00
case TokenType::Await:
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
if (!m_state.await_expression_is_valid)
LibJS: Add support for await expressions
2021-11-09 20:52:21 +00:00
goto read_as_identifier;
return { parse_await_expression() };
LibJS: Add support for the '#privateName in obj' expression
2021-10-14 00:05:24 +00:00
case TokenType::PrivateIdentifier:
if (!is_private_identifier_valid())
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Reference to undeclared private field or method '{}'", m_state.current_token.value()));
LibJS: Give the undeclared private identifier error more precedence Before this could give the `must be followed by in` error before the undeclared private identifier error. Fixing the `in` error would not have resolved the other error so this order makes the errors more actionable.
2022-11-17 09:26:27 +00:00
if (next_token().type() != TokenType::In)
syntax_error("Cannot have a private identifier in expression if not followed by 'in'");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return { create_ast_node<PrivateIdentifier>({ m_source_code, rule_start.position(), position() }, consume().value()) };
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
default:
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
if (match_identifier_name())
goto read_as_identifier;
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
break;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
expected("primary expression");
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return { create_ast_node<ErrorExpression>({ m_source_code, rule_start.position(), position() }) };
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Lex and parse regex literals, add RegExp objects This adds regex parsing/lexing, as well as a relatively empty RegExpObject. The purpose of this patch is to allow the engine to not get hung up on parsing regexes. This will aid in finding new syntax errors (say, from google or twitter) without having to replace all of their regexes first!
2020-06-03 23:05:49 +00:00
NonnullRefPtr<RegExpLiteral> Parser::parse_regexp_literal()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Rename RegExpLiteral m_content to m_pattern This is what we call it elsewhere, let's be consistent.
2021-05-10 10:56:08 +00:00
auto pattern = consume().value();
// Remove leading and trailing slash.
pattern = pattern.substring_view(1, pattern.length() - 2);
LibJS: Parse RegExp literals at AST creation time, not execution time The spec requires that invalid RegExp literals must cause a Syntax Error before the JavaScript is executed. See: https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors This is explicitly tested in the RegExp/property-escapes test262 tests. For example, see unsupported-property-Line_Break.js: $DONOTEVALUATE(); /\p{Line_Break}/u; That RegExp literal is invalid because Line_Break is not a supported Unicode property. $DONOTEVALUATE() just throws an exception when it is executed. The test expects that this file will fail to be parsed. Note that RegExp patterns can still be parsed at execution time by way of "new RegExp(...)".
2021-07-29 14:34:37 +00:00
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
auto flags = DeprecatedString::empty();
LibJS: Parse RegExp literals at AST creation time, not execution time The spec requires that invalid RegExp literals must cause a Syntax Error before the JavaScript is executed. See: https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors This is explicitly tested in the RegExp/property-escapes test262 tests. For example, see unsupported-property-Line_Break.js: $DONOTEVALUATE(); /\p{Line_Break}/u; That RegExp literal is invalid because Line_Break is not a supported Unicode property. $DONOTEVALUATE() just throws an exception when it is executed. The test expects that this file will fail to be parsed. Note that RegExp patterns can still be parsed at execution time by way of "new RegExp(...)".
2021-07-29 14:34:37 +00:00
auto parsed_flags = RegExpObject::default_flags;
LibJS: Make invalid RegExp flags a SyntaxError at parse time This patch changes the validation of RegExp flags (checking for invalid and duplicate values) from a SyntaxError at runtime to a SyntaxError at parse time - it's not something that's supposed to be catchable. As a nice side effect, this simplifies the RegExpObject constructor a bit, as it can no longer throw an exception and doesn't have to validate the flags itself.
2021-05-10 11:01:38 +00:00
if (match(TokenType::RegexFlags)) {
auto flags_start = position();
flags = consume().value();
LibJS: Parse RegExp literals at AST creation time, not execution time The spec requires that invalid RegExp literals must cause a Syntax Error before the JavaScript is executed. See: https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors This is explicitly tested in the RegExp/property-escapes test262 tests. For example, see unsupported-property-Line_Break.js: $DONOTEVALUATE(); /\p{Line_Break}/u; That RegExp literal is invalid because Line_Break is not a supported Unicode property. $DONOTEVALUATE() just throws an exception when it is executed. The test expects that this file will fail to be parsed. Note that RegExp patterns can still be parsed at execution time by way of "new RegExp(...)".
2021-07-29 14:34:37 +00:00
auto parsed_flags_or_error = regex_flags_from_string(flags);
if (parsed_flags_or_error.is_error())
syntax_error(parsed_flags_or_error.release_error(), flags_start);
else
parsed_flags = parsed_flags_or_error.release_value();
LibJS: Make invalid RegExp flags a SyntaxError at parse time This patch changes the validation of RegExp flags (checking for invalid and duplicate values) from a SyntaxError at runtime to a SyntaxError at parse time - it's not something that's supposed to be catchable. As a nice side effect, this simplifies the RegExpObject constructor a bit, as it can no longer throw an exception and doesn't have to validate the flags itself.
2021-05-10 11:01:38 +00:00
}
LibJS: Parse RegExp literals at AST creation time, not execution time The spec requires that invalid RegExp literals must cause a Syntax Error before the JavaScript is executed. See: https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors This is explicitly tested in the RegExp/property-escapes test262 tests. For example, see unsupported-property-Line_Break.js: $DONOTEVALUATE(); /\p{Line_Break}/u; That RegExp literal is invalid because Line_Break is not a supported Unicode property. $DONOTEVALUATE() just throws an exception when it is executed. The test expects that this file will fail to be parsed. Note that RegExp patterns can still be parsed at execution time by way of "new RegExp(...)".
2021-07-29 14:34:37 +00:00
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
DeprecatedString parsed_pattern;
LibJS: Hook up the 'v' (unicodeSets) RegExp flag
2022-07-16 05:44:03 +00:00
auto parsed_pattern_result = parse_regex_pattern(pattern, parsed_flags.has_flag_set(ECMAScriptFlags::Unicode), parsed_flags.has_flag_set(ECMAScriptFlags::UnicodeSets));
if (parsed_pattern_result.is_error()) {
syntax_error(parsed_pattern_result.release_error().error, rule_start.position());
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
parsed_pattern = DeprecatedString::empty();
LibJS: Hook up the 'v' (unicodeSets) RegExp flag
2022-07-16 05:44:03 +00:00
} else {
parsed_pattern = parsed_pattern_result.release_value();
}
LibJS: Parse RegExp literals at AST creation time, not execution time The spec requires that invalid RegExp literals must cause a Syntax Error before the JavaScript is executed. See: https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors This is explicitly tested in the RegExp/property-escapes test262 tests. For example, see unsupported-property-Line_Break.js: $DONOTEVALUATE(); /\p{Line_Break}/u; That RegExp literal is invalid because Line_Break is not a supported Unicode property. $DONOTEVALUATE() just throws an exception when it is executed. The test expects that this file will fail to be parsed. Note that RegExp patterns can still be parsed at execution time by way of "new RegExp(...)".
2021-07-29 14:34:37 +00:00
auto parsed_regex = Regex<ECMA262>::parse_pattern(parsed_pattern, parsed_flags);
if (parsed_regex.error != regex::Error::NoError)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("RegExp compile error: {}", Regex<ECMA262>(parsed_regex, parsed_pattern, parsed_flags).error_string()), rule_start.position());
LibJS: Parse RegExp literals at AST creation time, not execution time The spec requires that invalid RegExp literals must cause a Syntax Error before the JavaScript is executed. See: https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors This is explicitly tested in the RegExp/property-escapes test262 tests. For example, see unsupported-property-Line_Break.js: $DONOTEVALUATE(); /\p{Line_Break}/u; That RegExp literal is invalid because Line_Break is not a supported Unicode property. $DONOTEVALUATE() just throws an exception when it is executed. The test expects that this file will fail to be parsed. Note that RegExp patterns can still be parsed at execution time by way of "new RegExp(...)".
2021-07-29 14:34:37 +00:00
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
SourceRange range { m_source_code, rule_start.position(), position() };
Everywhere: Rename to_{string => deprecated_string}() where applicable This will make it easier to support both string types at the same time while we convert code, and tracking down remaining uses. One big exception is Value::to_string() in LibJS, where the name is dictated by the ToString AO.
2022-12-06 01:12:49 +00:00
return create_ast_node<RegExpLiteral>(move(range), move(parsed_regex), move(parsed_pattern), move(parsed_flags), pattern.to_deprecated_string(), move(flags));
LibJS: Lex and parse regex literals, add RegExp objects This adds regex parsing/lexing, as well as a relatively empty RegExpObject. The purpose of this patch is to allow the engine to not get hung up on parsing regexes. This will aid in finding new syntax errors (say, from google or twitter) without having to replace all of their regexes first!
2020-06-03 23:05:49 +00:00
}
LibJS: Allow CallExpressions as lhs of assignments in most cases Although not quite like the spec says the web reality is that a lhs target of CallExpression should not give a SyntaxError but only a ReferenceError once executed.
2022-11-30 00:47:25 +00:00
static bool is_simple_assignment_target(Expression const& expression, bool allow_web_reality_call_expression = true)
{
return is<Identifier>(expression) || is<MemberExpression>(expression) || (allow_web_reality_call_expression && is<CallExpression>(expression));
}
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 10:23:53 +00:00
NonnullRefPtr<Expression> Parser::parse_unary_prefixed_expression()
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto precedence = g_operator_precedence.get(m_state.current_token.type());
auto associativity = operator_associativity(m_state.current_token.type());
switch (m_state.current_token.type()) {
LibJS: Enforce that ++/-- operand is an identifier or member expression
2020-04-29 20:39:00 +00:00
case TokenType::PlusPlus: {
consume();
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
auto rhs_start = position();
LibJS: Enforce that ++/-- operand is an identifier or member expression
2020-04-29 20:39:00 +00:00
auto rhs = parse_expression(precedence, associativity);
LibJS: Allow CallExpressions as lhs of assignments in most cases Although not quite like the spec says the web reality is that a lhs target of CallExpression should not give a SyntaxError but only a ReferenceError once executed.
2022-11-30 00:47:25 +00:00
if (!is_simple_assignment_target(*rhs))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Right-hand side of prefix increment operator must be identifier or member expression, got {}", rhs->class_name()), rhs_start);
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
if (m_state.strict_mode && is<Identifier>(*rhs)) {
auto& identifier = static_cast<Identifier&>(*rhs);
auto& name = identifier.string();
check_identifier_name_for_assignment_validity(name);
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UpdateExpression>({ m_source_code, rule_start.position(), position() }, UpdateOp::Increment, move(rhs), true);
LibJS: Enforce that ++/-- operand is an identifier or member expression
2020-04-29 20:39:00 +00:00
}
case TokenType::MinusMinus: {
consume();
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
auto rhs_start = position();
LibJS: Enforce that ++/-- operand is an identifier or member expression
2020-04-29 20:39:00 +00:00
auto rhs = parse_expression(precedence, associativity);
LibJS: Allow CallExpressions as lhs of assignments in most cases Although not quite like the spec says the web reality is that a lhs target of CallExpression should not give a SyntaxError but only a ReferenceError once executed.
2022-11-30 00:47:25 +00:00
if (!is_simple_assignment_target(*rhs))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Right-hand side of prefix decrement operator must be identifier or member expression, got {}", rhs->class_name()), rhs_start);
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
if (m_state.strict_mode && is<Identifier>(*rhs)) {
auto& identifier = static_cast<Identifier&>(*rhs);
auto& name = identifier.string();
check_identifier_name_for_assignment_validity(name);
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UpdateExpression>({ m_source_code, rule_start.position(), position() }, UpdateOp::Decrement, move(rhs), true);
LibJS: Enforce that ++/-- operand is an identifier or member expression
2020-04-29 20:39:00 +00:00
}
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
case TokenType::ExclamationMark:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UnaryExpression>({ m_source_code, rule_start.position(), position() }, UnaryOp::Not, parse_expression(precedence, associativity));
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
case TokenType::Tilde:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UnaryExpression>({ m_source_code, rule_start.position(), position() }, UnaryOp::BitwiseNot, parse_expression(precedence, associativity));
LibJS: Implement unary plus / minus
2020-04-02 16:58:39 +00:00
case TokenType::Plus:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UnaryExpression>({ m_source_code, rule_start.position(), position() }, UnaryOp::Plus, parse_expression(precedence, associativity));
LibJS: Implement unary plus / minus
2020-04-02 16:58:39 +00:00
case TokenType::Minus:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UnaryExpression>({ m_source_code, rule_start.position(), position() }, UnaryOp::Minus, parse_expression(precedence, associativity));
LibJS: Implement typeof operator
2020-03-17 19:33:32 +00:00
case TokenType::Typeof:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UnaryExpression>({ m_source_code, rule_start.position(), position() }, UnaryOp::Typeof, parse_expression(precedence, associativity));
LibJS: Implement void operator
2020-04-15 16:55:03 +00:00
case TokenType::Void:
consume();
LibJS: Don't treat yield after void as identifier in generator functions
2021-12-19 01:28:18 +00:00
// FIXME: This check is really hiding the fact that we don't deal with different expressions correctly.
if (match(TokenType::Yield) && m_state.in_generator_function_context)
syntax_error("'yield' is not an identifier in generator function context");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UnaryExpression>({ m_source_code, rule_start.position(), position() }, UnaryOp::Void, parse_expression(precedence, associativity));
LibJS: Disallow unqualified deletes in strict mode
2021-07-11 23:32:24 +00:00
case TokenType::Delete: {
LibJS: Implement basic support for the "delete" operator It turns out "delete" is actually a unary op :) This patch implements deletion of object properties, it doesn't yet work for casually deleting properties from the global object. When deleting a property from an object, we switch that object to having a unique shape, no longer sharing shapes with others. Once an object has a unique shape, it no longer needs to care about shape transitions.
2020-04-26 11:53:40 +00:00
consume();
LibJS: Disallow unqualified deletes in strict mode
2021-07-11 23:32:24 +00:00
auto rhs_start = position();
auto rhs = parse_expression(precedence, associativity);
if (is<Identifier>(*rhs) && m_state.strict_mode) {
syntax_error("Delete of an unqualified identifier in strict mode.", rhs_start);
}
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
if (is<MemberExpression>(*rhs)) {
auto& member_expression = static_cast<MemberExpression const&>(*rhs);
if (member_expression.ends_in_private_name())
syntax_error("Private fields cannot be deleted");
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UnaryExpression>({ m_source_code, rule_start.position(), position() }, UnaryOp::Delete, move(rhs));
LibJS: Disallow unqualified deletes in strict mode
2021-07-11 23:32:24 +00:00
}
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
default:
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
expected("primary expression");
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ErrorExpression>({ m_source_code, rule_start.position(), position() });
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
}
}
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
NonnullRefPtr<Expression> Parser::parse_property_key()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
if (match(TokenType::StringLiteral)) {
return parse_string_literal(consume());
} else if (match(TokenType::NumericLiteral)) {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<NumericLiteral>({ m_source_code, rule_start.position(), position() }, consume().double_value());
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
} else if (match(TokenType::BigIntLiteral)) {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BigIntLiteral>({ m_source_code, rule_start.position(), position() }, consume().value());
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
} else if (match(TokenType::BracketOpen)) {
consume(TokenType::BracketOpen);
LibJS: Parse only AssignmentExpressions in ComputedPropertyNames The property name in an object literal can either be a literal or a computed name, in which case any AssignmentExpression can be used, we now only parse AssignmentExpression instead of the previous incorrect behaviour which allowed any Expression (Specifically, comma expressions).
2021-06-11 15:43:28 +00:00
auto result = parse_expression(2);
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
consume(TokenType::BracketClose);
return result;
} else {
if (!match_identifier_name())
expected("IdentifierName");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<StringLiteral>({ m_source_code, rule_start.position(), position() }, consume().value());
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
}
}
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 10:23:53 +00:00
NonnullRefPtr<ObjectExpression> Parser::parse_object_expression()
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume(TokenType::CurlyOpen);
LibJS: Parse object expressions
2020-03-21 00:29:00 +00:00
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
NonnullRefPtrVector<ObjectProperty> properties;
ObjectProperty::Type property_type;
LibJS: Allow 'name = value' in object literals as the spec does Currently, these are _always_ a syntax error, future commits will make it valid in certain contexts.
2021-07-11 10:18:30 +00:00
Optional<SourceRange> invalid_object_literal_property_range;
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
auto skip_to_next_property = [&] {
while (!done() && !match(TokenType::Comma) && !match(TokenType::CurlyOpen))
consume();
};
LibJS: Add object literal getter/setter shorthand Adds support for the following syntax: let foo = { get x() { // ... }, set x(value) { // ... } }
2020-05-22 00:28:28 +00:00
LibJS: Disallow multiple __proto__ keys in object expression
2021-07-11 23:32:01 +00:00
// It is a Syntax Error if PropertyNameList of PropertyDefinitionList contains any duplicate
// entries for "__proto__" and at least two of those entries were obtained from productions of
LibJS: Rename PropertyName to PropertyKey Let's use the same name as the spec. :^)
2021-10-24 14:01:24 +00:00
// the form PropertyDefinition : PropertyKey : AssignmentExpression .
LibJS: Disallow multiple __proto__ keys in object expression
2021-07-11 23:32:01 +00:00
bool has_direct_proto_property = false;
LibJS: Break loop on EOF when parsing object expression
2020-04-07 17:29:37 +00:00
while (!done() && !match(TokenType::CurlyClose)) {
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
property_type = ObjectProperty::Type::KeyValue;
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
RefPtr<Expression> property_key;
LibJS: Implement computed properties in object expressions
2020-04-23 18:37:53 +00:00
RefPtr<Expression> property_value;
LibJS: Rename FunctionKind::{Regular => Normal} This is what CreateDynamicFunction calls it.
2022-01-14 23:30:02 +00:00
FunctionKind function_kind { FunctionKind::Normal };
LibJS: Add spreading in object literals Supports spreading strings, arrays, and other objects within object literals.
2020-04-28 04:52:47 +00:00
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
if (match(TokenType::TripleDot)) {
consume();
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
property_key = parse_expression(4);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
properties.append(create_ast_node<ObjectProperty>({ m_source_code, rule_start.position(), position() }, *property_key, nullptr, ObjectProperty::Type::Spread, false));
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
if (!match(TokenType::Comma))
break;
consume(TokenType::Comma);
continue;
}
LibJS: Disallow multiple __proto__ keys in object expression
2021-07-11 23:32:01 +00:00
auto type = m_state.current_token.type();
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto function_start = position();
LibJS: Disallow multiple __proto__ keys in object expression
2021-07-11 23:32:01 +00:00
LibJS: Parse async generator functions
2021-11-15 00:53:24 +00:00
if (match(TokenType::Async)) {
auto lookahead_token = next_token();
LibJS: Allow object properties called 'async'
2021-11-26 15:40:27 +00:00
if (lookahead_token.type() != TokenType::ParenOpen && lookahead_token.type() != TokenType::Colon
LibJS: Disallow shorthand properties with reserved names
2021-11-26 22:36:43 +00:00
&& lookahead_token.type() != TokenType::Comma && lookahead_token.type() != TokenType::CurlyClose
&& lookahead_token.type() != TokenType::Async
LibJS: Allow object properties called 'async'
2021-11-26 15:40:27 +00:00
&& !lookahead_token.trivia_contains_line_terminator()) {
LibJS: Parse async generator functions
2021-11-15 00:53:24 +00:00
consume(TokenType::Async);
function_kind = FunctionKind::Async;
}
}
LibJS: Parse generator functions in object literals Also add some parser tests
2021-06-14 10:22:59 +00:00
if (match(TokenType::Asterisk)) {
consume();
property_type = ObjectProperty::Type::KeyValue;
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
property_key = parse_property_key();
LibJS: Rename FunctionKind::{Regular => Normal} This is what CreateDynamicFunction calls it.
2022-01-14 23:30:02 +00:00
VERIFY(function_kind == FunctionKind::Normal || function_kind == FunctionKind::Async);
function_kind = function_kind == FunctionKind::Normal ? FunctionKind::Generator : FunctionKind::AsyncGenerator;
LibJS: Treat yield as an identifier in more non-generator contexts And disallow some cases where we are in a generator context.
2021-07-28 23:45:55 +00:00
} else if (match_identifier()) {
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
auto identifier = consume();
LibJS: Parse async generator functions
2021-11-15 00:53:24 +00:00
if (identifier.original_value() == "get"sv && match_property_key()) {
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
property_type = ObjectProperty::Type::Getter;
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
property_key = parse_property_key();
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
} else if (identifier.original_value() == "set"sv && match_property_key()) {
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
property_type = ObjectProperty::Type::Setter;
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
property_key = parse_property_key();
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
} else {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
property_key = create_ast_node<StringLiteral>({ m_source_code, rule_start.position(), position() }, identifier.value());
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
property_value = create_ast_node<Identifier>({ m_source_code, rule_start.position(), position() }, identifier.DeprecatedFlyString_value());
LibJS: Add object literal getter/setter shorthand Adds support for the following syntax: let foo = { get x() { // ... }, set x(value) { // ... } }
2020-05-22 00:28:28 +00:00
}
LibJS: Allow parsing numeric and string literals in object expressions Also updated the object-basic.js test to include this change
2020-04-06 20:17:05 +00:00
} else {
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
property_key = parse_property_key();
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
}
LibJS: Implement the object literal __proto__ property key special case
2022-03-05 21:44:49 +00:00
// 4. Else if propKey is the String value "__proto__" and if IsComputedPropertyKey of PropertyName is false, then
// a. Let isProtoSetter be true.
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
bool is_proto = (type == TokenType::StringLiteral || type == TokenType::Identifier) && is<StringLiteral>(*property_key) && static_cast<StringLiteral const&>(*property_key).value() == "__proto__";
LibJS: Disallow multiple __proto__ keys in object expression
2021-07-11 23:32:01 +00:00
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
if (property_type == ObjectProperty::Type::Getter || property_type == ObjectProperty::Type::Setter) {
if (!match(TokenType::ParenOpen)) {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("'(' for object getter or setter property");
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
skip_to_next_property();
LibJS: Add object literal getter/setter shorthand Adds support for the following syntax: let foo = { get x() { // ... }, set x(value) { // ... } }
2020-05-22 00:28:28 +00:00
continue;
}
LibJS: Allow parsing numeric and string literals in object expressions Also updated the object-basic.js test to include this change
2020-04-06 20:17:05 +00:00
}
LibJS: Allow 'name = value' in object literals as the spec does Currently, these are _always_ a syntax error, future commits will make it valid in certain contexts.
2021-07-11 10:18:30 +00:00
if (match(TokenType::Equals)) {
// Not a valid object literal, but a valid assignment target
consume();
// Parse the expression and throw it away
auto expression = parse_expression(2);
if (!invalid_object_literal_property_range.has_value())
invalid_object_literal_property_range = expression->source_range();
} else if (match(TokenType::ParenOpen)) {
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
VERIFY(property_key);
LibJS: Move checks for invalid getter/setter params to parse_function_node This allows us to provide better error messages as we can point the syntax error location to the exact first invalid parameter instead of always the end of the function within a object literal or class definition. Before this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Object setter property must have one argument (line: 1, column: 28) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Class setter method must have one argument (line: 1, column: 26) After this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 23) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 21) The only possible downside of this change is that class getters/setters and functions in objects are not distinguished in the message anymore - I don't think that's important though, and classes are (mostly) just syntactic sugar anyway.
2020-10-20 17:32:51 +00:00
u8 parse_options = FunctionNodeParseOptions::AllowSuperPropertyLookup;
if (property_type == ObjectProperty::Type::Getter)
parse_options |= FunctionNodeParseOptions::IsGetterFunction;
if (property_type == ObjectProperty::Type::Setter)
parse_options |= FunctionNodeParseOptions::IsSetterFunction;
LibJS: Parse async generator functions
2021-11-15 00:53:24 +00:00
if (function_kind == FunctionKind::Generator || function_kind == FunctionKind::AsyncGenerator)
LibJS: Parse generator functions in object literals Also add some parser tests
2021-06-14 10:22:59 +00:00
parse_options |= FunctionNodeParseOptions::IsGeneratorFunction;
LibJS: Parse async generator functions
2021-11-15 00:53:24 +00:00
if (function_kind == FunctionKind::Async || function_kind == FunctionKind::AsyncGenerator)
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
parse_options |= FunctionNodeParseOptions::IsAsyncFunction;
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto function = parse_function_node<FunctionExpression>(parse_options, function_start);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
properties.append(create_ast_node<ObjectProperty>({ m_source_code, rule_start.position(), position() }, *property_key, function, property_type, true));
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
} else if (match(TokenType::Colon)) {
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
if (!property_key) {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("a property name");
LibJS: Replace some parser assertions by syntax errors When parsing JavaScript, we can get pretty much any sequnce of tokens, and we shouldn't crash if it's not something that we normally expect. Instead, emit syntax errors.
2020-06-01 14:08:34 +00:00
skip_to_next_property();
continue;
}
LibJS: Rewrite Parser.parse_object_expression() This rewrite drastically increases the accuracy of object literals. Additionally, an "assertIsSyntaxError" function has been added to test-common.js to assist in testing syntax errors.
2020-05-29 05:50:06 +00:00
consume();
LibJS: Disallow multiple __proto__ keys in object expression
2021-07-11 23:32:01 +00:00
if (is_proto) {
if (has_direct_proto_property)
syntax_error("Property name '__proto__' must not appear more than once in object literal");
has_direct_proto_property = true;
}
LibJS: Implement the object literal __proto__ property key special case
2022-03-05 21:44:49 +00:00
if (is_proto && property_type == ObjectProperty::Type::KeyValue)
property_type = ObjectProperty::Type::ProtoSetter;
LibJS: Don't update names of resulting functions in object expression The only cases where the name should be set is if the function comes from a direct anonymous function expression.
2022-12-13 00:30:32 +00:00
auto rhs_expression = parse_expression(2);
bool is_method = is<FunctionExpression>(*rhs_expression);
properties.append(create_ast_node<ObjectProperty>({ m_source_code, rule_start.position(), position() }, *property_key, move(rhs_expression), property_type, is_method));
Everywhere: Rename JS::PropertyKey variables from property_{name => key} PropertyKey used to be called PropertyName, but got renamed. Let's update all the variables of this type as well.
2022-02-06 15:59:04 +00:00
} else if (property_key && property_value) {
if (m_state.strict_mode && is<StringLiteral>(*property_key)) {
auto& string_literal = static_cast<StringLiteral const&>(*property_key);
LibJS: Disallow shorthand properties with reserved names
2021-11-26 22:36:43 +00:00
if (is_strict_reserved_word(string_literal.value()))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("'{}' is a reserved keyword", string_literal.value()));
LibJS: Disallow shorthand properties with reserved names
2021-11-26 22:36:43 +00:00
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
properties.append(create_ast_node<ObjectProperty>({ m_source_code, rule_start.position(), position() }, *property_key, *property_value, property_type, false));
LibJS: Replace some parser assertions by syntax errors When parsing JavaScript, we can get pretty much any sequnce of tokens, and we shouldn't crash if it's not something that we normally expect. Instead, emit syntax errors.
2020-06-01 14:08:34 +00:00
} else {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("a property");
LibJS: Replace some parser assertions by syntax errors When parsing JavaScript, we can get pretty much any sequnce of tokens, and we shouldn't crash if it's not something that we normally expect. Instead, emit syntax errors.
2020-06-01 14:08:34 +00:00
skip_to_next_property();
continue;
LibJS: Parse object expressions
2020-03-21 00:29:00 +00:00
}
if (!match(TokenType::Comma))
break;
consume(TokenType::Comma);
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume(TokenType::CurlyClose);
LibJS: Remove m_first_invalid_property_range from ObjectExpression This was state only used by the parser to output an error with appropriate location. This shrinks the size of ObjectExpression from 120 bytes down to just 56. This saves roughly 2.5 MiB when loading twitter.
2022-11-27 01:24:38 +00:00
if (invalid_object_literal_property_range.has_value()) {
size_t object_expression_offset = rule_start.position().offset;
VERIFY(!m_state.invalid_property_range_in_object_expression.contains(object_expression_offset));
m_state.invalid_property_range_in_object_expression.set(object_expression_offset, invalid_object_literal_property_range->start);
}
LibJS: Call shrink_to_fit() on various Vectors created during parse Vectors that stick around in the AST were wasting a fair bit of memory due to the growth padding we keep by default. This patch goes after some of these vectors with the shrink_to_fit() stick to reduce waste. Since the AST can stay around for a long time, it is worth making an effort to shrink it down when we have a chance.
2022-11-26 19:45:06 +00:00
properties.shrink_to_fit();
LibJS: Allow 'name = value' in object literals as the spec does Currently, these are _always_ a syntax error, future commits will make it valid in certain contexts.
2021-07-11 10:18:30 +00:00
return create_ast_node<ObjectExpression>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Remove m_first_invalid_property_range from ObjectExpression This was state only used by the parser to output an error with appropriate location. This shrinks the size of ObjectExpression from 120 bytes down to just 56. This saves roughly 2.5 MiB when loading twitter.
2022-11-27 01:24:38 +00:00
move(properties));
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Parse ArrayExpression and start implementing Array objects Note that property lookup is not functional yet.
2020-03-20 19:29:57 +00:00
NonnullRefPtr<ArrayExpression> Parser::parse_array_expression()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Parse ArrayExpression and start implementing Array objects Note that property lookup is not functional yet.
2020-03-20 19:29:57 +00:00
consume(TokenType::BracketOpen);
LibJS: Support empty values in array expression
2020-04-15 19:09:06 +00:00
Vector<RefPtr<Expression>> elements;
LibJS: Add spreading in array literals Implement the syntax and behavor necessary to support array literals such as [...[1, 2, 3]]. A type error is thrown if the target of the spread operator does not evaluate to an array (though it should eventually just check for an iterable). Note that the spread token's name is TripleDot, since the '...' token is used for two features: spread and rest. Calling it anything involving 'spread' or 'rest' would be a bit confusing.
2020-04-27 06:05:37 +00:00
while (match_expression() || match(TokenType::TripleDot) || match(TokenType::Comma)) {
LibJS: Support empty values in array expression
2020-04-15 19:09:06 +00:00
RefPtr<Expression> expression;
LibJS: Add spreading in array literals Implement the syntax and behavor necessary to support array literals such as [...[1, 2, 3]]. A type error is thrown if the target of the spread operator does not evaluate to an array (though it should eventually just check for an iterable). Note that the spread token's name is TripleDot, since the '...' token is used for two features: spread and rest. Calling it anything involving 'spread' or 'rest' would be a bit confusing.
2020-04-27 06:05:37 +00:00
if (match(TokenType::TripleDot)) {
consume(TokenType::TripleDot);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
expression = create_ast_node<SpreadExpression>({ m_source_code, rule_start.position(), position() }, parse_expression(2));
LibJS: Add spreading in array literals Implement the syntax and behavor necessary to support array literals such as [...[1, 2, 3]]. A type error is thrown if the target of the spread operator does not evaluate to an array (though it should eventually just check for an iterable). Note that the spread token's name is TripleDot, since the '...' token is used for two features: spread and rest. Calling it anything involving 'spread' or 'rest' would be a bit confusing.
2020-04-27 06:05:37 +00:00
} else if (match_expression()) {
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
expression = parse_expression(2);
LibJS: Add spreading in array literals Implement the syntax and behavor necessary to support array literals such as [...[1, 2, 3]]. A type error is thrown if the target of the spread operator does not evaluate to an array (though it should eventually just check for an iterable). Note that the spread token's name is TripleDot, since the '...' token is used for two features: spread and rest. Calling it anything involving 'spread' or 'rest' would be a bit confusing.
2020-04-27 06:05:37 +00:00
}
LibJS: Support empty values in array expression
2020-04-15 19:09:06 +00:00
elements.append(expression);
LibJS: Parse ArrayExpression and start implementing Array objects Note that property lookup is not functional yet.
2020-03-20 19:29:57 +00:00
if (!match(TokenType::Comma))
break;
consume(TokenType::Comma);
}
consume(TokenType::BracketClose);
LibJS: Call shrink_to_fit() on various Vectors created during parse Vectors that stick around in the AST were wasting a fair bit of memory due to the growth padding we keep by default. This patch goes after some of these vectors with the shrink_to_fit() stick to reduce waste. Since the AST can stay around for a long time, it is worth making an effort to shrink it down when we have a chance.
2022-11-26 19:45:06 +00:00
elements.shrink_to_fit();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ArrayExpression>({ m_source_code, rule_start.position(), position() }, move(elements));
LibJS: Parse ArrayExpression and start implementing Array objects Note that property lookup is not functional yet.
2020-03-20 19:29:57 +00:00
}
LibJS: Allow invalid string in tagged template literals Since tagged template literals can inspect the raw string it is not a syntax error to have invalid escapes. However the cooked value should be `undefined`. We accomplish this by tracking whether parse_string_literal fails and then using a NullLiteral (since UndefinedLiteral is not a thing) and finally converting null in tagged template execution to undefined.
2022-08-17 00:04:27 +00:00
NonnullRefPtr<StringLiteral> Parser::parse_string_literal(Token const& token, StringLiteralType string_literal_type, bool* contains_invalid_escape)
LibJS: Handle hex and unicode escape sequences in string literals Introduces the following syntax: '\x55' '\u26a0' '\u{1f41e}'
2020-05-17 06:27:25 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Handle hex and unicode escape sequences in string literals Introduces the following syntax: '\x55' '\u26a0' '\u{1f41e}'
2020-05-17 06:27:25 +00:00
auto status = Token::StringValueStatus::Ok;
auto string = token.string_value(status);
LibJS: Allow invalid string in tagged template literals Since tagged template literals can inspect the raw string it is not a syntax error to have invalid escapes. However the cooked value should be `undefined`. We accomplish this by tracking whether parse_string_literal fails and then using a NullLiteral (since UndefinedLiteral is not a thing) and finally converting null in tagged template execution to undefined.
2022-08-17 00:04:27 +00:00
// NOTE: Tagged templates should not fail on invalid strings as their raw contents can still be accessed.
LibJS: Handle hex and unicode escape sequences in string literals Introduces the following syntax: '\x55' '\u26a0' '\u{1f41e}'
2020-05-17 06:27:25 +00:00
if (status != Token::StringValueStatus::Ok) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
DeprecatedString message;
LibJS: Support LegacyOctalEscapeSequence in string literals https://tc39.es/ecma262/#sec-additional-syntax-string-literals The syntax and semantics of 11.8.4 is extended as follows except that this extension is not allowed for strict mode code: Syntax EscapeSequence:: CharacterEscapeSequence LegacyOctalEscapeSequence NonOctalDecimalEscapeSequence HexEscapeSequence UnicodeEscapeSequence LegacyOctalEscapeSequence:: OctalDigit [lookahead ∉ OctalDigit] ZeroToThree OctalDigit [lookahead ∉ OctalDigit] FourToSeven OctalDigit ZeroToThree OctalDigit OctalDigit ZeroToThree :: one of 0 1 2 3 FourToSeven :: one of 4 5 6 7 NonOctalDecimalEscapeSequence :: one of 8 9 This definition of EscapeSequence is not used in strict mode or when parsing TemplateCharacter. Note It is possible for string literals to precede a Use Strict Directive that places the enclosing code in strict mode, and implementations must take care to not use this extended definition of EscapeSequence with such literals. For example, attempting to parse the following source text must fail: function invalid() { "\7"; "use strict"; }
2020-10-24 12:30:57 +00:00
if (status == Token::StringValueStatus::LegacyOctalEscapeSequence) {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
m_state.string_legacy_octal_escape_sequence_in_scope = true;
LibJS: Allow invalid string in tagged template literals Since tagged template literals can inspect the raw string it is not a syntax error to have invalid escapes. However the cooked value should be `undefined`. We accomplish this by tracking whether parse_string_literal fails and then using a NullLiteral (since UndefinedLiteral is not a thing) and finally converting null in tagged template execution to undefined.
2022-08-17 00:04:27 +00:00
// It is a Syntax Error if the [Tagged] parameter was not set and Template{Head, Middle, Tail} Contains NotEscapeSequence.
if (string_literal_type != StringLiteralType::Normal)
LibJS: Support LegacyOctalEscapeSequence in string literals https://tc39.es/ecma262/#sec-additional-syntax-string-literals The syntax and semantics of 11.8.4 is extended as follows except that this extension is not allowed for strict mode code: Syntax EscapeSequence:: CharacterEscapeSequence LegacyOctalEscapeSequence NonOctalDecimalEscapeSequence HexEscapeSequence UnicodeEscapeSequence LegacyOctalEscapeSequence:: OctalDigit [lookahead ∉ OctalDigit] ZeroToThree OctalDigit [lookahead ∉ OctalDigit] FourToSeven OctalDigit ZeroToThree OctalDigit OctalDigit ZeroToThree :: one of 0 1 2 3 FourToSeven :: one of 4 5 6 7 NonOctalDecimalEscapeSequence :: one of 8 9 This definition of EscapeSequence is not used in strict mode or when parsing TemplateCharacter. Note It is possible for string literals to precede a Use Strict Directive that places the enclosing code in strict mode, and implementations must take care to not use this extended definition of EscapeSequence with such literals. For example, attempting to parse the following source text must fail: function invalid() { "\7"; "use strict"; }
2020-10-24 12:30:57 +00:00
message = "Octal escape sequence not allowed in template literal";
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
else if (m_state.strict_mode)
LibJS: Support LegacyOctalEscapeSequence in string literals https://tc39.es/ecma262/#sec-additional-syntax-string-literals The syntax and semantics of 11.8.4 is extended as follows except that this extension is not allowed for strict mode code: Syntax EscapeSequence:: CharacterEscapeSequence LegacyOctalEscapeSequence NonOctalDecimalEscapeSequence HexEscapeSequence UnicodeEscapeSequence LegacyOctalEscapeSequence:: OctalDigit [lookahead ∉ OctalDigit] ZeroToThree OctalDigit [lookahead ∉ OctalDigit] FourToSeven OctalDigit ZeroToThree OctalDigit OctalDigit ZeroToThree :: one of 0 1 2 3 FourToSeven :: one of 4 5 6 7 NonOctalDecimalEscapeSequence :: one of 8 9 This definition of EscapeSequence is not used in strict mode or when parsing TemplateCharacter. Note It is possible for string literals to precede a Use Strict Directive that places the enclosing code in strict mode, and implementations must take care to not use this extended definition of EscapeSequence with such literals. For example, attempting to parse the following source text must fail: function invalid() { "\7"; "use strict"; }
2020-10-24 12:30:57 +00:00
message = "Octal escape sequence in string literal not allowed in strict mode";
} else if (status == Token::StringValueStatus::MalformedHexEscape || status == Token::StringValueStatus::MalformedUnicodeEscape) {
LibJS: Handle hex and unicode escape sequences in string literals Introduces the following syntax: '\x55' '\u26a0' '\u{1f41e}'
2020-05-17 06:27:25 +00:00
auto type = status == Token::StringValueStatus::MalformedUnicodeEscape ? "unicode" : "hexadecimal";
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
message = DeprecatedString::formatted("Malformed {} escape sequence", type);
LibJS: Handle hex and unicode escape sequences in string literals Introduces the following syntax: '\x55' '\u26a0' '\u{1f41e}'
2020-05-17 06:27:25 +00:00
} else if (status == Token::StringValueStatus::UnicodeEscapeOverflow) {
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 20:31:20 +00:00
message = "Unicode code_point must not be greater than 0x10ffff in escape sequence";
LibJS: Support LegacyOctalEscapeSequence in string literals https://tc39.es/ecma262/#sec-additional-syntax-string-literals The syntax and semantics of 11.8.4 is extended as follows except that this extension is not allowed for strict mode code: Syntax EscapeSequence:: CharacterEscapeSequence LegacyOctalEscapeSequence NonOctalDecimalEscapeSequence HexEscapeSequence UnicodeEscapeSequence LegacyOctalEscapeSequence:: OctalDigit [lookahead ∉ OctalDigit] ZeroToThree OctalDigit [lookahead ∉ OctalDigit] FourToSeven OctalDigit ZeroToThree OctalDigit OctalDigit ZeroToThree :: one of 0 1 2 3 FourToSeven :: one of 4 5 6 7 NonOctalDecimalEscapeSequence :: one of 8 9 This definition of EscapeSequence is not used in strict mode or when parsing TemplateCharacter. Note It is possible for string literals to precede a Use Strict Directive that places the enclosing code in strict mode, and implementations must take care to not use this extended definition of EscapeSequence with such literals. For example, attempting to parse the following source text must fail: function invalid() { "\7"; "use strict"; }
2020-10-24 12:30:57 +00:00
} else {
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 19:42:32 +00:00
VERIFY_NOT_REACHED();
LibJS: Handle hex and unicode escape sequences in string literals Introduces the following syntax: '\x55' '\u26a0' '\u{1f41e}'
2020-05-17 06:27:25 +00:00
}
LibJS: Allow invalid string in tagged template literals Since tagged template literals can inspect the raw string it is not a syntax error to have invalid escapes. However the cooked value should be `undefined`. We accomplish this by tracking whether parse_string_literal fails and then using a NullLiteral (since UndefinedLiteral is not a thing) and finally converting null in tagged template execution to undefined.
2022-08-17 00:04:27 +00:00
if (!message.is_empty()) {
if (contains_invalid_escape != nullptr) {
VERIFY(string_literal_type == StringLiteralType::TaggedTemplate);
*contains_invalid_escape = true;
} else {
syntax_error(message, Position { token.line_number(), token.line_column() });
}
}
LibJS: Handle hex and unicode escape sequences in string literals Introduces the following syntax: '\x55' '\u26a0' '\u{1f41e}'
2020-05-17 06:27:25 +00:00
}
LibJS: Add strict mode Adds the ability for a scope (either a function or the entire program) to be in strict mode. Scopes default to non-strict mode. There are two ways to determine the strict-ness of the JS engine: 1. In the parser, this can be accessed with the parser_state variable m_is_strict_mode boolean. If true, the Parser is currently parsing in strict mode. This is done so that the Parser can generate syntax errors at parse time, which is required in some cases. 2. With Interpreter.is_strict_mode(). This allows strict mode checking at runtime as opposed to compile time. Additionally, in order to test this, a global isStrictMode() function has been added to the JS ReplObject under the test-mode flag.
2020-05-28 05:22:08 +00:00
LibJS: Remove is_use_strict_directive for all StringLiterals This value was only used in the parser so no need to have this in every string literal in the ast.
2022-11-27 01:21:25 +00:00
return create_ast_node<StringLiteral>({ m_source_code, rule_start.position(), position() }, string);
LibJS: Handle hex and unicode escape sequences in string literals Introduces the following syntax: '\x55' '\u26a0' '\u{1f41e}'
2020-05-17 06:27:25 +00:00
}
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
NonnullRefPtr<TemplateLiteral> Parser::parse_template_literal(bool is_tagged)
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
consume(TokenType::TemplateLiteralStart);
NonnullRefPtrVector<Expression> expressions;
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
NonnullRefPtrVector<Expression> raw_strings;
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto append_empty_string = [this, &rule_start, &expressions, &raw_strings, is_tagged]() {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto string_literal = create_ast_node<StringLiteral>({ m_source_code, rule_start.position(), position() }, "");
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
expressions.append(string_literal);
if (is_tagged)
raw_strings.append(string_literal);
};
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
LibJS: Implement tagged template literals (foo`bar`) To make processing tagged template literals easier, template literals will now add one empty StringLiteral before and after each template expression *if* there's no other string - e.g.: `${foo}` -> "", foo, "" `test${foo}${bar}test` -> "test", foo, "", bar, "test" This also matches the behaviour of many other parsers.
2020-05-06 09:17:35 +00:00
if (!match(TokenType::TemplateLiteralString))
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
append_empty_string();
LibJS: Implement tagged template literals (foo`bar`) To make processing tagged template literals easier, template literals will now add one empty StringLiteral before and after each template expression *if* there's no other string - e.g.: `${foo}` -> "", foo, "" `test${foo}${bar}test` -> "test", foo, "", bar, "test" This also matches the behaviour of many other parsers.
2020-05-06 09:17:35 +00:00
LibJS: Fix Parser.parse_template_literal looping forever parse_template_literal now breaks out of the loop if it sees something it doesn't expect. Additionally, it now checks for EOFs.
2020-06-03 22:36:25 +00:00
while (!done() && !match(TokenType::TemplateLiteralEnd) && !match(TokenType::UnterminatedTemplateLiteral)) {
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
if (match(TokenType::TemplateLiteralString)) {
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
auto token = consume();
LibJS: Allow invalid string in tagged template literals Since tagged template literals can inspect the raw string it is not a syntax error to have invalid escapes. However the cooked value should be `undefined`. We accomplish this by tracking whether parse_string_literal fails and then using a NullLiteral (since UndefinedLiteral is not a thing) and finally converting null in tagged template execution to undefined.
2022-08-17 00:04:27 +00:00
bool contains_invalid_escape = false;
auto parsed_string_value = parse_string_literal(token,
is_tagged ? StringLiteralType::TaggedTemplate : StringLiteralType::NonTaggedTemplate,
is_tagged ? &contains_invalid_escape : nullptr);
// An invalid string leads to a cooked value of `undefined` but still gives the raw string.
if (contains_invalid_escape)
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
expressions.append(create_ast_node<NullLiteral>({ m_source_code, rule_start.position(), position() }));
LibJS: Allow invalid string in tagged template literals Since tagged template literals can inspect the raw string it is not a syntax error to have invalid escapes. However the cooked value should be `undefined`. We accomplish this by tracking whether parse_string_literal fails and then using a NullLiteral (since UndefinedLiteral is not a thing) and finally converting null in tagged template execution to undefined.
2022-08-17 00:04:27 +00:00
else
expressions.append(move(parsed_string_value));
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
if (is_tagged)
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
raw_strings.append(create_ast_node<StringLiteral>({ m_source_code, rule_start.position(), position() }, token.raw_template_value()));
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
} else if (match(TokenType::TemplateLiteralExprStart)) {
consume(TokenType::TemplateLiteralExprStart);
if (match(TokenType::TemplateLiteralExprEnd)) {
syntax_error("Empty template literal expression block");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<TemplateLiteral>({ m_source_code, rule_start.position(), position() }, expressions);
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
}
expressions.append(parse_expression(0));
if (match(TokenType::UnterminatedTemplateLiteral)) {
syntax_error("Unterminated template literal");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<TemplateLiteral>({ m_source_code, rule_start.position(), position() }, expressions);
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
}
consume(TokenType::TemplateLiteralExprEnd);
LibJS: Implement tagged template literals (foo`bar`) To make processing tagged template literals easier, template literals will now add one empty StringLiteral before and after each template expression *if* there's no other string - e.g.: `${foo}` -> "", foo, "" `test${foo}${bar}test` -> "test", foo, "", bar, "test" This also matches the behaviour of many other parsers.
2020-05-06 09:17:35 +00:00
if (!match(TokenType::TemplateLiteralString))
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
append_empty_string();
LibJS: Fix Parser.parse_template_literal looping forever parse_template_literal now breaks out of the loop if it sees something it doesn't expect. Additionally, it now checks for EOFs.
2020-06-03 22:36:25 +00:00
} else {
expected("Template literal string or expression");
break;
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
}
}
if (match(TokenType::UnterminatedTemplateLiteral)) {
syntax_error("Unterminated template literal");
} else {
consume(TokenType::TemplateLiteralEnd);
}
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
if (is_tagged)
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<TemplateLiteral>({ m_source_code, rule_start.position(), position() }, expressions, raw_strings);
return create_ast_node<TemplateLiteral>({ m_source_code, rule_start.position(), position() }, expressions);
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
}
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
NonnullRefPtr<Expression> Parser::parse_expression(int min_precedence, Associativity associativity, ForbiddenTokens forbidden)
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
auto [expression, should_continue_parsing] = parse_primary_expression();
LibJS: Allow 'name = value' in object literals as the spec does Currently, these are _always_ a syntax error, future commits will make it valid in certain contexts.
2021-07-11 10:18:30 +00:00
auto check_for_invalid_object_property = [&](auto& expression) {
if (is<ObjectExpression>(*expression)) {
LibJS: Remove m_first_invalid_property_range from ObjectExpression This was state only used by the parser to output an error with appropriate location. This shrinks the size of ObjectExpression from 120 bytes down to just 56. This saves roughly 2.5 MiB when loading twitter.
2022-11-27 01:24:38 +00:00
if (auto start_offset = m_state.invalid_property_range_in_object_expression.get(expression->start_offset()); start_offset.has_value())
syntax_error("Invalid property in object literal", start_offset.value());
LibJS: Allow 'name = value' in object literals as the spec does Currently, these are _always_ a syntax error, future commits will make it valid in certain contexts.
2021-07-11 10:18:30 +00:00
}
};
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
if (is<Identifier>(*expression) && m_state.current_scope_pusher) {
auto identifier_instance = static_ptr_cast<Identifier>(expression);
auto function_scope = m_state.current_scope_pusher->last_function_scope();
auto function_parent_scope = function_scope ? function_scope->parent_scope() : nullptr;
bool has_not_been_declared_as_variable = true;
for (auto scope = m_state.current_scope_pusher; scope != function_parent_scope; scope = scope->parent_scope()) {
if (scope->has_declaration(identifier_instance->string())) {
has_not_been_declared_as_variable = false;
break;
}
}
if (has_not_been_declared_as_variable) {
if (identifier_instance->string() == "arguments"sv)
m_state.current_scope_pusher->set_contains_access_to_arguments_object();
}
}
LibJS: Implement tagged template literals (foo`bar`) To make processing tagged template literals easier, template literals will now add one empty StringLiteral before and after each template expression *if* there's no other string - e.g.: `${foo}` -> "", foo, "" `test${foo}${bar}test` -> "test", foo, "", bar, "test" This also matches the behaviour of many other parsers.
2020-05-06 09:17:35 +00:00
while (match(TokenType::TemplateLiteralStart)) {
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 23:34:14 +00:00
auto template_literal = parse_template_literal(true);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
expression = create_ast_node<TaggedTemplateLiteral>({ m_source_code, rule_start.position(), position() }, move(expression), move(template_literal));
LibJS: Implement tagged template literals (foo`bar`) To make processing tagged template literals easier, template literals will now add one empty StringLiteral before and after each template expression *if* there's no other string - e.g.: `${foo}` -> "", foo, "" `test${foo}${bar}test` -> "test", foo, "", bar, "test" This also matches the behaviour of many other parsers.
2020-05-06 09:17:35 +00:00
}
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
if (should_continue_parsing) {
while (match_secondary_expression(forbidden)) {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
int new_precedence = g_operator_precedence.get(m_state.current_token.type());
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
if (new_precedence < min_precedence)
break;
if (new_precedence == min_precedence && associativity == Associativity::Left)
break;
LibJS: Allow 'name = value' in object literals as the spec does Currently, these are _always_ a syntax error, future commits will make it valid in certain contexts.
2021-07-11 10:18:30 +00:00
check_for_invalid_object_property(expression);
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
Associativity new_associativity = operator_associativity(m_state.current_token.type());
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
auto result = parse_secondary_expression(move(expression), new_precedence, new_associativity, forbidden);
expression = result.expression;
forbidden = forbidden.merge(result.forbidden);
LibJS: Handle '++' and '--' more correctly within expression
2021-08-14 15:02:15 +00:00
while (match(TokenType::TemplateLiteralStart) && !is<UpdateExpression>(*expression)) {
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
auto template_literal = parse_template_literal(true);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
expression = create_ast_node<TaggedTemplateLiteral>({ m_source_code, rule_start.position(), position() }, move(expression), move(template_literal));
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
}
LibJS: Implement tagged template literals (foo`bar`) To make processing tagged template literals easier, template literals will now add one empty StringLiteral before and after each template expression *if* there's no other string - e.g.: `${foo}` -> "", foo, "" `test${foo}${bar}test` -> "test", foo, "", bar, "test" This also matches the behaviour of many other parsers.
2020-05-06 09:17:35 +00:00
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Allow 'name = value' in object literals as the spec does Currently, these are _always_ a syntax error, future commits will make it valid in certain contexts.
2021-07-11 10:18:30 +00:00
LibJS: Disallow standalone super expression
2021-08-14 15:08:09 +00:00
if (is<SuperExpression>(*expression))
syntax_error("'super' keyword unexpected here");
LibJS: Allow 'name = value' in object literals as the spec does Currently, these are _always_ a syntax error, future commits will make it valid in certain contexts.
2021-07-11 10:18:30 +00:00
check_for_invalid_object_property(expression);
LibJS: Make accessing the current function's arguments cheaper Instead of going through an environment record, make arguments of the currently executing function generate references via the argument index, which can later be resolved directly through the ExecutionContext.
2021-10-07 16:43:22 +00:00
if (is<CallExpression>(*expression) && m_state.current_scope_pusher) {
auto& callee = static_ptr_cast<CallExpression>(expression)->callee();
if (is<Identifier>(callee)) {
auto& identifier_instance = static_cast<Identifier const&>(callee);
if (identifier_instance.string() == "eval"sv) {
bool has_not_been_declared_as_variable = true;
for (auto scope = m_state.current_scope_pusher; scope; scope = scope->parent_scope()) {
if (scope->has_declaration(identifier_instance.string())) {
has_not_been_declared_as_variable = false;
break;
}
}
if (has_not_been_declared_as_variable)
m_state.current_scope_pusher->set_contains_direct_call_to_eval();
}
}
}
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
if (match(TokenType::Comma) && min_precedence <= 1) {
NonnullRefPtrVector<Expression> expressions;
expressions.append(expression);
while (match(TokenType::Comma)) {
consume();
expressions.append(parse_expression(2));
}
LibJS: Call shrink_to_fit() on various Vectors created during parse Vectors that stick around in the AST were wasting a fair bit of memory due to the growth padding we keep by default. This patch goes after some of these vectors with the shrink_to_fit() stick to reduce waste. Since the AST can stay around for a long time, it is worth making an effort to shrink it down when we have a chance.
2022-11-26 19:45:06 +00:00
expressions.shrink_to_fit();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
expression = create_ast_node<SequenceExpression>({ m_source_code, rule_start.position(), position() }, move(expressions));
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
return expression;
}
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
Parser::ExpressionResult Parser::parse_secondary_expression(NonnullRefPtr<Expression> lhs, int min_precedence, Associativity associativity, ForbiddenTokens forbidden)
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
switch (m_state.current_token.type()) {
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
case TokenType::Plus:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::Addition, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement +=, -=, *=, and /= assignment operators
2020-03-12 12:09:15 +00:00
case TokenType::PlusEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::AdditionAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
case TokenType::Minus:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::Subtraction, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement +=, -=, *=, and /= assignment operators
2020-03-12 12:09:15 +00:00
case TokenType::MinusEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::SubtractionAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Implement multiplication and division operators
2020-03-12 12:04:52 +00:00
case TokenType::Asterisk:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::Multiplication, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement +=, -=, *=, and /= assignment operators
2020-03-12 12:09:15 +00:00
case TokenType::AsteriskEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::MultiplicationAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Implement multiplication and division operators
2020-03-12 12:04:52 +00:00
case TokenType::Slash:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::Division, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement +=, -=, *=, and /= assignment operators
2020-03-12 12:09:15 +00:00
case TokenType::SlashEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::DivisionAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Add basic support for modulo (%) in binary expressions
2020-04-04 19:17:34 +00:00
case TokenType::Percent:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::Modulo, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement modulo assignment operator (%=)
2020-05-04 22:07:05 +00:00
case TokenType::PercentEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::ModuloAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Parse "this" as ThisExpression
2020-04-12 22:42:14 +00:00
case TokenType::DoubleAsterisk:
LibJS: Implement exponentiation (** operator)
2020-04-05 12:40:00 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::Exponentiation, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement exponentiation assignment operator (**=)
2020-05-04 22:03:35 +00:00
case TokenType::DoubleAsteriskEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::ExponentiationAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Parse > and < binary operators
2020-03-12 12:10:27 +00:00
case TokenType::GreaterThan:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::GreaterThan, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement <= and >= binary operators
2020-03-12 12:07:08 +00:00
case TokenType::GreaterThanEquals:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::GreaterThanEquals, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Parse > and < binary operators
2020-03-12 12:10:27 +00:00
case TokenType::LessThan:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::LessThan, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement <= and >= binary operators
2020-03-12 12:07:08 +00:00
case TokenType::LessThanEquals:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::LessThanEquals, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Parse === and !== binary operators
2020-03-12 12:11:33 +00:00
case TokenType::EqualsEqualsEquals:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::StrictlyEquals, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Parse === and !== binary operators
2020-03-12 12:11:33 +00:00
case TokenType::ExclamationMarkEqualsEquals:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::StrictlyInequals, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement abstract equality and inequality
2020-03-15 22:23:38 +00:00
case TokenType::EqualsEquals:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::LooselyEquals, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement abstract equality and inequality
2020-03-15 22:23:38 +00:00
case TokenType::ExclamationMarkEquals:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::LooselyInequals, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement 'in' operator
2020-04-23 15:06:01 +00:00
case TokenType::In:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::In, move(lhs), parse_expression(min_precedence, associativity));
LibJS: Implement the "instanceof" operator This operator walks the prototype chain of the RHS value and looks for a "prototype" property with the same value as the prototype of the LHS. This is pretty cool. :^)
2020-03-28 15:56:54 +00:00
case TokenType::Instanceof:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::InstanceOf, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Parse binary bitwise operators The Lexer and AST already have all the functionality required in place, so this is just updating Parser::match_secondary_expression() and Parser::parse_expression() to handle TokenType::{Ampersand,Pipe,Caret}, as well as adding some tests.
2020-04-03 12:02:31 +00:00
case TokenType::Ampersand:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::BitwiseAnd, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement bitwise assignment operators (&=, |=, ^=)
2020-05-04 21:34:45 +00:00
case TokenType::AmpersandEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::BitwiseAndAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Parse binary bitwise operators The Lexer and AST already have all the functionality required in place, so this is just updating Parser::match_secondary_expression() and Parser::parse_expression() to handle TokenType::{Ampersand,Pipe,Caret}, as well as adding some tests.
2020-04-03 12:02:31 +00:00
case TokenType::Pipe:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::BitwiseOr, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement bitwise assignment operators (&=, |=, ^=)
2020-05-04 21:34:45 +00:00
case TokenType::PipeEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::BitwiseOrAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Parse binary bitwise operators The Lexer and AST already have all the functionality required in place, so this is just updating Parser::match_secondary_expression() and Parser::parse_expression() to handle TokenType::{Ampersand,Pipe,Caret}, as well as adding some tests.
2020-04-03 12:02:31 +00:00
case TokenType::Caret:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::BitwiseXor, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement bitwise assignment operators (&=, |=, ^=)
2020-05-04 21:34:45 +00:00
case TokenType::CaretEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::BitwiseXorAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Implement bitwise left shift operator (<<)
2020-04-23 12:36:14 +00:00
case TokenType::ShiftLeft:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::LeftShift, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement bitwise left shift operator (<<)
2020-04-23 12:36:14 +00:00
case TokenType::ShiftLeftEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::LeftShiftAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Implement bitwise right shift operator (>>)
2020-04-23 12:45:19 +00:00
case TokenType::ShiftRight:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::RightShift, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement bitwise right shift operator (>>)
2020-04-23 12:45:19 +00:00
case TokenType::ShiftRightEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::RightShiftAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Implement bitwise unsigned right shift operator (>>>)
2020-04-23 14:43:10 +00:00
case TokenType::UnsignedShiftRight:
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BinaryExpression>({ m_source_code, rule_start.position(), position() }, BinaryOp::UnsignedRightShift, move(lhs), parse_expression(min_precedence, associativity, forbidden));
LibJS: Implement bitwise unsigned right shift operator (>>>)
2020-04-23 14:43:10 +00:00
case TokenType::UnsignedShiftRightEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::UnsignedRightShiftAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
case TokenType::ParenOpen:
return parse_call_expression(move(lhs));
case TokenType::Equals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::Assignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Implement basic MemberExpression parsing At last we can parse "hello friends".length :^)
2020-03-12 12:05:57 +00:00
case TokenType::Period:
consume();
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
if (match(TokenType::PrivateIdentifier)) {
if (!is_private_identifier_valid())
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Reference to undeclared private field or method '{}'", m_state.current_token.value()));
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
else if (is<SuperExpression>(*lhs))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Cannot access private field or method '{}' on super", m_state.current_token.value()));
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<MemberExpression>({ m_source_code, rule_start.position(), position() }, move(lhs), create_ast_node<PrivateIdentifier>({ m_source_code, rule_start.position(), position() }, consume().value()));
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
} else if (!match_identifier_name()) {
LibJS: Allow reserved words as keys in object expressions.
2020-04-18 18:31:27 +00:00
expected("IdentifierName");
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
}
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
return create_ast_node<MemberExpression>({ m_source_code, rule_start.position(), position() }, move(lhs), create_ast_node<Identifier>({ m_source_code, rule_start.position(), position() }, consume().DeprecatedFlyString_value()));
LibJS: Parse computed MemberExpressions MemberExpression comes in two flavors: computed: a[b] non-computed: a.b We can now parse both of the types. :^)
2020-03-20 19:51:03 +00:00
case TokenType::BracketOpen: {
consume(TokenType::BracketOpen);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto expression = create_ast_node<MemberExpression>({ m_source_code, rule_start.position(), position() }, move(lhs), parse_expression(0), true);
LibJS: Parse computed MemberExpressions MemberExpression comes in two flavors: computed: a[b] non-computed: a.b We can now parse both of the types. :^)
2020-03-20 19:51:03 +00:00
consume(TokenType::BracketClose);
return expression;
}
LibJS: Implement update expressions Note that currently only the non-prefixed variant is supported (i.e i++ not ++i), this variant returns the value of the argument before the update.
2020-03-12 11:45:45 +00:00
case TokenType::PlusPlus:
LibJS: Allow CallExpressions as lhs of assignments in most cases Although not quite like the spec says the web reality is that a lhs target of CallExpression should not give a SyntaxError but only a ReferenceError once executed.
2022-11-30 00:47:25 +00:00
if (!is_simple_assignment_target(*lhs))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Left-hand side of postfix increment operator must be identifier or member expression, got {}", lhs->class_name()));
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
if (m_state.strict_mode && is<Identifier>(*lhs)) {
auto& identifier = static_cast<Identifier&>(*lhs);
auto& name = identifier.string();
check_identifier_name_for_assignment_validity(name);
}
LibJS: Implement update expressions Note that currently only the non-prefixed variant is supported (i.e i++ not ++i), this variant returns the value of the argument before the update.
2020-03-12 11:45:45 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UpdateExpression>({ m_source_code, rule_start.position(), position() }, UpdateOp::Increment, move(lhs));
LibJS: Implement update expressions Note that currently only the non-prefixed variant is supported (i.e i++ not ++i), this variant returns the value of the argument before the update.
2020-03-12 11:45:45 +00:00
case TokenType::MinusMinus:
LibJS: Allow CallExpressions as lhs of assignments in most cases Although not quite like the spec says the web reality is that a lhs target of CallExpression should not give a SyntaxError but only a ReferenceError once executed.
2022-11-30 00:47:25 +00:00
if (!is_simple_assignment_target(*lhs))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Left-hand side of postfix increment operator must be identifier or member expression, got {}", lhs->class_name()));
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
if (m_state.strict_mode && is<Identifier>(*lhs)) {
auto& identifier = static_cast<Identifier&>(*lhs);
auto& name = identifier.string();
check_identifier_name_for_assignment_validity(name);
}
LibJS: Implement update expressions Note that currently only the non-prefixed variant is supported (i.e i++ not ++i), this variant returns the value of the argument before the update.
2020-03-12 11:45:45 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<UpdateExpression>({ m_source_code, rule_start.position(), position() }, UpdateOp::Decrement, move(lhs));
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
case TokenType::DoubleAmpersand: {
LibJS/Parser: Parse logical expressions
2020-03-15 21:35:22 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto expression = create_ast_node<LogicalExpression>({ m_source_code, rule_start.position(), position() }, LogicalOp::And, move(lhs), parse_expression(min_precedence, associativity, forbidden.forbid({ TokenType::DoubleQuestionMark })));
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
return { expression, { TokenType::DoubleQuestionMark } };
}
LibJS: Implement logical assignment operators (&&=, ||=, ??=) TC39 proposal, stage 4 as of 2020-07. https://tc39.es/proposal-logical-assignment/
2020-10-05 15:49:43 +00:00
case TokenType::DoubleAmpersandEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::AndAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
case TokenType::DoublePipe: {
LibJS/Parser: Parse logical expressions
2020-03-15 21:35:22 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto expression = create_ast_node<LogicalExpression>({ m_source_code, rule_start.position(), position() }, LogicalOp::Or, move(lhs), parse_expression(min_precedence, associativity, forbidden.forbid({ TokenType::DoubleQuestionMark })));
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
return { expression, { TokenType::DoubleQuestionMark } };
}
LibJS: Implement logical assignment operators (&&=, ||=, ??=) TC39 proposal, stage 4 as of 2020-07. https://tc39.es/proposal-logical-assignment/
2020-10-05 15:49:43 +00:00
case TokenType::DoublePipeEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::OrAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
case TokenType::DoubleQuestionMark: {
LibJS: Implement nullish coalescing operator (??)
2020-04-17 23:49:11 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto expression = create_ast_node<LogicalExpression>({ m_source_code, rule_start.position(), position() }, LogicalOp::NullishCoalescing, move(lhs), parse_expression(min_precedence, associativity, forbidden.forbid({ TokenType::DoubleAmpersand, TokenType::DoublePipe })));
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
return { expression, { TokenType::DoubleAmpersand, TokenType::DoublePipe } };
}
LibJS: Implement logical assignment operators (&&=, ||=, ??=) TC39 proposal, stage 4 as of 2020-07. https://tc39.es/proposal-logical-assignment/
2020-10-05 15:49:43 +00:00
case TokenType::DoubleQuestionMarkEquals:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_assignment_expression(AssignmentOp::NullishAssignment, move(lhs), min_precedence, associativity, forbidden);
LibJS: Implement ConditionalExpression (ternary "?:" operator)
2020-04-03 10:14:28 +00:00
case TokenType::QuestionMark:
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
return parse_conditional_expression(move(lhs), forbidden);
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
case TokenType::QuestionMarkPeriod:
// FIXME: This should allow `(new Foo)?.bar', but as our parser strips parenthesis,
// we can't really tell if `lhs' was parenthesized at this point.
if (is<NewExpression>(lhs.ptr())) {
syntax_error("'new' cannot be used with optional chaining", position());
consume();
return lhs;
}
return parse_optional_chain(move(lhs));
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
default:
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
expected("secondary expression");
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ErrorExpression>({ m_source_code, rule_start.position(), position() });
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
}
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 20:45:52 +00:00
bool Parser::is_private_identifier_valid() const
{
VERIFY(match(TokenType::PrivateIdentifier));
if (!m_state.referenced_private_names)
return false;
// We might not have hit the declaration yet so class will check this in the end
m_state.referenced_private_names->set(m_state.current_token.value());
return true;
}
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
RefPtr<BindingPattern> Parser::synthesize_binding_pattern(Expression const& expression)
{
VERIFY(is<ArrayExpression>(expression) || is<ObjectExpression>(expression));
// Clear any syntax error that has occurred in the range that 'expression' spans.
m_state.errors.remove_all_matching([range = expression.source_range()](auto const& error) {
return error.position.has_value() && range.contains(*error.position);
});
// Make a parser and parse the source for this expression as a binding pattern.
LibJS: Set Token's m_offset to the value's start index This makes much more sense than the current way of setting it to the Lexer's m_position after consuming the full value.
2022-01-18 23:44:33 +00:00
// NOTE: There's currently a fundamental problem that we pass the *next* (a.k.a. `current_token`)
// token's position to most nodes' SourceRange when using `rule_start.position(), position()`.
// This means that `source` will contain the subsequent token's trivia, if any (which is fine).
auto source_start_offset = expression.source_range().start.offset;
auto source_end_offset = expression.source_range().end.offset;
auto source = m_state.lexer.source().substring_view(source_start_offset, source_end_offset - source_start_offset);
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
Lexer lexer { source, m_state.lexer.filename(), expression.source_range().start.line, expression.source_range().start.column };
Parser parser { lexer };
parser.m_state.strict_mode = m_state.strict_mode;
parser.m_state.allow_super_property_lookup = m_state.allow_super_property_lookup;
parser.m_state.allow_super_constructor_call = m_state.allow_super_constructor_call;
parser.m_state.in_function_context = m_state.in_function_context;
parser.m_state.in_formal_parameter_context = m_state.in_formal_parameter_context;
parser.m_state.in_generator_function_context = m_state.in_generator_function_context;
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
parser.m_state.await_expression_is_valid = m_state.await_expression_is_valid;
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
parser.m_state.in_arrow_function_context = m_state.in_arrow_function_context;
parser.m_state.in_break_context = m_state.in_break_context;
parser.m_state.in_continue_context = m_state.in_continue_context;
parser.m_state.string_legacy_octal_escape_sequence_in_scope = m_state.string_legacy_octal_escape_sequence_in_scope;
parser.m_state.in_class_field_initializer = m_state.in_class_field_initializer;
LibJS: Add support for await expressions
2021-11-09 20:52:21 +00:00
parser.m_state.in_class_static_init_block = m_state.in_class_static_init_block;
LibJS: Treat private identifier as divisible token And also make sure private identifiers are correctly checked when synthesizing a binding pattern.
2021-11-26 20:10:24 +00:00
parser.m_state.referenced_private_names = m_state.referenced_private_names;
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
auto result = parser.parse_binding_pattern(AllowDuplicates::Yes, AllowMemberExpressions::Yes);
if (parser.has_errors())
m_state.errors.extend(parser.errors());
return result;
}
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
NonnullRefPtr<AssignmentExpression> Parser::parse_assignment_expression(AssignmentOp assignment_op, NonnullRefPtr<Expression> lhs, int min_precedence, Associativity associativity, ForbiddenTokens forbidden)
LibJS: Validate all assignment expressions, not just "=" The check for invalid lhs and assignment to eval/arguments in strict mode should happen for all kinds of assignment expressions, not just AssignmentOp::Assignment.
2020-10-04 22:58:57 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 19:42:32 +00:00
VERIFY(match(TokenType::Equals)
LibJS: Validate all assignment expressions, not just "=" The check for invalid lhs and assignment to eval/arguments in strict mode should happen for all kinds of assignment expressions, not just AssignmentOp::Assignment.
2020-10-04 22:58:57 +00:00
|| match(TokenType::PlusEquals)
|| match(TokenType::MinusEquals)
|| match(TokenType::AsteriskEquals)
|| match(TokenType::SlashEquals)
|| match(TokenType::PercentEquals)
|| match(TokenType::DoubleAsteriskEquals)
|| match(TokenType::AmpersandEquals)
|| match(TokenType::PipeEquals)
|| match(TokenType::CaretEquals)
|| match(TokenType::ShiftLeftEquals)
|| match(TokenType::ShiftRightEquals)
LibJS: Implement logical assignment operators (&&=, ||=, ??=) TC39 proposal, stage 4 as of 2020-07. https://tc39.es/proposal-logical-assignment/
2020-10-05 15:49:43 +00:00
|| match(TokenType::UnsignedShiftRightEquals)
|| match(TokenType::DoubleAmpersandEquals)
|| match(TokenType::DoublePipeEquals)
|| match(TokenType::DoubleQuestionMarkEquals));
LibJS: Validate all assignment expressions, not just "=" The check for invalid lhs and assignment to eval/arguments in strict mode should happen for all kinds of assignment expressions, not just AssignmentOp::Assignment.
2020-10-04 22:58:57 +00:00
consume();
LibJS: Implement parsing and evaluation for AssignmentPatterns e.g. `[...foo] = bar` can now be evaluated :^)
2021-07-10 20:46:17 +00:00
if (assignment_op == AssignmentOp::Assignment) {
if (is<ArrayExpression>(*lhs) || is<ObjectExpression>(*lhs)) {
auto binding_pattern = synthesize_binding_pattern(*lhs);
if (binding_pattern) {
auto rhs = parse_expression(min_precedence, associativity);
return create_ast_node<AssignmentExpression>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Implement parsing and evaluation for AssignmentPatterns e.g. `[...foo] = bar` can now be evaluated :^)
2021-07-10 20:46:17 +00:00
assignment_op,
binding_pattern.release_nonnull(),
move(rhs));
}
}
}
LibJS: Allow CallExpressions as lhs of assignments in most cases Although not quite like the spec says the web reality is that a lhs target of CallExpression should not give a SyntaxError but only a ReferenceError once executed.
2022-11-30 00:47:25 +00:00
// Note: The web reality is that all but &&=, ||= and ??= do allow left hand side CallExpresions.
// These are the exception as they are newer.
auto has_web_reality_assignment_target_exceptions = assignment_op != AssignmentOp::AndAssignment
&& assignment_op != AssignmentOp::OrAssignment
&& assignment_op != AssignmentOp::NullishAssignment;
if (!is_simple_assignment_target(*lhs, has_web_reality_assignment_target_exceptions)) {
LibJS: Validate all assignment expressions, not just "=" The check for invalid lhs and assignment to eval/arguments in strict mode should happen for all kinds of assignment expressions, not just AssignmentOp::Assignment.
2020-10-04 22:58:57 +00:00
syntax_error("Invalid left-hand side in assignment");
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
} else if (m_state.strict_mode && is<Identifier>(*lhs)) {
LibJS: Make more use of Token::flystring_value() This patch makes check_identifier_name_for_assignment_validity() take a FlyString instead of a StringView. We then exploit this by passing FlyString in more places via flystring_value(). This gives a ~1% speedup when parsing the largest Discord JS file.
2022-02-13 12:34:26 +00:00
auto const& name = static_cast<Identifier const&>(*lhs).string();
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
check_identifier_name_for_assignment_validity(name);
LibJS: Validate all assignment expressions, not just "=" The check for invalid lhs and assignment to eval/arguments in strict mode should happen for all kinds of assignment expressions, not just AssignmentOp::Assignment.
2020-10-04 22:58:57 +00:00
}
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
auto rhs = parse_expression(min_precedence, associativity, forbidden);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<AssignmentExpression>({ m_source_code, rule_start.position(), position() }, assignment_op, move(lhs), move(rhs));
LibJS: Validate all assignment expressions, not just "=" The check for invalid lhs and assignment to eval/arguments in strict mode should happen for all kinds of assignment expressions, not just AssignmentOp::Assignment.
2020-10-04 22:58:57 +00:00
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
NonnullRefPtr<Identifier> Parser::parse_identifier()
{
auto identifier_start = position();
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
auto token = consume_identifier();
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
if (m_state.in_class_field_initializer && token.value() == "arguments"sv)
syntax_error("'arguments' is not allowed in class field initializer");
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
return create_ast_node<Identifier>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, identifier_start, position() },
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
token.DeprecatedFlyString_value());
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
}
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
Vector<CallExpression::Argument> Parser::parse_arguments()
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Add function call spreading Adds support for the following syntax: myFunction(...x, ...[1, 2, 3], ...o.foo, ...'abcd')
2020-05-06 05:36:24 +00:00
Vector<CallExpression::Argument> arguments;
LibJS: Parse CallExpression arguments
2020-03-12 18:35:23 +00:00
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
consume(TokenType::ParenOpen);
LibJS: Add function call spreading Adds support for the following syntax: myFunction(...x, ...[1, 2, 3], ...o.foo, ...'abcd')
2020-05-06 05:36:24 +00:00
while (match_expression() || match(TokenType::TripleDot)) {
if (match(TokenType::TripleDot)) {
consume();
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
arguments.append({ parse_expression(2), true });
LibJS: Add function call spreading Adds support for the following syntax: myFunction(...x, ...[1, 2, 3], ...o.foo, ...'abcd')
2020-05-06 05:36:24 +00:00
} else {
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
arguments.append({ parse_expression(2), false });
LibJS: Add function call spreading Adds support for the following syntax: myFunction(...x, ...[1, 2, 3], ...o.foo, ...'abcd')
2020-05-06 05:36:24 +00:00
}
LibJS: Fix broken parsing of 0-argument CallExpression
2020-03-12 19:03:12 +00:00
if (!match(TokenType::Comma))
break;
consume();
LibJS: Parse CallExpression arguments
2020-03-12 18:35:23 +00:00
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume(TokenType::ParenClose);
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
return arguments;
}
NonnullRefPtr<Expression> Parser::parse_call_expression(NonnullRefPtr<Expression> lhs)
{
auto rule_start = push_start();
if (!m_state.allow_super_constructor_call && is<SuperExpression>(*lhs))
syntax_error("'super' keyword unexpected here");
auto arguments = parse_arguments();
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
LibJS: Make SuperCall a proper AST node and clean up evaluation
2021-07-02 17:30:38 +00:00
if (is<SuperExpression>(*lhs))
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<SuperCall>({ m_source_code, rule_start.position(), position() }, move(arguments));
LibJS: Make SuperCall a proper AST node and clean up evaluation
2021-07-02 17:30:38 +00:00
LibJS: Make one compact allocation for CallExpression and its Arguments Instead of CallExpression storing its arguments in a Vector<Argument>, we now custom-allocate the memory slot for CallExpression (and its subclass NewExpression) so that it fits both CallExpression and its list of Arguments in one allocation. This reduces memory usage on twitter.com/awesomekling by 8.8 MiB :^)
2022-11-26 18:51:50 +00:00
return CallExpression::create({ m_source_code, rule_start.position(), position() }, move(lhs), arguments.span());
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Implement basic support for the "new" keyword NewExpression mostly piggybacks on the existing CallExpression. The big difference is that "new" creates a new Object and passes it as |this| to the callee.
2020-03-28 15:33:52 +00:00
NonnullRefPtr<NewExpression> Parser::parse_new_expression()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Implement basic support for the "new" keyword NewExpression mostly piggybacks on the existing CallExpression. The big difference is that "new" creates a new Object and passes it as |this| to the callee.
2020-03-28 15:33:52 +00:00
consume(TokenType::New);
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
auto callee = parse_expression(g_operator_precedence.get(TokenType::New), Associativity::Right, { TokenType::ParenOpen, TokenType::QuestionMarkPeriod });
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
if (is<ImportCall>(*callee))
syntax_error("Cannot call new on dynamic import", callee->source_range().start);
LibJS: Implement basic support for the "new" keyword NewExpression mostly piggybacks on the existing CallExpression. The big difference is that "new" creates a new Object and passes it as |this| to the callee.
2020-03-28 15:33:52 +00:00
LibJS: Add function call spreading Adds support for the following syntax: myFunction(...x, ...[1, 2, 3], ...o.foo, ...'abcd')
2020-05-06 05:36:24 +00:00
Vector<CallExpression::Argument> arguments;
LibJS: Implement basic support for the "new" keyword NewExpression mostly piggybacks on the existing CallExpression. The big difference is that "new" creates a new Object and passes it as |this| to the callee.
2020-03-28 15:33:52 +00:00
if (match(TokenType::ParenOpen)) {
consume(TokenType::ParenOpen);
LibJS: Add function call spreading Adds support for the following syntax: myFunction(...x, ...[1, 2, 3], ...o.foo, ...'abcd')
2020-05-06 05:36:24 +00:00
while (match_expression() || match(TokenType::TripleDot)) {
if (match(TokenType::TripleDot)) {
consume();
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
arguments.append({ parse_expression(2), true });
LibJS: Add function call spreading Adds support for the following syntax: myFunction(...x, ...[1, 2, 3], ...o.foo, ...'abcd')
2020-05-06 05:36:24 +00:00
} else {
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
arguments.append({ parse_expression(2), false });
LibJS: Add function call spreading Adds support for the following syntax: myFunction(...x, ...[1, 2, 3], ...o.foo, ...'abcd')
2020-05-06 05:36:24 +00:00
}
LibJS: Implement basic support for the "new" keyword NewExpression mostly piggybacks on the existing CallExpression. The big difference is that "new" creates a new Object and passes it as |this| to the callee.
2020-03-28 15:33:52 +00:00
if (!match(TokenType::Comma))
break;
consume();
}
consume(TokenType::ParenClose);
}
LibJS: Make one compact allocation for CallExpression and its Arguments Instead of CallExpression storing its arguments in a Vector<Argument>, we now custom-allocate the memory slot for CallExpression (and its subclass NewExpression) so that it fits both CallExpression and its list of Arguments in one allocation. This reduces memory usage on twitter.com/awesomekling by 8.8 MiB :^)
2022-11-26 18:51:50 +00:00
return NewExpression::create({ m_source_code, rule_start.position(), position() }, move(callee), move(arguments));
LibJS: Implement basic support for the "new" keyword NewExpression mostly piggybacks on the existing CallExpression. The big difference is that "new" creates a new Object and passes it as |this| to the callee.
2020-03-28 15:33:52 +00:00
}
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
NonnullRefPtr<YieldExpression> Parser::parse_yield_expression()
{
auto rule_start = push_start();
LibJS: Disallow yield expression correctly in formal parameters And add ZERO WIDTH NO BREAK SPACE to valid whitespace.
2021-08-21 09:31:36 +00:00
if (m_state.in_formal_parameter_context)
syntax_error("'Yield' expression is not allowed in formal parameters of generator function");
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
consume(TokenType::Yield);
RefPtr<Expression> argument;
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
bool yield_from = false;
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (!m_state.current_token.trivia_contains_line_terminator()) {
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
if (match(TokenType::Asterisk)) {
consume();
yield_from = true;
}
LibJS: Allow yielding a class
2021-07-29 00:03:38 +00:00
if (yield_from || match_expression() || match(TokenType::Class))
LibJS: Stop parsing an expression on comma after a yield
2021-11-26 22:38:36 +00:00
argument = parse_expression(2);
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 11:16:41 +00:00
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<YieldExpression>({ m_source_code, rule_start.position(), position() }, move(argument), yield_from);
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
}
LibJS: Add support for await expressions
2021-11-09 20:52:21 +00:00
NonnullRefPtr<AwaitExpression> Parser::parse_await_expression()
{
auto rule_start = push_start();
if (m_state.in_formal_parameter_context)
syntax_error("'Await' expression is not allowed in formal parameters of an async function");
consume(TokenType::Await);
auto precedence = g_operator_precedence.get(TokenType::Await);
auto associativity = operator_associativity(TokenType::Await);
auto argument = parse_expression(precedence, associativity);
LibJS: Track whether a program has a top level await statement
2022-01-18 17:55:19 +00:00
m_state.current_scope_pusher->set_contains_await_expression();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<AwaitExpression>({ m_source_code, rule_start.position(), position() }, move(argument));
LibJS: Add support for await expressions
2021-11-09 20:52:21 +00:00
}
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 10:23:53 +00:00
NonnullRefPtr<ReturnStatement> Parser::parse_return_statement()
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (!m_state.in_function_context && !m_state.in_arrow_function_context)
LibJS: Disallow 'return' outside of a function
2020-10-07 18:33:48 +00:00
syntax_error("'return' not allowed outside of a function");
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume(TokenType::Return);
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
// Automatic semicolon insertion: terminate statement when return is followed by newline
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (m_state.current_token.trivia_contains_line_terminator())
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ReturnStatement>({ m_source_code, rule_start.position(), position() }, nullptr);
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
if (match_expression()) {
LibJS: Fix parsing of IfStatement, fixes #1829
2020-04-18 12:00:43 +00:00
auto expression = parse_expression(0);
consume_or_insert_semicolon();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ReturnStatement>({ m_source_code, rule_start.position(), position() }, move(expression));
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Fix parsing of IfStatement, fixes #1829
2020-04-18 12:00:43 +00:00
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
consume_or_insert_semicolon();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ReturnStatement>({ m_source_code, rule_start.position(), position() }, nullptr);
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
void Parser::parse_statement_list(ScopeNode& output_node, AllowLabelledFunction allow_labelled_functions)
LibJS: Strict mode is now handled by Functions and Programs, not Blocks Since blocks can't be strict by themselves, it makes no sense for them to store whether or not they are strict. Strict-ness is now stored in the Program and FunctionNode ASTNodes. Fixes issue #3641
2020-10-04 00:02:43 +00:00
{
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
while (!done()) {
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
if (match_declaration(AllowUsingDeclaration::Yes)) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
auto declaration = parse_declaration();
VERIFY(m_state.current_scope_pusher);
m_state.current_scope_pusher->add_declaration(declaration);
output_node.append(move(declaration));
} else if (match_statement()) {
output_node.append(parse_statement(allow_labelled_functions));
} else {
break;
}
}
LibJS: Call shrink_to_fit() on various Vectors created during parse Vectors that stick around in the AST were wasting a fair bit of memory due to the growth padding we keep by default. This patch goes after some of these vectors with the shrink_to_fit() stick to reduce waste. Since the AST can stay around for a long time, it is worth making an effort to shrink it down when we have a chance.
2022-11-26 19:45:06 +00:00
output_node.shrink_to_fit();
LibJS: Strict mode is now handled by Functions and Programs, not Blocks Since blocks can't be strict by themselves, it makes no sense for them to store whether or not they are strict. Strict-ness is now stored in the Program and FunctionNode ASTNodes. Fixes issue #3641
2020-10-04 00:02:43 +00:00
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
// FunctionBody, https://tc39.es/ecma262/#prod-FunctionBody
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
NonnullRefPtr<FunctionBody> Parser::parse_function_body(Vector<FunctionParameter> const& parameters, FunctionKind function_kind, bool& contains_direct_call_to_eval)
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto function_body = create_ast_node<FunctionBody>({ m_source_code, rule_start.position(), position() });
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopePusher function_scope = ScopePusher::function_scope(*this, function_body, parameters); // FIXME <-
auto has_use_strict = parse_directive(function_body);
bool previous_strict_mode = m_state.strict_mode;
if (has_use_strict) {
m_state.strict_mode = true;
function_body->set_strict_mode();
if (!is_simple_parameter_list(parameters))
syntax_error("Illegal 'use strict' directive in function with non-simple parameter list");
} else if (previous_strict_mode) {
function_body->set_strict_mode();
}
LibJS: Add strict mode Adds the ability for a scope (either a function or the entire program) to be in strict mode. Scopes default to non-strict mode. There are two ways to determine the strict-ness of the JS engine: 1. In the parser, this can be accessed with the parser_state variable m_is_strict_mode boolean. If true, the Parser is currently parsing in strict mode. This is done so that the Parser can generate syntax errors at parse time, which is required in some cases. 2. With Interpreter.is_strict_mode(). This allows strict mode checking at runtime as opposed to compile time. Additionally, in order to test this, a global isStrictMode() function has been added to the JS ReplObject under the test-mode flag.
2020-05-28 05:22:08 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
parse_statement_list(function_body);
LibJS: Add strict mode Adds the ability for a scope (either a function or the entire program) to be in strict mode. Scopes default to non-strict mode. There are two ways to determine the strict-ness of the JS engine: 1. In the parser, this can be accessed with the parser_state variable m_is_strict_mode boolean. If true, the Parser is currently parsing in strict mode. This is done so that the Parser can generate syntax errors at parse time, which is required in some cases. 2. With Interpreter.is_strict_mode(). This allows strict mode checking at runtime as opposed to compile time. Additionally, in order to test this, a global isStrictMode() function has been added to the JS ReplObject under the test-mode flag.
2020-05-28 05:22:08 +00:00
LibJS: Consume curly braces outside of Parser::parse_function_body() The curly braces are not part of the FunctionBody production. This becomes relevant when parsing a standalone function body as per the spec in the CreateDynamicFunction AO.
2022-01-15 16:07:51 +00:00
// If we're parsing the function body standalone, e.g. via CreateDynamicFunction, we must have reached EOF here.
// Otherwise, we need a closing curly bracket (which is consumed elsewhere). If we get neither, it's an error.
if (!match(TokenType::Eof) && !match(TokenType::CurlyClose))
expected(Token::name(TokenType::CurlyClose));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
// If the function contains 'use strict' we need to check the parameters (again).
LibJS: Rename FunctionKind::{Regular => Normal} This is what CreateDynamicFunction calls it.
2022-01-14 23:30:02 +00:00
if (function_body->in_strict_mode() || function_kind != FunctionKind::Normal) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
Vector<StringView> parameter_names;
for (auto& parameter : parameters) {
parameter.binding.visit(
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
[&](DeprecatedFlyString const& parameter_name) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
check_identifier_name_for_assignment_validity(parameter_name, function_body->in_strict_mode());
if (function_kind == FunctionKind::Generator && parameter_name == "yield"sv)
syntax_error("Parameter name 'yield' not allowed in this context");
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (function_kind == FunctionKind::Async && parameter_name == "await"sv)
syntax_error("Parameter name 'await' not allowed in this context");
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
for (auto& previous_name : parameter_names) {
if (previous_name == parameter_name) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Duplicate parameter '{}' not allowed in strict mode", parameter_name));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
}
LibJS: Be less strict about 'use strict' It is allowed to have (bogus) directives e.g. 'does nothing'; 'same'; 'use strict'; Should still trigger strict mode.
2021-07-11 23:25:32 +00:00
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
parameter_names.append(parameter_name);
},
[&](NonnullRefPtr<BindingPattern> const& binding) {
binding->for_each_bound_name([&](auto& bound_name) {
if (function_kind == FunctionKind::Generator && bound_name == "yield"sv)
syntax_error("Parameter name 'yield' not allowed in this context");
LibJS: Be less strict about 'use strict' It is allowed to have (bogus) directives e.g. 'does nothing'; 'same'; 'use strict'; Should still trigger strict mode.
2021-07-11 23:25:32 +00:00
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (function_kind == FunctionKind::Async && bound_name == "await"sv)
syntax_error("Parameter name 'await' not allowed in this context");
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
for (auto& previous_name : parameter_names) {
if (previous_name == bound_name) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Duplicate parameter '{}' not allowed in strict mode", bound_name));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
break;
}
}
parameter_names.append(bound_name);
});
});
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
m_state.strict_mode = previous_strict_mode;
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 10:43:38 +00:00
contains_direct_call_to_eval = function_scope.contains_direct_call_to_eval();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
return function_body;
}
NonnullRefPtr<BlockStatement> Parser::parse_block_statement()
{
auto rule_start = push_start();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto block = create_ast_node<BlockStatement>({ m_source_code, rule_start.position(), position() });
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopePusher block_scope = ScopePusher::block_scope(*this, block);
consume(TokenType::CurlyOpen);
parse_statement_list(block);
consume(TokenType::CurlyClose);
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
return block;
}
LibJS: Parse FunctionExpressions FunctionExpression is mostly like FunctionDeclaration, except the name is optional. Share the parsing logic in parse_function_node<NodeType>. This allows us to do nice things like: document.addEventListener("DOMContentLoaded", function() { alert("Hello friends!"); });
2020-03-19 10:52:56 +00:00
template<typename FunctionNodeType>
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
NonnullRefPtr<FunctionNodeType> Parser::parse_function_node(u16 parse_options, Optional<Position> const& function_start)
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto rule_start = function_start.has_value()
? RulePosition { *this, *function_start }
: push_start();
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 19:42:32 +00:00
VERIFY(!(parse_options & FunctionNodeParseOptions::IsGetterFunction && parse_options & FunctionNodeParseOptions::IsSetterFunction));
LibJS: Move checks for invalid getter/setter params to parse_function_node This allows us to provide better error messages as we can point the syntax error location to the exact first invalid parameter instead of always the end of the function within a object literal or class definition. Before this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Object setter property must have one argument (line: 1, column: 28) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Class setter method must have one argument (line: 1, column: 26) After this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 23) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 21) The only possible downside of this change is that class getters/setters and functions in objects are not distinguished in the message anymore - I don't think that's important though, and classes are (mostly) just syntactic sugar anyway.
2020-10-20 17:32:51 +00:00
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
TemporaryChange super_property_access_rollback(m_state.allow_super_property_lookup, !!(parse_options & FunctionNodeParseOptions::AllowSuperPropertyLookup));
TemporaryChange super_constructor_call_rollback(m_state.allow_super_constructor_call, !!(parse_options & FunctionNodeParseOptions::AllowSuperConstructorCall));
LibJS: Make functions reset break and continue context
2021-07-25 09:51:33 +00:00
TemporaryChange break_context_rollback(m_state.in_break_context, false);
TemporaryChange continue_context_rollback(m_state.in_continue_context, false);
LibJS: Add support for public fields in classes
2021-08-28 15:11:05 +00:00
TemporaryChange class_field_initializer_rollback(m_state.in_class_field_initializer, false);
LibJS: Add an optimization to avoid needless arguments object creation This gives FunctionNode a "might need arguments object" boolean flag and sets it based on the simplest possible heuristic for this: if we encounter an identifier called "arguments" or "eval" up to the next (nested) function declaration or expression, we won't need an arguments object. Otherwise, we *might* need one - the final decision is made in the FunctionDeclarationInstantiation AO. Now, this is obviously not perfect. Even if you avoid eval, something like `foo.arguments` will still trigger a false positive - but it's a start and already massively cuts down on needlessly allocated objects, especially in real-world code that is often minified, and so a full "arguments" identifier will be an actual arguments object more often than not. To illustrate the actual impact of this change, here's the number of allocated arguments objects during a full test-js run: Before: - Unmapped arguments objects: 78765 - Mapped arguments objects: 2455 After: - Unmapped arguments objects: 18 - Mapped arguments objects: 37 This results in a ~5% speedup of test-js on my Linux host machine, and about 3.5% on i686 Serenity in QEMU (warm runs, average of 5). The following microbenchmark (calling an empty function 1M times) runs 25% faster on Linux and 45% on Serenity: function foo() {} for (var i = 0; i < 1_000_000; ++i) foo(); test262 reports no changes in either direction, apart from a speedup :^)
2021-10-05 07:44:58 +00:00
TemporaryChange might_need_arguments_object_rollback(m_state.function_might_need_arguments_object, false);
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
LibJS: Allow 'yield' and 'await' as function expression names The spec says so, and test262 checks for this too.
2021-07-02 10:41:11 +00:00
constexpr auto is_function_expression = IsSame<FunctionNodeType, FunctionExpression>;
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
FunctionKind function_kind;
if ((parse_options & FunctionNodeParseOptions::IsGeneratorFunction) != 0 && (parse_options & FunctionNodeParseOptions::IsAsyncFunction) != 0)
LibJS: Parse async generator functions
2021-11-15 00:53:24 +00:00
function_kind = FunctionKind::AsyncGenerator;
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
else if ((parse_options & FunctionNodeParseOptions::IsGeneratorFunction) != 0)
function_kind = FunctionKind::Generator;
else if ((parse_options & FunctionNodeParseOptions::IsAsyncFunction) != 0)
function_kind = FunctionKind::Async;
else
LibJS: Rename FunctionKind::{Regular => Normal} This is what CreateDynamicFunction calls it.
2022-01-14 23:30:02 +00:00
function_kind = FunctionKind::Normal;
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString name;
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 16:56:49 +00:00
if (parse_options & FunctionNodeParseOptions::CheckForFunctionAndName) {
LibJS: Rename FunctionKind::{Regular => Normal} This is what CreateDynamicFunction calls it.
2022-01-14 23:30:02 +00:00
if (function_kind == FunctionKind::Normal && match(TokenType::Async) && !next_token().trivia_contains_line_terminator()) {
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
function_kind = FunctionKind::Async;
consume(TokenType::Async);
LibJS: Implement create_dynamic_function() according to the spec The three major changes are: - Parsing parameters, the function body, and then the full assembled function source all separately. This is required by the spec, as function parameters and body must be valid each on their own, which cannot be guaranteed if we only ever parse the full function. - Returning an ECMAScriptFunctionObject instead of a FunctionExpression that needs to be evaluated separately. This vastly simplifies the {Async,AsyncGenerator,Generator,}Function constructor implementations. Drop '_node' from the function name accordingly. - The prototype is now determined via GetPrototypeFromConstructor and passed to OrdinaryFunctionCreate.
2022-01-15 16:26:06 +00:00
parse_options |= FunctionNodeParseOptions::IsAsyncFunction;
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
}
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 16:56:49 +00:00
consume(TokenType::Function);
LibJS: Parse async generator functions
2021-11-15 00:53:24 +00:00
if (match(TokenType::Asterisk)) {
LibJS: Rename FunctionKind::{Regular => Normal} This is what CreateDynamicFunction calls it.
2022-01-14 23:30:02 +00:00
function_kind = function_kind == FunctionKind::Normal ? FunctionKind::Generator : FunctionKind::AsyncGenerator;
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
consume(TokenType::Asterisk);
LibJS: Implement create_dynamic_function() according to the spec The three major changes are: - Parsing parameters, the function body, and then the full assembled function source all separately. This is required by the spec, as function parameters and body must be valid each on their own, which cannot be guaranteed if we only ever parse the full function. - Returning an ECMAScriptFunctionObject instead of a FunctionExpression that needs to be evaluated separately. This vastly simplifies the {Async,AsyncGenerator,Generator,}Function constructor implementations. Drop '_node' from the function name accordingly. - The prototype is now determined via GetPrototypeFromConstructor and passed to OrdinaryFunctionCreate.
2022-01-15 16:26:06 +00:00
parse_options |= FunctionNodeParseOptions::IsGeneratorFunction;
LibJS: Parse generator functions in object literals Also add some parser tests
2021-06-14 10:22:59 +00:00
}
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
if (parse_options & FunctionNodeParseOptions::HasDefaultExportName)
name = ExportStatement::local_name_for_default;
else if (FunctionNodeType::must_have_name() || match_identifier())
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
name = consume_identifier().DeprecatedFlyString_value();
LibJS: Allow 'yield' and 'await' as function expression names The spec says so, and test262 checks for this too.
2021-07-02 10:41:11 +00:00
else if (is_function_expression && (match(TokenType::Yield) || match(TokenType::Await)))
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
name = consume().DeprecatedFlyString_value();
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
check_identifier_name_for_assignment_validity(name);
LibJS: Disallow await keywords in static init blocks In static init blocks 'await' cannot be used. Note that this does not cover all the cases since the parser currently cannot distinguish between expressions within parenthesis and direct expressions.
2021-11-26 22:29:05 +00:00
LibJS: Disallow async generator functions called 'await' or 'yield'
2021-12-19 19:19:15 +00:00
if (function_kind == FunctionKind::AsyncGenerator && (name == "await"sv || name == "yield"sv))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("async generator function is not allowed to be called '{}'", name));
LibJS: Disallow async generator functions called 'await' or 'yield'
2021-12-19 19:19:15 +00:00
LibJS: Disallow await keywords in static init blocks In static init blocks 'await' cannot be used. Note that this does not cover all the cases since the parser currently cannot distinguish between expressions within parenthesis and direct expressions.
2021-11-26 22:29:05 +00:00
if (m_state.in_class_static_init_block && name == "await"sv)
syntax_error("'await' is a reserved word");
LibJS: Parse FunctionExpressions FunctionExpression is mostly like FunctionDeclaration, except the name is optional. Share the parsing logic in parse_function_node<NodeType>. This allows us to do nice things like: document.addEventListener("DOMContentLoaded", function() { alert("Hello friends!"); });
2020-03-19 10:52:56 +00:00
}
LibJS: Disallow await keywords in static init blocks In static init blocks 'await' cannot be used. Note that this does not cover all the cases since the parser currently cannot distinguish between expressions within parenthesis and direct expressions.
2021-11-26 22:29:05 +00:00
TemporaryChange class_static_initializer_rollback(m_state.in_class_static_init_block, false);
LibJS: Parse async generator functions
2021-11-15 00:53:24 +00:00
TemporaryChange generator_change(m_state.in_generator_function_context, function_kind == FunctionKind::Generator || function_kind == FunctionKind::AsyncGenerator);
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
TemporaryChange async_change(m_state.await_expression_is_valid, function_kind == FunctionKind::Async || function_kind == FunctionKind::AsyncGenerator);
LibJS: Treat yield as an identifier in more non-generator contexts And disallow some cases where we are in a generator context.
2021-07-28 23:45:55 +00:00
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
consume(TokenType::ParenOpen);
LibJS: Function.length respects default and rest parameters "[Function.length is] the number of formal parameters. This number excludes the rest parameter and only includes parameters before the first one with a default value." - MDN
2020-05-06 03:02:14 +00:00
i32 function_length = -1;
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
auto parameters = parse_formal_parameters(function_length, parse_options);
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
consume(TokenType::ParenClose);
if (function_length == -1)
function_length = parameters.size();
LibJS: Add an optimization to avoid needless arguments object creation This gives FunctionNode a "might need arguments object" boolean flag and sets it based on the simplest possible heuristic for this: if we encounter an identifier called "arguments" or "eval" up to the next (nested) function declaration or expression, we won't need an arguments object. Otherwise, we *might* need one - the final decision is made in the FunctionDeclarationInstantiation AO. Now, this is obviously not perfect. Even if you avoid eval, something like `foo.arguments` will still trigger a false positive - but it's a start and already massively cuts down on needlessly allocated objects, especially in real-world code that is often minified, and so a full "arguments" identifier will be an actual arguments object more often than not. To illustrate the actual impact of this change, here's the number of allocated arguments objects during a full test-js run: Before: - Unmapped arguments objects: 78765 - Mapped arguments objects: 2455 After: - Unmapped arguments objects: 18 - Mapped arguments objects: 37 This results in a ~5% speedup of test-js on my Linux host machine, and about 3.5% on i686 Serenity in QEMU (warm runs, average of 5). The following microbenchmark (calling an empty function 1M times) runs 25% faster on Linux and 45% on Serenity: function foo() {} for (var i = 0; i < 1_000_000; ++i) foo(); test262 reports no changes in either direction, apart from a speedup :^)
2021-10-05 07:44:58 +00:00
TemporaryChange function_context_rollback(m_state.in_function_context, true);
LibJS: Treat yield as an identifier in more non-generator contexts And disallow some cases where we are in a generator context.
2021-07-28 23:45:55 +00:00
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto old_labels_in_scope = move(m_state.labels_in_scope);
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
ScopeGuard guard([&]() {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
m_state.labels_in_scope = move(old_labels_in_scope);
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
});
LibJS: Consume curly braces outside of Parser::parse_function_body() The curly braces are not part of the FunctionBody production. This becomes relevant when parsing a standalone function body as per the spec in the CreateDynamicFunction AO.
2022-01-15 16:07:51 +00:00
consume(TokenType::CurlyOpen);
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 10:43:38 +00:00
bool contains_direct_call_to_eval = false;
auto body = parse_function_body(parameters, function_kind, contains_direct_call_to_eval);
LibJS: Consume curly braces outside of Parser::parse_function_body() The curly braces are not part of the FunctionBody production. This becomes relevant when parsing a standalone function body as per the spec in the CreateDynamicFunction AO.
2022-01-15 16:07:51 +00:00
consume(TokenType::CurlyClose);
LibJS: Treat yield as an identifier in more non-generator contexts And disallow some cases where we are in a generator context.
2021-07-28 23:45:55 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
auto has_strict_directive = body->in_strict_mode();
LibJS: Handle strict mode for functions more correctly If a function is strict (has 'use strict' directive) it cannot have bindings, cannot have duplicated parameter names and cannot have some reserved keywords and identifiers as parameter names. The logic partly applies depending on whether we are already in strict mode or the function contains 'use strict';
2021-07-11 23:29:07 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (has_strict_directive)
check_identifier_name_for_assignment_validity(name, true);
LibJS: Improve function hoisting across blocks The parser now keeps track of a scope chain so that it can hoist function declarations to the closest function scope.
2021-07-04 01:15:52 +00:00
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
auto function_start_offset = rule_start.position().offset;
auto function_end_offset = position().offset - m_state.current_token.trivia().length();
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
auto source_text = DeprecatedString { m_state.lexer.source().substring_view(function_start_offset, function_end_offset - function_start_offset) };
LibJS: Use an enum class instead of 'bool is_generator' This avoid confusion in the order of the multiple boolean parameters that exist.
2021-06-10 23:08:05 +00:00
return create_ast_node<FunctionNodeType>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Capture source text of FunctionNode and ClassExpression
2022-01-18 23:46:16 +00:00
name, move(source_text), move(body), move(parameters), function_length,
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 10:43:38 +00:00
function_kind, has_strict_directive, m_state.function_might_need_arguments_object,
contains_direct_call_to_eval);
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
}
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
Vector<FunctionParameter> Parser::parse_formal_parameters(int& function_length, u16 parse_options)
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
bool has_default_parameter = false;
bool has_rest_parameter = false;
LibJS: Disallow yield expression correctly in formal parameters And add ZERO WIDTH NO BREAK SPACE to valid whitespace.
2021-08-21 09:31:36 +00:00
TemporaryChange formal_parameter_context_change { m_state.in_formal_parameter_context, true };
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
LibJS: Make FunctionNode::Parameter be a standalone FunctionParameter This will allow us to forward declare it and avoid including AST.h in a number of places.
2022-11-23 12:12:36 +00:00
Vector<FunctionParameter> parameters;
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
auto consume_identifier_or_binding_pattern = [&]() -> Variant<DeprecatedFlyString, NonnullRefPtr<BindingPattern>> {
LibJS: Disallow member expression in binding pattern as parameters
2021-11-26 22:37:14 +00:00
if (auto pattern = parse_binding_pattern(AllowDuplicates::No, AllowMemberExpressions::No))
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
return pattern.release_nonnull();
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
auto token = consume_identifier();
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
auto parameter_name = token.DeprecatedFlyString_value();
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
check_identifier_name_for_assignment_validity(parameter_name);
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
for (auto& parameter : parameters) {
LibJS: Handle strict mode for functions more correctly If a function is strict (has 'use strict' directive) it cannot have bindings, cannot have duplicated parameter names and cannot have some reserved keywords and identifiers as parameter names. The logic partly applies depending on whether we are already in strict mode or the function contains 'use strict';
2021-07-11 23:29:07 +00:00
bool has_same_name = parameter.binding.visit(
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
[&](DeprecatedFlyString const& name) {
LibJS: Handle strict mode for functions more correctly If a function is strict (has 'use strict' directive) it cannot have bindings, cannot have duplicated parameter names and cannot have some reserved keywords and identifiers as parameter names. The logic partly applies depending on whether we are already in strict mode or the function contains 'use strict';
2021-07-11 23:29:07 +00:00
return name == parameter_name;
},
[&](NonnullRefPtr<BindingPattern> const& bindings) {
bool found_duplicate = false;
bindings->for_each_bound_name([&](auto& bound_name) {
if (bound_name == parameter_name)
found_duplicate = true;
});
return found_duplicate;
});
if (!has_same_name)
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
continue;
LibJS: Handle strict mode for functions more correctly If a function is strict (has 'use strict' directive) it cannot have bindings, cannot have duplicated parameter names and cannot have some reserved keywords and identifiers as parameter names. The logic partly applies depending on whether we are already in strict mode or the function contains 'use strict';
2021-07-11 23:29:07 +00:00
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
DeprecatedString message;
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
if (parse_options & FunctionNodeParseOptions::IsArrowFunction)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
message = DeprecatedString::formatted("Duplicate parameter '{}' not allowed in arrow function", parameter_name);
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
else if (m_state.strict_mode)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
message = DeprecatedString::formatted("Duplicate parameter '{}' not allowed in strict mode", parameter_name);
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
else if (has_default_parameter || match(TokenType::Equals))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
message = DeprecatedString::formatted("Duplicate parameter '{}' not allowed in function with default parameter", parameter_name);
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
else if (has_rest_parameter)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
message = DeprecatedString::formatted("Duplicate parameter '{}' not allowed in function with rest parameter", parameter_name);
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
if (!message.is_empty())
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
syntax_error(message, Position { token.line_number(), token.line_column() });
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
break;
}
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
return DeprecatedFlyString { token.value() };
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
};
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
while (match(TokenType::CurlyOpen) || match(TokenType::BracketOpen) || match_identifier() || match(TokenType::TripleDot)) {
LibJS: Move checks for invalid getter/setter params to parse_function_node This allows us to provide better error messages as we can point the syntax error location to the exact first invalid parameter instead of always the end of the function within a object literal or class definition. Before this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Object setter property must have one argument (line: 1, column: 28) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Class setter method must have one argument (line: 1, column: 26) After this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 23) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 21) The only possible downside of this change is that class getters/setters and functions in objects are not distinguished in the message anymore - I don't think that's important though, and classes are (mostly) just syntactic sugar anyway.
2020-10-20 17:32:51 +00:00
if (parse_options & FunctionNodeParseOptions::IsGetterFunction)
syntax_error("Getter function must have no arguments");
LibJS: Rest parameter in setter functions is a syntax error
2020-10-20 17:43:58 +00:00
if (parse_options & FunctionNodeParseOptions::IsSetterFunction && (parameters.size() >= 1 || match(TokenType::TripleDot)))
LibJS: Move checks for invalid getter/setter params to parse_function_node This allows us to provide better error messages as we can point the syntax error location to the exact first invalid parameter instead of always the end of the function within a object literal or class definition. Before this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Object setter property must have one argument (line: 1, column: 28) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Class setter method must have one argument (line: 1, column: 26) After this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 23) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 21) The only possible downside of this change is that class getters/setters and functions in objects are not distinguished in the message anymore - I don't think that's important though, and classes are (mostly) just syntactic sugar anyway.
2020-10-20 17:32:51 +00:00
syntax_error("Setter function must have one argument");
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
auto is_rest = false;
LibJS: Implement rest parameters
2020-05-04 15:05:13 +00:00
if (match(TokenType::TripleDot)) {
consume();
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
has_rest_parameter = true;
LibJS: Function.length respects default and rest parameters "[Function.length is] the number of formal parameters. This number excludes the rest parameter and only includes parameters before the first one with a default value." - MDN
2020-05-06 03:02:14 +00:00
function_length = parameters.size();
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
is_rest = true;
LibJS: Implement rest parameters
2020-05-04 15:05:13 +00:00
}
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
auto parameter = consume_identifier_or_binding_pattern();
LibJS: Add function default arguments Adds the ability for function arguments to have default values. This works for standard functions as well as arrow functions. Default values are not printed in a <function>.toString() call, as nodes cannot print their source string representation.
2020-05-02 18:46:39 +00:00
RefPtr<Expression> default_value;
if (match(TokenType::Equals)) {
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
consume();
LibJS: Tighten default values in formal parameter parsing Disallow default parameter for rest parameters. Disallow yield expressions as default values.
2021-07-28 23:47:36 +00:00
if (is_rest)
syntax_error("Rest parameter may not have a default initializer");
LibJS: Treat default parameter values as being in function context e.g. `new.target` should be permitted in `function foo(x = new.target) {}`.
2021-07-10 18:18:25 +00:00
TemporaryChange change(m_state.in_function_context, true);
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
has_default_parameter = true;
LibJS: Function.length respects default and rest parameters "[Function.length is] the number of formal parameters. This number excludes the rest parameter and only includes parameters before the first one with a default value." - MDN
2020-05-06 03:02:14 +00:00
function_length = parameters.size();
LibJS: Parse comma operator into SequenceExpression
2020-05-11 16:27:31 +00:00
default_value = parse_expression(2);
LibJS: Disallow 'yield' identifier initializer in GeneratorFunctions
2021-06-17 17:14:45 +00:00
bool is_generator = parse_options & FunctionNodeParseOptions::IsGeneratorFunction;
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
if ((is_generator || m_state.strict_mode) && default_value && default_value->fast_is<Identifier>() && static_cast<Identifier&>(*default_value).string() == "yield"sv)
LibJS: Disallow 'yield' identifier initializer in GeneratorFunctions
2021-06-17 17:14:45 +00:00
syntax_error("Generator function parameter initializer cannot contain a reference to an identifier named \"yield\"");
LibJS: Add function default arguments Adds the ability for function arguments to have default values. This works for standard functions as well as arrow functions. Default values are not printed in a <function>.toString() call, as nodes cannot print their source string representation.
2020-05-02 18:46:39 +00:00
}
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
parameters.append({ move(parameter), default_value, is_rest });
LibJS: Don't require ParenClose in Parser::parse_formal_parameters() The parentheses are dealt with outside this function, so we shouldn't use the (non)existence of one as the condition for consuming another comma and then parameter. Just check for a comma instead. This becomes relevant when parsing standalone function parameters as per the spec in the CreateDynamicFunction AO.
2022-01-15 16:10:20 +00:00
if (!match(TokenType::Comma) || is_rest)
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
break;
consume(TokenType::Comma);
}
LibJS: Move checks for invalid getter/setter params to parse_function_node This allows us to provide better error messages as we can point the syntax error location to the exact first invalid parameter instead of always the end of the function within a object literal or class definition. Before this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Object setter property must have one argument (line: 1, column: 28) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Class setter method must have one argument (line: 1, column: 26) After this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 23) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 21) The only possible downside of this change is that class getters/setters and functions in objects are not distinguished in the message anymore - I don't think that's important though, and classes are (mostly) just syntactic sugar anyway.
2020-10-20 17:32:51 +00:00
if (parse_options & FunctionNodeParseOptions::IsSetterFunction && parameters.is_empty())
syntax_error("Setter function must have one argument");
LibJS: Don't require ParenClose in Parser::parse_formal_parameters() The parentheses are dealt with outside this function, so we shouldn't use the (non)existence of one as the condition for consuming another comma and then parameter. Just check for a comma instead. This becomes relevant when parsing standalone function parameters as per the spec in the CreateDynamicFunction AO.
2022-01-15 16:10:20 +00:00
// If we're parsing the parameters standalone, e.g. via CreateDynamicFunction, we must have reached EOF here.
// Otherwise, we need a closing parenthesis (which is consumed elsewhere). If we get neither, it's an error.
if (!match(TokenType::Eof) && !match(TokenType::ParenClose))
expected(Token::name(TokenType::ParenClose));
LibJS: Call shrink_to_fit() on various Vectors created during parse Vectors that stick around in the AST were wasting a fair bit of memory due to the growth padding we keep by default. This patch goes after some of these vectors with the shrink_to_fit() stick to reduce waste. Since the AST can stay around for a long time, it is worth making an effort to shrink it down when we have a chance.
2022-11-26 19:45:06 +00:00
parameters.shrink_to_fit();
LibJS: Share parameter parsing between regular and arrow functions This simplifies try_parse_arrow_function_expression() and fixes a few cases that should not produce an arrow function AST but did: (a,,) => {} (a b) => {} (a ...b) => {} (...b a) => {} The new parsing logic checks whether parens are expected and uses parse_function_parameters() if so, rolling back if a new syntax error occurs during that. Otherwise it's just an identifier in which case we parse the single parameter ourselves.
2020-10-18 23:26:41 +00:00
return parameters;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
static AK::Array<DeprecatedFlyString, 36> s_reserved_words = { "break", "case", "catch", "class", "const", "continue", "debugger", "default", "delete", "do", "else", "enum", "export", "extends", "false", "finally", "for", "function", "if", "import", "in", "instanceof", "new", "null", "return", "super", "switch", "this", "throw", "true", "try", "typeof", "var", "void", "while", "with" };
LibJS: Add optional extra strict checks in parse_binding_pattern
2021-07-28 23:49:25 +00:00
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
RefPtr<BindingPattern> Parser::parse_binding_pattern(Parser::AllowDuplicates allow_duplicates, Parser::AllowMemberExpressions allow_member_expressions)
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
{
auto rule_start = push_start();
TokenType closing_token;
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
bool is_object = true;
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
if (match(TokenType::BracketOpen)) {
consume();
closing_token = TokenType::BracketClose;
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
is_object = false;
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
} else if (match(TokenType::CurlyOpen)) {
consume();
closing_token = TokenType::CurlyClose;
} else {
return {};
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
Vector<BindingPattern::BindingEntry> entries;
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
while (!match(closing_token)) {
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
if (!is_object && match(TokenType::Comma)) {
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
consume();
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
entries.append(BindingPattern::BindingEntry {});
continue;
}
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
auto is_rest = false;
if (match(TokenType::TripleDot)) {
consume();
is_rest = true;
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
decltype(BindingPattern::BindingEntry::name) name = Empty {};
decltype(BindingPattern::BindingEntry::alias) alias = Empty {};
RefPtr<Expression> initializer = {};
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
if (is_object) {
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
bool needs_alias = false;
if (allow_member_expressions == AllowMemberExpressions::Yes && is_rest) {
auto expression_position = position();
auto expression = parse_expression(2, Associativity::Right, { TokenType::Equals });
if (is<MemberExpression>(*expression))
alias = static_ptr_cast<MemberExpression>(expression);
else if (is<Identifier>(*expression))
name = static_ptr_cast<Identifier>(expression);
else
syntax_error("Invalid destructuring assignment target", expression_position);
LibJS: Allow BigInts as destructuring property names These are simply treated as their numerical value which means that above 2^32 - 1 they are strings.
2022-08-19 22:24:30 +00:00
} else if (match_identifier_name() || match(TokenType::StringLiteral) || match(TokenType::NumericLiteral) || match(TokenType::BigIntLiteral)) {
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
if (match(TokenType::StringLiteral) || match(TokenType::NumericLiteral))
needs_alias = true;
if (match(TokenType::StringLiteral)) {
auto token = consume(TokenType::StringLiteral);
auto string_literal = parse_string_literal(token);
name = create_ast_node<Identifier>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
string_literal->value());
LibJS: Allow BigInts as destructuring property names These are simply treated as their numerical value which means that above 2^32 - 1 they are strings.
2022-08-19 22:24:30 +00:00
} else if (match(TokenType::BigIntLiteral)) {
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
auto string_value = consume().DeprecatedFlyString_value();
LibJS: Allow BigInts as destructuring property names These are simply treated as their numerical value which means that above 2^32 - 1 they are strings.
2022-08-19 22:24:30 +00:00
VERIFY(string_value.ends_with("n"sv));
name = create_ast_node<Identifier>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString(string_value.view().substring_view(0, string_value.length() - 1)));
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
} else {
name = create_ast_node<Identifier>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
consume().DeprecatedFlyString_value());
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
} else if (match(TokenType::BracketOpen)) {
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
consume();
LibJS: Add optional extra strict checks in parse_binding_pattern
2021-07-28 23:49:25 +00:00
auto expression = parse_expression(0);
name = move(expression);
LibJS: Fix computed property ending token in binding pattern parsing The syntax is supposed to be '[expression]', not '[expression['.
2021-07-11 10:19:42 +00:00
consume(TokenType::BracketClose);
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
} else {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("identifier or computed property name");
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
return {};
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
if (!is_rest && match(TokenType::Colon)) {
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
consume();
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
if (allow_member_expressions == AllowMemberExpressions::Yes) {
auto expression_position = position();
auto expression = parse_expression(2, Associativity::Right, { TokenType::Equals });
if (is<ArrayExpression>(*expression) || is<ObjectExpression>(*expression)) {
if (auto synthesized_binding_pattern = synthesize_binding_pattern(*expression))
alias = synthesized_binding_pattern.release_nonnull();
else
syntax_error("Invalid destructuring assignment target", expression_position);
} else if (is<MemberExpression>(*expression)) {
alias = static_ptr_cast<MemberExpression>(expression);
} else if (is<Identifier>(*expression)) {
alias = static_ptr_cast<Identifier>(expression);
} else {
syntax_error("Invalid destructuring assignment target", expression_position);
}
} else if (match(TokenType::CurlyOpen) || match(TokenType::BracketOpen)) {
auto binding_pattern = parse_binding_pattern(allow_duplicates, allow_member_expressions);
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
if (!binding_pattern)
return {};
alias = binding_pattern.release_nonnull();
} else if (match_identifier_name()) {
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
alias = create_ast_node<Identifier>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
consume().DeprecatedFlyString_value());
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
} else {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("identifier or binding pattern");
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
return {};
}
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
} else if (needs_alias) {
expected("alias for string or numeric literal name");
return {};
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
}
} else {
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
if (allow_member_expressions == AllowMemberExpressions::Yes) {
auto expression_position = position();
auto expression = parse_expression(2, Associativity::Right, { TokenType::Equals });
if (is<ArrayExpression>(*expression) || is<ObjectExpression>(*expression)) {
if (auto synthesized_binding_pattern = synthesize_binding_pattern(*expression))
alias = synthesized_binding_pattern.release_nonnull();
else
syntax_error("Invalid destructuring assignment target", expression_position);
} else if (is<MemberExpression>(*expression)) {
alias = static_ptr_cast<MemberExpression>(expression);
} else if (is<Identifier>(*expression)) {
alias = static_ptr_cast<Identifier>(expression);
} else {
syntax_error("Invalid destructuring assignment target", expression_position);
LibJS: Add optional extra strict checks in parse_binding_pattern
2021-07-28 23:49:25 +00:00
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
} else if (match(TokenType::BracketOpen) || match(TokenType::CurlyOpen)) {
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
auto pattern = parse_binding_pattern(allow_duplicates, allow_member_expressions);
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
if (!pattern) {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("binding pattern");
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
return {};
}
alias = pattern.release_nonnull();
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
} else if (match_identifier_name()) {
// BindingElement must always have an Empty name field
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
auto identifier_name = consume_identifier().DeprecatedFlyString_value();
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
alias = create_ast_node<Identifier>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
identifier_name);
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
} else {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("identifier or binding pattern");
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
return {};
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
}
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
if (match(TokenType::Equals)) {
if (is_rest) {
syntax_error("Unexpected initializer after rest element");
return {};
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
consume();
initializer = parse_expression(2);
if (!initializer) {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("initialization expression");
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
return {};
}
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
entries.append(BindingPattern::BindingEntry { move(name), move(alias), move(initializer), is_rest });
if (match(TokenType::Comma)) {
if (is_rest) {
syntax_error("Rest element may not be followed by a comma");
return {};
}
consume();
LibJS: Disallow member expression in binding pattern as parameters
2021-11-26 22:37:14 +00:00
} else if (is_object && !match(TokenType::CurlyClose)) {
consume(TokenType::Comma);
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
}
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
}
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
while (!is_object && match(TokenType::Comma))
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
consume();
consume(closing_token);
LibJS: Restructure and fully implement BindingPatterns
2021-06-13 01:04:28 +00:00
auto kind = is_object ? BindingPattern::Kind::Object : BindingPattern::Kind::Array;
auto pattern = adopt_ref(*new BindingPattern);
pattern->entries = move(entries);
pattern->kind = kind;
LibJS: Add optional extra strict checks in parse_binding_pattern
2021-07-28 23:49:25 +00:00
Vector<StringView> bound_names;
pattern->for_each_bound_name([&](auto& name) {
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
if (allow_duplicates == AllowDuplicates::No) {
LibJS: Add optional extra strict checks in parse_binding_pattern
2021-07-28 23:49:25 +00:00
if (bound_names.contains_slow(name))
syntax_error("Duplicate parameter names in bindings");
bound_names.append(name);
}
check_identifier_name_for_assignment_validity(name);
});
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
return pattern;
}
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
RefPtr<Identifier> Parser::parse_lexical_binding()
{
auto binding_start = push_start();
if (match_identifier()) {
auto name = consume_identifier().DeprecatedFlyString_value();
return create_ast_node<Identifier>(
{ m_source_code, binding_start.position(), position() },
name);
}
if (!m_state.in_generator_function_context && match(TokenType::Yield)) {
if (m_state.strict_mode)
syntax_error("Identifier must not be a reserved word in strict mode ('yield')");
return create_ast_node<Identifier>(
{ m_source_code, binding_start.position(), position() },
consume().DeprecatedFlyString_value());
}
if (!m_state.await_expression_is_valid && match(TokenType::Async)) {
if (m_program_type == Program::Type::Module)
syntax_error("Identifier must not be a reserved word in modules ('async')");
return create_ast_node<Identifier>(
{ m_source_code, binding_start.position(), position() },
consume().DeprecatedFlyString_value());
}
return {};
}
NonnullRefPtr<VariableDeclaration> Parser::parse_variable_declaration(IsForLoopVariableDeclaration is_for_loop_variable_declaration)
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: rename JS::DeclarationType => JS::DeclarationKind Many other parsers call it with this name. Also Type can be confusing in this context since the DeclarationType is not the type (number, string, etc.) of the variables that are being declared by the VariableDeclaration.
2020-04-08 09:59:18 +00:00
DeclarationKind declaration_kind;
LibJS/Parser: Parse let declarations
2020-03-12 11:16:48 +00:00
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
switch (m_state.current_token.type()) {
LibJS/Parser: Parse let declarations
2020-03-12 11:16:48 +00:00
case TokenType::Var:
LibJS: rename JS::DeclarationType => JS::DeclarationKind Many other parsers call it with this name. Also Type can be confusing in this context since the DeclarationType is not the type (number, string, etc.) of the variables that are being declared by the VariableDeclaration.
2020-04-08 09:59:18 +00:00
declaration_kind = DeclarationKind::Var;
LibJS/Parser: Parse let declarations
2020-03-12 11:16:48 +00:00
break;
case TokenType::Let:
LibJS: rename JS::DeclarationType => JS::DeclarationKind Many other parsers call it with this name. Also Type can be confusing in this context since the DeclarationType is not the type (number, string, etc.) of the variables that are being declared by the VariableDeclaration.
2020-04-08 09:59:18 +00:00
declaration_kind = DeclarationKind::Let;
LibJS/Parser: Parse let declarations
2020-03-12 11:16:48 +00:00
break;
LibJS: Implement const variable declarations This also tightens the means of redeclaration of a variable by proxy, since we now have a way of knowing how a variable was initially declared, we can check if it was declared using `let` or `const` and not tolerate redeclaration like we did previously.
2020-03-12 12:24:34 +00:00
case TokenType::Const:
LibJS: rename JS::DeclarationType => JS::DeclarationKind Many other parsers call it with this name. Also Type can be confusing in this context since the DeclarationType is not the type (number, string, etc.) of the variables that are being declared by the VariableDeclaration.
2020-04-08 09:59:18 +00:00
declaration_kind = DeclarationKind::Const;
LibJS: Implement const variable declarations This also tightens the means of redeclaration of a variable by proxy, since we now have a way of knowing how a variable was initially declared, we can check if it was declared using `let` or `const` and not tolerate redeclaration like we did previously.
2020-03-12 12:24:34 +00:00
break;
LibJS/Parser: Parse let declarations
2020-03-12 11:16:48 +00:00
default:
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 19:42:32 +00:00
VERIFY_NOT_REACHED();
LibJS/Parser: Parse let declarations
2020-03-12 11:16:48 +00:00
}
LibJS: Require initializer for 'const' variable declaration
2020-10-30 19:08:45 +00:00
consume();
LibJS: Support VariableDeclaration with multiple declarators This patch adds support in the parser and interpreter for this: var a = 1, b = 2, c = a + b; VariableDeclaration is now a sequence of VariableDeclarators. :^)
2020-04-04 19:46:25 +00:00
NonnullRefPtrVector<VariableDeclarator> declarations;
for (;;) {
Libraries: Use AK::Variant default initialization where appropriate
2021-09-19 21:00:45 +00:00
Variant<NonnullRefPtr<Identifier>, NonnullRefPtr<BindingPattern>, Empty> target {};
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
if (auto pattern = parse_binding_pattern(declaration_kind != DeclarationKind::Var ? AllowDuplicates::No : AllowDuplicates::Yes, AllowMemberExpressions::No)) {
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
if ((declaration_kind == DeclarationKind::Let || declaration_kind == DeclarationKind::Const)) {
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
pattern->for_each_bound_name([this](auto& name) {
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
if (name == "let"sv)
syntax_error("Lexical binding may not be called 'let'");
});
}
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
target = pattern.release_nonnull();
} else if (auto lexical_binding = parse_lexical_binding()) {
check_identifier_name_for_assignment_validity(lexical_binding->string());
if ((declaration_kind == DeclarationKind::Let || declaration_kind == DeclarationKind::Const) && lexical_binding->string() == "let"sv)
syntax_error("Lexical binding may not be called 'let'");
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
target = lexical_binding.release_nonnull();
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
}
if (target.has<Empty>()) {
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("identifier or a binding pattern");
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
if (match(TokenType::Comma)) {
consume();
continue;
}
break;
}
LibJS: Support VariableDeclaration with multiple declarators This patch adds support in the parser and interpreter for this: var a = 1, b = 2, c = a + b; VariableDeclaration is now a sequence of VariableDeclarators. :^)
2020-04-04 19:46:25 +00:00
RefPtr<Expression> init;
if (match(TokenType::Equals)) {
consume();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
// In a for loop 'in' can be ambiguous so we do not allow it
// 14.7.4 The for Statement, https://tc39.es/ecma262/#prod-ForStatement and 14.7.5 The for-in, for-of, and for-await-of Statements, https://tc39.es/ecma262/#prod-ForInOfStatement
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
if (is_for_loop_variable_declaration == IsForLoopVariableDeclaration::Yes)
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
init = parse_expression(2, Associativity::Right, { TokenType::In });
else
init = parse_expression(2);
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
} else if (is_for_loop_variable_declaration == IsForLoopVariableDeclaration::No && declaration_kind == DeclarationKind::Const) {
LibJS: Require initializer for 'const' variable declaration
2020-10-30 19:08:45 +00:00
syntax_error("Missing initializer in 'const' variable declaration");
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
} else if (is_for_loop_variable_declaration == IsForLoopVariableDeclaration::No && target.has<NonnullRefPtr<BindingPattern>>()) {
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
syntax_error("Missing initializer in destructuring assignment");
LibJS: Support VariableDeclaration with multiple declarators This patch adds support in the parser and interpreter for this: var a = 1, b = 2, c = a + b; VariableDeclaration is now a sequence of VariableDeclarators. :^)
2020-04-04 19:46:25 +00:00
}
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
declarations.append(create_ast_node<VariableDeclarator>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
AK: Undo bogus Variant::downcast() rename I accidentally renamed these to verify_cast() when doing the global AK::downcast() rename.
2021-06-26 19:22:49 +00:00
move(target).downcast<NonnullRefPtr<Identifier>, NonnullRefPtr<BindingPattern>>(),
LibJS: Implement destructuring assignments and function parameters
2021-05-29 11:33:19 +00:00
move(init)));
LibJS: Support VariableDeclaration with multiple declarators This patch adds support in the parser and interpreter for this: var a = 1, b = 2, c = a + b; VariableDeclaration is now a sequence of VariableDeclarators. :^)
2020-04-04 19:46:25 +00:00
if (match(TokenType::Comma)) {
consume();
continue;
}
break;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
if (is_for_loop_variable_declaration == IsForLoopVariableDeclaration::No)
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
consume_or_insert_semicolon();
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
LibJS: Call shrink_to_fit() on various Vectors created during parse Vectors that stick around in the AST were wasting a fair bit of memory due to the growth padding we keep by default. This patch goes after some of these vectors with the shrink_to_fit() stick to reduce waste. Since the AST can stay around for a long time, it is worth making an effort to shrink it down when we have a chance.
2022-11-26 19:45:06 +00:00
declarations.shrink_to_fit();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto declaration = create_ast_node<VariableDeclaration>({ m_source_code, rule_start.position(), position() }, declaration_kind, move(declarations));
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 14:42:54 +00:00
return declaration;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
NonnullRefPtr<UsingDeclaration> Parser::parse_using_declaration(IsForLoopVariableDeclaration is_for_loop_variable_declaration)
{
// using [no LineTerminator here] BindingList[?In, ?Yield, ?Await, +Using] ;
auto rule_start = push_start();
VERIFY(m_state.current_token.original_value() == "using"sv);
consume(TokenType::Identifier);
VERIFY(!m_state.current_token.trivia_contains_line_terminator());
NonnullRefPtrVector<VariableDeclarator> declarations;
for (;;) {
auto lexical_binding = parse_lexical_binding();
if (!lexical_binding) {
expected("lexical binding");
break;
}
check_identifier_name_for_assignment_validity(lexical_binding->string());
if (lexical_binding->string() == "let"sv)
syntax_error("Lexical binding may not be called 'let'");
RefPtr<Expression> initializer;
if (match(TokenType::Equals)) {
consume();
if (is_for_loop_variable_declaration == IsForLoopVariableDeclaration::Yes)
initializer = parse_expression(2, Associativity::Right, { TokenType::In });
else
initializer = parse_expression(2);
} else if (is_for_loop_variable_declaration == IsForLoopVariableDeclaration::No) {
consume(TokenType::Equals);
}
declarations.append(create_ast_node<VariableDeclarator>(
{ m_source_code, rule_start.position(), position() },
lexical_binding.release_nonnull(),
move(initializer)));
if (match(TokenType::Comma)) {
consume();
continue;
}
break;
}
if (is_for_loop_variable_declaration == IsForLoopVariableDeclaration::No)
consume_or_insert_semicolon();
return create_ast_node<UsingDeclaration>({ m_source_code, rule_start.position(), position() }, move(declarations));
}
LibJS: Implement "throw" You can now throw an expression to the nearest catcher! :^) To support throwing arbitrary values, I added an Exception class that sits as a wrapper around whatever is thrown. In the future it will be a logical place to store a call stack.
2020-03-24 21:03:50 +00:00
NonnullRefPtr<ThrowStatement> Parser::parse_throw_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Implement "throw" You can now throw an expression to the nearest catcher! :^) To support throwing arbitrary values, I added an Exception class that sits as a wrapper around whatever is thrown. In the future it will be a logical place to store a call stack.
2020-03-24 21:03:50 +00:00
consume(TokenType::Throw);
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
// Automatic semicolon insertion: terminate statement when throw is followed by newline
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (m_state.current_token.trivia_contains_line_terminator()) {
LibJS: Add Parser::syntax_error() helper Instead of having fprintf()s all over the place we can now use syntax_error("message") or syntax_error("message", line, column). This takes care of a consistent format, appending a newline and getting the line number and column of the current token if the last two params are omitted.
2020-04-29 20:37:51 +00:00
syntax_error("No line break is allowed between 'throw' and its expression");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ThrowStatement>({ m_source_code, rule_start.position(), position() }, create_ast_node<ErrorExpression>({ m_source_code, rule_start.position(), position() }));
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
}
auto expression = parse_expression(0);
consume_or_insert_semicolon();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ThrowStatement>({ m_source_code, rule_start.position(), position() }, move(expression));
LibJS: Implement "throw" You can now throw an expression to the nearest catcher! :^) To support throwing arbitrary values, I added an Exception class that sits as a wrapper around whatever is thrown. In the future it will be a logical place to store a call stack.
2020-03-24 21:03:50 +00:00
}
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
NonnullRefPtr<BreakStatement> Parser::parse_break_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
consume(TokenType::Break);
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString target_label;
LibJS: Parse labels in continue and break statements
2020-05-28 19:02:32 +00:00
if (match(TokenType::Semicolon)) {
consume();
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
} else {
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
if (!m_state.current_token.trivia_contains_line_terminator() && match_identifier()) {
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
target_label = consume().value();
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
auto label = m_state.labels_in_scope.find(target_label);
if (label == m_state.labels_in_scope.end())
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Label '{}' not found", target_label));
LibJS: break or continue with nonexistent label is a syntax error
2020-10-08 17:49:08 +00:00
}
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
consume_or_insert_semicolon();
LibJS: Parse labels in continue and break statements
2020-05-28 19:02:32 +00:00
}
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (target_label.is_null() && !m_state.in_break_context)
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
syntax_error("Unlabeled 'break' not allowed outside of a loop or switch statement");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<BreakStatement>({ m_source_code, rule_start.position(), position() }, target_label);
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
}
LibJS: Add support for "continue" inside "for" statements :^)
2020-04-04 22:22:42 +00:00
NonnullRefPtr<ContinueStatement> Parser::parse_continue_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (!m_state.in_continue_context)
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
syntax_error("'continue' not allow outside of a loop");
LibJS: Add support for "continue" inside "for" statements :^)
2020-04-04 22:22:42 +00:00
consume(TokenType::Continue);
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString target_label;
LibJS: Parse labels in continue and break statements
2020-05-28 19:02:32 +00:00
if (match(TokenType::Semicolon)) {
consume();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ContinueStatement>({ m_source_code, rule_start.position(), position() }, target_label);
LibJS: Parse labels in continue and break statements
2020-05-28 19:02:32 +00:00
}
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
if (!m_state.current_token.trivia_contains_line_terminator() && match_identifier()) {
auto label_position = position();
LibJS: Parse labels in continue and break statements
2020-05-28 19:02:32 +00:00
target_label = consume().value();
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
auto label = m_state.labels_in_scope.find(target_label);
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
if (label == m_state.labels_in_scope.end())
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Label '{}' not found or invalid", target_label));
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 21:01:54 +00:00
else
label->value = label_position;
LibJS: break or continue with nonexistent label is a syntax error
2020-10-08 17:49:08 +00:00
}
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
consume_or_insert_semicolon();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ContinueStatement>({ m_source_code, rule_start.position(), position() }, target_label);
LibJS: Add support for "continue" inside "for" statements :^)
2020-04-04 22:22:42 +00:00
}
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
NonnullRefPtr<ConditionalExpression> Parser::parse_conditional_expression(NonnullRefPtr<Expression> test, ForbiddenTokens forbidden)
LibJS: Implement ConditionalExpression (ternary "?:" operator)
2020-04-03 10:14:28 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Implement ConditionalExpression (ternary "?:" operator)
2020-04-03 10:14:28 +00:00
consume(TokenType::QuestionMark);
LibJS: Fix conditional expression precedence This fixes the following from parsing incorrectly due to the comma that occurs after the conditional: let o = { foo: true ? 1 : 2, bar: 'baz', };
2020-05-28 21:42:20 +00:00
auto consequent = parse_expression(2);
LibJS: Implement ConditionalExpression (ternary "?:" operator)
2020-04-03 10:14:28 +00:00
consume(TokenType::Colon);
LibJS: Fix cases where we incorrectly allowed 'in' in for loops We needed to propagate the forbidden token set to all parsing functions that can call back into parse_expression.
2022-02-15 10:11:56 +00:00
auto alternate = parse_expression(2, Associativity::Right, forbidden);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ConditionalExpression>({ m_source_code, rule_start.position(), position() }, move(test), move(consequent), move(alternate));
LibJS: Implement ConditionalExpression (ternary "?:" operator)
2020-04-03 10:14:28 +00:00
}
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
NonnullRefPtr<OptionalChain> Parser::parse_optional_chain(NonnullRefPtr<Expression> base)
{
auto rule_start = push_start();
Vector<OptionalChain::Reference> chain;
do {
if (match(TokenType::QuestionMarkPeriod)) {
consume(TokenType::QuestionMarkPeriod);
switch (m_state.current_token.type()) {
case TokenType::ParenOpen:
chain.append(OptionalChain::Call { parse_arguments(), OptionalChain::Mode::Optional });
break;
case TokenType::BracketOpen:
consume();
chain.append(OptionalChain::ComputedReference { parse_expression(0), OptionalChain::Mode::Optional });
consume(TokenType::BracketClose);
break;
LibJS: Implement private identifiers in optional chains
2021-10-18 21:32:47 +00:00
case TokenType::PrivateIdentifier: {
if (!is_private_identifier_valid())
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Reference to undeclared private field or method '{}'", m_state.current_token.value()));
LibJS: Implement private identifiers in optional chains
2021-10-18 21:32:47 +00:00
auto start = position();
auto private_identifier = consume();
chain.append(OptionalChain::PrivateMemberReference {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
create_ast_node<PrivateIdentifier>({ m_source_code, start, position() }, private_identifier.value()),
LibJS: Implement private identifiers in optional chains
2021-10-18 21:32:47 +00:00
OptionalChain::Mode::Optional });
break;
}
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
case TokenType::TemplateLiteralStart:
// 13.3.1.1 - Static Semantics: Early Errors
// OptionalChain :
// ?. TemplateLiteral
// OptionalChain TemplateLiteral
// This is a hard error.
syntax_error("Invalid tagged template literal after ?.", position());
break;
default:
if (match_identifier_name()) {
auto start = position();
auto identifier = consume();
chain.append(OptionalChain::MemberReference {
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
create_ast_node<Identifier>({ m_source_code, start, position() }, identifier.DeprecatedFlyString_value()),
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
OptionalChain::Mode::Optional,
});
} else {
syntax_error("Invalid optional chain reference after ?.", position());
}
break;
}
} else if (match(TokenType::ParenOpen)) {
chain.append(OptionalChain::Call { parse_arguments(), OptionalChain::Mode::NotOptional });
} else if (match(TokenType::Period)) {
consume();
LibJS: Implement private identifiers in optional chains
2021-10-18 21:32:47 +00:00
if (match(TokenType::PrivateIdentifier)) {
LibJS: Make sure private identifier is valid in optional chain If we don't check that a private identifier is valid this can break the assumption that we have a private environment when evaluation the private identifier. Also an unknown private identifier this should be a SyntaxError.
2022-11-17 09:24:20 +00:00
if (!is_private_identifier_valid())
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Reference to undeclared private field or method '{}'", m_state.current_token.value()));
LibJS: Make sure private identifier is valid in optional chain If we don't check that a private identifier is valid this can break the assumption that we have a private environment when evaluation the private identifier. Also an unknown private identifier this should be a SyntaxError.
2022-11-17 09:24:20 +00:00
LibJS: Implement private identifiers in optional chains
2021-10-18 21:32:47 +00:00
auto start = position();
auto private_identifier = consume();
chain.append(OptionalChain::PrivateMemberReference {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
create_ast_node<PrivateIdentifier>({ m_source_code, start, position() }, private_identifier.value()),
LibJS: Implement private identifiers in optional chains
2021-10-18 21:32:47 +00:00
OptionalChain::Mode::NotOptional,
});
} else if (match_identifier_name()) {
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
auto start = position();
auto identifier = consume();
chain.append(OptionalChain::MemberReference {
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
create_ast_node<Identifier>({ m_source_code, start, position() }, identifier.DeprecatedFlyString_value()),
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
OptionalChain::Mode::NotOptional,
});
} else {
expected("an identifier");
break;
}
} else if (match(TokenType::TemplateLiteralStart)) {
// 13.3.1.1 - Static Semantics: Early Errors
// OptionalChain :
// ?. TemplateLiteral
// OptionalChain TemplateLiteral
syntax_error("Invalid tagged template literal after optional chain", position());
break;
} else if (match(TokenType::BracketOpen)) {
consume();
chain.append(OptionalChain::ComputedReference { parse_expression(2), OptionalChain::Mode::NotOptional });
consume(TokenType::BracketClose);
} else {
break;
}
} while (!done());
return create_ast_node<OptionalChain>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
move(base),
move(chain));
}
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
NonnullRefPtr<TryStatement> Parser::parse_try_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
consume(TokenType::Try);
auto block = parse_block_statement();
LibJS: Allow try statement with only finally clause This was a regression introduced by 9ffe45b - a TryStatement without 'catch' clause *is* allowed, if it has a 'finally' clause. It is now checked properly that at least one of both is present.
2020-10-23 19:54:00 +00:00
RefPtr<CatchClause> handler;
if (match(TokenType::Catch))
handler = parse_catch_clause();
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
RefPtr<BlockStatement> finalizer;
if (match(TokenType::Finally)) {
consume();
finalizer = parse_block_statement();
}
LibJS: Allow try statement with only finally clause This was a regression introduced by 9ffe45b - a TryStatement without 'catch' clause *is* allowed, if it has a 'finally' clause. It is now checked properly that at least one of both is present.
2020-10-23 19:54:00 +00:00
if (!handler && !finalizer)
syntax_error("try statement must have a 'catch' or 'finally' clause");
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<TryStatement>({ m_source_code, rule_start.position(), position() }, move(block), move(handler), move(finalizer));
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
}
LibJS: Add support for do..while statements
2020-04-04 19:29:23 +00:00
NonnullRefPtr<DoWhileStatement> Parser::parse_do_while_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Add support for do..while statements
2020-04-04 19:29:23 +00:00
consume(TokenType::Do);
LibJS: Fix do..while parsing by consuming parentheses explicitly (#1652) Before this patch the parser accepted conditions without enclosing parentheses (like: .."while number < 9").
2020-04-05 14:57:01 +00:00
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
auto body = [&]() -> NonnullRefPtr<Statement> {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
TemporaryChange break_change(m_state.in_break_context, true);
TemporaryChange continue_change(m_state.in_continue_context, true);
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
return parse_statement();
}();
LibJS: Fix do..while parsing by consuming parentheses explicitly (#1652) Before this patch the parser accepted conditions without enclosing parentheses (like: .."while number < 9").
2020-04-05 14:57:01 +00:00
LibJS: Add support for do..while statements
2020-04-04 19:29:23 +00:00
consume(TokenType::While);
LibJS: Fix do..while parsing by consuming parentheses explicitly (#1652) Before this patch the parser accepted conditions without enclosing parentheses (like: .."while number < 9").
2020-04-05 14:57:01 +00:00
consume(TokenType::ParenOpen);
LibJS: Add support for do..while statements
2020-04-04 19:29:23 +00:00
auto test = parse_expression(0);
LibJS: Fix do..while parsing by consuming parentheses explicitly (#1652) Before this patch the parser accepted conditions without enclosing parentheses (like: .."while number < 9").
2020-04-05 14:57:01 +00:00
consume(TokenType::ParenClose);
LibJS: Always insert semicolon after do-while statement if missing https://tc39.es/ecma262/#sec-additions-and-changes-that-introduce-incompatibilities-with-prior-editions 11.9.1: In ECMAScript 2015, Automatic Semicolon Insertion adds a semicolon at the end of a do-while statement if the semicolon is missing. This change aligns the specification with the actual behaviour of most existing implementations.
2020-10-27 19:16:23 +00:00
// Since ES 2015 a missing semicolon is inserted here, despite the regular ASI rules not applying
if (match(TokenType::Semicolon))
consume();
LibJS: Fix do..while parsing by consuming parentheses explicitly (#1652) Before this patch the parser accepted conditions without enclosing parentheses (like: .."while number < 9").
2020-04-05 14:57:01 +00:00
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<DoWhileStatement>({ m_source_code, rule_start.position(), position() }, move(test), move(body));
LibJS: Add support for do..while statements
2020-04-04 19:29:23 +00:00
}
LibJS: Parse while statements
2020-04-21 18:27:57 +00:00
NonnullRefPtr<WhileStatement> Parser::parse_while_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Parse while statements
2020-04-21 18:27:57 +00:00
consume(TokenType::While);
consume(TokenType::ParenOpen);
auto test = parse_expression(0);
consume(TokenType::ParenClose);
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
TemporaryChange break_change(m_state.in_break_context, true);
TemporaryChange continue_change(m_state.in_continue_context, true);
LibJS: Parse while statements
2020-04-21 18:27:57 +00:00
auto body = parse_statement();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<WhileStatement>({ m_source_code, rule_start.position(), position() }, move(test), move(body));
LibJS: Parse while statements
2020-04-21 18:27:57 +00:00
}
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
NonnullRefPtr<SwitchStatement> Parser::parse_switch_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
consume(TokenType::Switch);
consume(TokenType::ParenOpen);
auto determinant = parse_expression(0);
consume(TokenType::ParenClose);
consume(TokenType::CurlyOpen);
NonnullRefPtrVector<SwitchCase> cases;
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto switch_statement = create_ast_node<SwitchStatement>({ m_source_code, rule_start.position(), position() }, move(determinant));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopePusher switch_scope = ScopePusher::block_scope(*this, switch_statement);
LibJS: Multiple 'default' clauses in switch statement are a syntax error
2020-10-18 22:12:27 +00:00
auto has_default = false;
while (match(TokenType::Case) || match(TokenType::Default)) {
if (match(TokenType::Default)) {
if (has_default)
syntax_error("Multiple 'default' clauses in switch statement");
has_default = true;
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
switch_statement->add_case(parse_switch_case());
LibJS: Multiple 'default' clauses in switch statement are a syntax error
2020-10-18 22:12:27 +00:00
}
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
consume(TokenType::CurlyClose);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
return switch_statement;
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
}
LibJS: Parse "with" statements :^)
2020-11-28 14:05:57 +00:00
NonnullRefPtr<WithStatement> Parser::parse_with_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Parse "with" statements :^)
2020-11-28 14:05:57 +00:00
consume(TokenType::With);
consume(TokenType::ParenOpen);
auto object = parse_expression(0);
consume(TokenType::ParenClose);
auto body = parse_statement();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<WithStatement>({ m_source_code, rule_start.position(), position() }, move(object), move(body));
LibJS: Parse "with" statements :^)
2020-11-28 14:05:57 +00:00
}
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
NonnullRefPtr<SwitchCase> Parser::parse_switch_case()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
RefPtr<Expression> test;
if (consume().type() == TokenType::Case) {
test = parse_expression(0);
}
consume(TokenType::Colon);
NonnullRefPtrVector<Statement> consequent;
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
TemporaryChange break_change(m_state.in_break_context, true);
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto switch_case = create_ast_node<SwitchCase>({ m_source_code, rule_start.position(), position() }, move(test));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
parse_statement_list(switch_case);
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
return switch_case;
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
}
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
NonnullRefPtr<CatchClause> Parser::parse_catch_clause()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
consume(TokenType::Catch);
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString parameter;
LibJS: Add support for binding patterns in catch clauses `try { ... } catch({a=foo}) {}` is valid, and now we parse and evaluate it correctly :^)
2021-07-11 10:45:38 +00:00
RefPtr<BindingPattern> pattern_parameter;
auto should_expect_parameter = false;
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
if (match(TokenType::ParenOpen)) {
LibJS: Add support for binding patterns in catch clauses `try { ... } catch({a=foo}) {}` is valid, and now we parse and evaluate it correctly :^)
2021-07-11 10:45:38 +00:00
should_expect_parameter = true;
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
consume();
LibJS: Disallow await keywords in static init blocks In static init blocks 'await' cannot be used. Note that this does not cover all the cases since the parser currently cannot distinguish between expressions within parenthesis and direct expressions.
2021-11-26 22:29:05 +00:00
if (match_identifier_name()
&& (!match(TokenType::Yield) || !m_state.in_generator_function_context)
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
&& (!match(TokenType::Async) || !m_state.await_expression_is_valid)
LibJS: Disallow await keywords in static init blocks In static init blocks 'await' cannot be used. Note that this does not cover all the cases since the parser currently cannot distinguish between expressions within parenthesis and direct expressions.
2021-11-26 22:29:05 +00:00
&& (!match(TokenType::Await) || !m_state.in_class_static_init_block))
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
parameter = consume().value();
LibJS: Add support for binding patterns in catch clauses `try { ... } catch({a=foo}) {}` is valid, and now we parse and evaluate it correctly :^)
2021-07-11 10:45:38 +00:00
else
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-17 23:11:32 +00:00
pattern_parameter = parse_binding_pattern(AllowDuplicates::No, AllowMemberExpressions::No);
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
consume(TokenType::ParenClose);
}
LibJS: Add support for binding patterns in catch clauses `try { ... } catch({a=foo}) {}` is valid, and now we parse and evaluate it correctly :^)
2021-07-11 10:45:38 +00:00
if (should_expect_parameter && parameter.is_empty() && !pattern_parameter)
LibJS: Use expected() instead of syntax_error("Expected ...")
2021-07-11 19:00:55 +00:00
expected("an identifier or a binding pattern");
LibJS: Add support for binding patterns in catch clauses `try { ... } catch({a=foo}) {}` is valid, and now we parse and evaluate it correctly :^)
2021-07-11 10:45:38 +00:00
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
HashTable<DeprecatedFlyString> bound_names;
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (pattern_parameter) {
pattern_parameter->for_each_bound_name(
[&](auto& name) {
check_identifier_name_for_assignment_validity(name);
bound_names.set(name);
});
}
if (!parameter.is_empty()) {
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
check_identifier_name_for_assignment_validity(parameter);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
bound_names.set(parameter);
}
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
LibJS: Treat the Catch binding identifier as a var binding
2021-10-08 10:04:13 +00:00
ScopePusher catch_scope = ScopePusher::catch_scope(*this, pattern_parameter, parameter);
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
auto body = parse_block_statement();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
body->for_each_lexically_declared_name([&](auto const& name) {
if (bound_names.contains(name))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Identifier '{}' already declared as catch parameter", name));
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
});
LibJS: Add support for binding patterns in catch clauses `try { ... } catch({a=foo}) {}` is valid, and now we parse and evaluate it correctly :^)
2021-07-11 10:45:38 +00:00
if (pattern_parameter) {
return create_ast_node<CatchClause>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Add support for binding patterns in catch clauses `try { ... } catch({a=foo}) {}` is valid, and now we parse and evaluate it correctly :^)
2021-07-11 10:45:38 +00:00
pattern_parameter.release_nonnull(),
move(body));
}
return create_ast_node<CatchClause>(
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
{ m_source_code, rule_start.position(), position() },
LibJS: Add support for binding patterns in catch clauses `try { ... } catch({a=foo}) {}` is valid, and now we parse and evaluate it correctly :^)
2021-07-11 10:45:38 +00:00
move(parameter),
move(body));
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 13:03:55 +00:00
}
LibJS: Parse "if" statements This patch implements basic parsing of "if" statements. We don't yet support parsing "else", so I added a FIXME about that.
2020-03-21 17:40:17 +00:00
NonnullRefPtr<IfStatement> Parser::parse_if_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Function declarations in if statement clauses https://tc39.es/ecma262/#sec-functiondeclarations-in-ifstatement-statement-clauses B.3.4 FunctionDeclarations in IfStatement Statement Clauses The following augments the IfStatement production in 13.6: IfStatement[Yield, Await, Return] : if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else Statement[?Yield, ?Await, ?Return] if ( Expression[+In, ?Yield, ?Await] ) Statement[?Yield, ?Await, ?Return] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] This production only applies when parsing non-strict code. Code matching this production is processed as if each matching occurrence of FunctionDeclaration[?Yield, ?Await, ~Default] was the sole StatementListItem of a BlockStatement occupying that position in the source code. The semantics of such a synthetic BlockStatement includes the web legacy compatibility semantics specified in B.3.3.
2020-10-31 13:37:09 +00:00
auto parse_function_declaration_as_block_statement = [&] {
// https://tc39.es/ecma262/#sec-functiondeclarations-in-ifstatement-statement-clauses
LibJS: Update a couple of outdated spec comments These are editorial changes in the ECMA-262 spec. See: - https://github.com/tc39/ecma262/commit/e080a7f - https://github.com/tc39/ecma262/commit/c5a9094 - https://github.com/tc39/ecma262/commit/5091520 - https://github.com/tc39/ecma262/commit/1c6564b - https://github.com/tc39/ecma262/commit/e06c80c
2022-04-30 20:22:52 +00:00
// This production only applies when parsing non-strict code. Source text matched
// by this production is processed as if each matching occurrence of
LibJS: Function declarations in if statement clauses https://tc39.es/ecma262/#sec-functiondeclarations-in-ifstatement-statement-clauses B.3.4 FunctionDeclarations in IfStatement Statement Clauses The following augments the IfStatement production in 13.6: IfStatement[Yield, Await, Return] : if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else Statement[?Yield, ?Await, ?Return] if ( Expression[+In, ?Yield, ?Await] ) Statement[?Yield, ?Await, ?Return] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] This production only applies when parsing non-strict code. Code matching this production is processed as if each matching occurrence of FunctionDeclaration[?Yield, ?Await, ~Default] was the sole StatementListItem of a BlockStatement occupying that position in the source code. The semantics of such a synthetic BlockStatement includes the web legacy compatibility semantics specified in B.3.3.
2020-10-31 13:37:09 +00:00
// FunctionDeclaration[?Yield, ?Await, ~Default] was the sole StatementListItem
LibJS: Update a couple of outdated spec comments These are editorial changes in the ECMA-262 spec. See: - https://github.com/tc39/ecma262/commit/e080a7f - https://github.com/tc39/ecma262/commit/c5a9094 - https://github.com/tc39/ecma262/commit/5091520 - https://github.com/tc39/ecma262/commit/1c6564b - https://github.com/tc39/ecma262/commit/e06c80c
2022-04-30 20:22:52 +00:00
// of a BlockStatement occupying that position in the source text.
// The semantics of such a synthetic BlockStatement includes the web legacy
// compatibility semantics specified in B.3.2.
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
VERIFY(match(TokenType::Function));
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
auto block = create_ast_node<BlockStatement>({ m_source_code, rule_start.position(), position() });
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopePusher block_scope = ScopePusher::block_scope(*this, *block);
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
auto declaration = parse_declaration();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
VERIFY(m_state.current_scope_pusher);
block_scope.add_declaration(declaration);
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
VERIFY(is<FunctionDeclaration>(*declaration));
auto& function_declaration = static_cast<FunctionDeclaration const&>(*declaration);
if (function_declaration.kind() == FunctionKind::Generator)
syntax_error("Generator functions can only be declared in top-level or within a block");
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (function_declaration.kind() == FunctionKind::Async)
syntax_error("Async functions can only be declared in top-level or within a block");
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
block->append(move(declaration));
LibJS: Function declarations in if statement clauses https://tc39.es/ecma262/#sec-functiondeclarations-in-ifstatement-statement-clauses B.3.4 FunctionDeclarations in IfStatement Statement Clauses The following augments the IfStatement production in 13.6: IfStatement[Yield, Await, Return] : if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else Statement[?Yield, ?Await, ?Return] if ( Expression[+In, ?Yield, ?Await] ) Statement[?Yield, ?Await, ?Return] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] This production only applies when parsing non-strict code. Code matching this production is processed as if each matching occurrence of FunctionDeclaration[?Yield, ?Await, ~Default] was the sole StatementListItem of a BlockStatement occupying that position in the source code. The semantics of such a synthetic BlockStatement includes the web legacy compatibility semantics specified in B.3.3.
2020-10-31 13:37:09 +00:00
return block;
};
LibJS: Parse "if" statements This patch implements basic parsing of "if" statements. We don't yet support parsing "else", so I added a FIXME about that.
2020-03-21 17:40:17 +00:00
consume(TokenType::If);
consume(TokenType::ParenOpen);
auto predicate = parse_expression(0);
consume(TokenType::ParenClose);
LibJS: Function declarations in if statement clauses https://tc39.es/ecma262/#sec-functiondeclarations-in-ifstatement-statement-clauses B.3.4 FunctionDeclarations in IfStatement Statement Clauses The following augments the IfStatement production in 13.6: IfStatement[Yield, Await, Return] : if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else Statement[?Yield, ?Await, ?Return] if ( Expression[+In, ?Yield, ?Await] ) Statement[?Yield, ?Await, ?Return] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] This production only applies when parsing non-strict code. Code matching this production is processed as if each matching occurrence of FunctionDeclaration[?Yield, ?Await, ~Default] was the sole StatementListItem of a BlockStatement occupying that position in the source code. The semantics of such a synthetic BlockStatement includes the web legacy compatibility semantics specified in B.3.3.
2020-10-31 13:37:09 +00:00
RefPtr<Statement> consequent;
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (!m_state.strict_mode && match(TokenType::Function))
LibJS: Function declarations in if statement clauses https://tc39.es/ecma262/#sec-functiondeclarations-in-ifstatement-statement-clauses B.3.4 FunctionDeclarations in IfStatement Statement Clauses The following augments the IfStatement production in 13.6: IfStatement[Yield, Await, Return] : if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else Statement[?Yield, ?Await, ?Return] if ( Expression[+In, ?Yield, ?Await] ) Statement[?Yield, ?Await, ?Return] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] This production only applies when parsing non-strict code. Code matching this production is processed as if each matching occurrence of FunctionDeclaration[?Yield, ?Await, ~Default] was the sole StatementListItem of a BlockStatement occupying that position in the source code. The semantics of such a synthetic BlockStatement includes the web legacy compatibility semantics specified in B.3.3.
2020-10-31 13:37:09 +00:00
consequent = parse_function_declaration_as_block_statement();
else
consequent = parse_statement();
LibJS: Implement "else" parsing We can now handle scripts with if/else in LibJS. Most of the changes are about fixing IfStatement to store the consequent and alternate node as Statements. Interpreter now also runs Statements, rather than running ScopeNodes.
2020-03-23 15:46:41 +00:00
RefPtr<Statement> alternate;
if (match(TokenType::Else)) {
LibJS: Function declarations in if statement clauses https://tc39.es/ecma262/#sec-functiondeclarations-in-ifstatement-statement-clauses B.3.4 FunctionDeclarations in IfStatement Statement Clauses The following augments the IfStatement production in 13.6: IfStatement[Yield, Await, Return] : if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else Statement[?Yield, ?Await, ?Return] if ( Expression[+In, ?Yield, ?Await] ) Statement[?Yield, ?Await, ?Return] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] This production only applies when parsing non-strict code. Code matching this production is processed as if each matching occurrence of FunctionDeclaration[?Yield, ?Await, ~Default] was the sole StatementListItem of a BlockStatement occupying that position in the source code. The semantics of such a synthetic BlockStatement includes the web legacy compatibility semantics specified in B.3.3.
2020-10-31 13:37:09 +00:00
consume();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (!m_state.strict_mode && match(TokenType::Function))
LibJS: Function declarations in if statement clauses https://tc39.es/ecma262/#sec-functiondeclarations-in-ifstatement-statement-clauses B.3.4 FunctionDeclarations in IfStatement Statement Clauses The following augments the IfStatement production in 13.6: IfStatement[Yield, Await, Return] : if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else Statement[?Yield, ?Await, ?Return] if ( Expression[+In, ?Yield, ?Await] ) Statement[?Yield, ?Await, ?Return] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] else FunctionDeclaration[?Yield, ?Await, ~Default] if ( Expression[+In, ?Yield, ?Await] ) FunctionDeclaration[?Yield, ?Await, ~Default] This production only applies when parsing non-strict code. Code matching this production is processed as if each matching occurrence of FunctionDeclaration[?Yield, ?Await, ~Default] was the sole StatementListItem of a BlockStatement occupying that position in the source code. The semantics of such a synthetic BlockStatement includes the web legacy compatibility semantics specified in B.3.3.
2020-10-31 13:37:09 +00:00
alternate = parse_function_declaration_as_block_statement();
else
alternate = parse_statement();
LibJS: Implement "else" parsing We can now handle scripts with if/else in LibJS. Most of the changes are about fixing IfStatement to store the consequent and alternate node as Statements. Interpreter now also runs Statements, rather than running ScopeNodes.
2020-03-23 15:46:41 +00:00
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<IfStatement>({ m_source_code, rule_start.position(), position() }, move(predicate), move(*consequent), move(alternate));
LibJS: Parse "if" statements This patch implements basic parsing of "if" statements. We don't yet support parsing "else", so I added a FIXME about that.
2020-03-21 17:40:17 +00:00
}
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
NonnullRefPtr<Statement> Parser::parse_for_statement()
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
auto is_await_loop = IsForAwaitLoop::No;
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
auto match_of = [&](Token const& token) {
return token.type() == TokenType::Identifier && token.original_value() == "of"sv;
};
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
auto match_for_in_of = [&]() {
bool is_of = match_of(m_state.current_token);
if (is_await_loop == IsForAwaitLoop::Yes) {
if (!is_of)
syntax_error("for await loop is only valid with 'of'");
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
else if (!m_state.await_expression_is_valid)
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
syntax_error("for await loop is only valid in async function or generator");
return true;
}
return match(TokenType::In) || is_of;
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
};
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
consume(TokenType::For);
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
if (match(TokenType::Await)) {
consume();
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
if (!m_state.await_expression_is_valid)
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
syntax_error("for-await-of is only allowed in async function context");
is_await_loop = IsForAwaitLoop::Yes;
}
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
consume(TokenType::ParenOpen);
LibJS: Use ScopePusher to correctly push the scope in for statements We were previously pushing a scope but forgetting to actually set it as the current scope.
2021-09-16 17:39:28 +00:00
Optional<ScopePusher> scope_pusher;
LibJS: Fix runaway let scope when parsing for-in/of statements This requires variables that should be exported to the script host or to other scripts to be declared as var (such as in test-common.js), since lexically-scoped variables won't be visible.
2021-07-05 19:31:51 +00:00
LibJS: Consume semicolon at the end of a statement A bunch of code was relying on this not happenind, in particular the parsing of "for" statements. Reorganized things so they work again.
2020-03-23 18:08:32 +00:00
RefPtr<ASTNode> init;
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
if (!match(TokenType::Semicolon)) {
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
auto match_for_using_declaration = [&] {
if (!match(TokenType::Identifier) || m_state.current_token.original_value() != "using"sv)
return false;
auto lookahead = next_token();
if (lookahead.trivia_contains_line_terminator())
return false;
if (lookahead.original_value() == "of"sv)
return false;
return token_is_identifier(lookahead);
};
if (match_for_using_declaration()) {
auto declaration = parse_using_declaration(IsForLoopVariableDeclaration::Yes);
if (match_of(m_state.current_token)) {
if (declaration->declarations().size() != 1)
syntax_error("Must have exactly one declaration in for using of");
else if (declaration->declarations().first().init())
syntax_error("Using declaration cannot have initializer");
return parse_for_in_of_statement(move(declaration), is_await_loop);
}
if (match(TokenType::In))
syntax_error("Using declaration not allowed in for-in loop");
init = move(declaration);
} else if (match_variable_declaration()) {
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
auto declaration = parse_variable_declaration(IsForLoopVariableDeclaration::Yes);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (declaration->declaration_kind() == DeclarationKind::Var) {
m_state.current_scope_pusher->add_declaration(declaration);
} else {
// This does not follow the normal declaration structure so we need additional checks.
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
HashTable<DeprecatedFlyString> bound_names;
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
declaration->for_each_bound_name([&](auto const& name) {
if (bound_names.set(name) != AK::HashSetResult::InsertedNewEntry)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Identifier '{}' already declared in for loop initializer", name), declaration->source_range().start);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
});
}
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
if (match_for_in_of()) {
if (declaration->declarations().size() > 1)
syntax_error("Multiple declarations not allowed in for..in/of");
else if (declaration->declarations().size() < 1)
syntax_error("Need exactly one variable declaration in for..in/of");
return parse_for_in_of_statement(move(declaration), is_await_loop);
}
if (declaration->declaration_kind() == DeclarationKind::Const) {
for (auto const& variable : declaration->declarations()) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (!variable.init())
LibJS: Require initializer for 'const' variable declaration
2020-10-30 19:08:45 +00:00
syntax_error("Missing initializer in 'const' variable declaration");
}
}
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
init = move(declaration);
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
} else if (match_expression()) {
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
auto lookahead_token = next_token();
bool starts_with_async_of = match(TokenType::Async) && match_of(lookahead_token);
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
init = parse_expression(0, Associativity::Right, { TokenType::In });
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
if (match_for_in_of()) {
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
if (is_await_loop != IsForAwaitLoop::Yes
&& starts_with_async_of && match_of(m_state.current_token))
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
syntax_error("for-of loop may not start with async of");
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
return parse_for_in_of_statement(*init, is_await_loop);
LibJS: Parse async arrow functions
2021-11-14 23:47:16 +00:00
}
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
} else {
LibJS: Replace some parser assertions by syntax errors When parsing JavaScript, we can get pretty much any sequnce of tokens, and we shouldn't crash if it's not something that we normally expect. Instead, emit syntax errors.
2020-06-01 14:08:34 +00:00
syntax_error("Unexpected token in for loop");
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
}
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
}
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
consume(TokenType::Semicolon);
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 10:23:53 +00:00
RefPtr<Expression> test;
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
if (!match(TokenType::Semicolon))
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
test = parse_expression(0);
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
consume(TokenType::Semicolon);
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 10:23:53 +00:00
RefPtr<Expression> update;
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
if (!match(TokenType::ParenClose))
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
update = parse_expression(0);
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
consume(TokenType::ParenClose);
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
TemporaryChange break_change(m_state.in_break_context, true);
TemporaryChange continue_change(m_state.in_continue_context, true);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopePusher for_loop_scope = ScopePusher::for_loop_scope(*this, init);
LibJS: Allow "for" statement without curly braces around body
2020-04-04 19:09:06 +00:00
auto body = parse_statement();
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ForStatement>({ m_source_code, rule_start.position(), position() }, move(init), move(test), move(update), move(body));
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
}
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
NonnullRefPtr<Statement> Parser::parse_for_in_of_statement(NonnullRefPtr<ASTNode> lhs, IsForAwaitLoop is_for_await_loop)
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
{
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
Variant<NonnullRefPtr<ASTNode>, NonnullRefPtr<BindingPattern>> for_declaration = lhs;
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
auto has_annexB_for_in_init_extension = false;
LibJS: Remove hand-rolled type information in JS AST in favor of RTTI
2021-01-01 18:34:07 +00:00
if (is<VariableDeclaration>(*lhs)) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
auto& declaration = static_cast<VariableDeclaration&>(*lhs);
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
// Syntax errors for wrong amounts of declaration should have already been hit.
if (!declaration.declarations().is_empty()) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
// AnnexB extension B.3.5 Initializers in ForIn Statement Heads, https://tc39.es/ecma262/#sec-initializers-in-forin-statement-heads
auto& variable = declaration.declarations().first();
if (variable.init()) {
if (m_state.strict_mode || declaration.declaration_kind() != DeclarationKind::Var || !variable.target().has<NonnullRefPtr<Identifier>>())
syntax_error("Variable initializer not allowed in for..in/of");
else
has_annexB_for_in_init_extension = true;
}
}
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
} else if (!lhs->is_identifier() && !is<MemberExpression>(*lhs) && !is<CallExpression>(*lhs) && !is<UsingDeclaration>(*lhs)) {
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
bool valid = false;
if (is<ObjectExpression>(*lhs) || is<ArrayExpression>(*lhs)) {
auto synthesized_binding_pattern = synthesize_binding_pattern(static_cast<Expression const&>(*lhs));
if (synthesized_binding_pattern) {
for_declaration = synthesized_binding_pattern.release_nonnull();
valid = true;
}
}
if (!valid)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Invalid left-hand side in for-loop ('{}')", lhs->class_name()));
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
}
auto in_or_of = consume();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
auto is_in = in_or_of.type() == TokenType::In;
LibJS: Be more strict about the lhs of a for in/of loop This is not entirely correct as really Object- and ArrayExpressions are not allowed but that requires a bigger refactoring of for statement parsing.
2021-07-28 21:09:57 +00:00
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (!is_in) {
LibJS: Be more strict about the lhs of a for in/of loop This is not entirely correct as really Object- and ArrayExpressions are not allowed but that requires a bigger refactoring of for statement parsing.
2021-07-28 21:09:57 +00:00
if (is<MemberExpression>(*lhs)) {
auto& member = static_cast<MemberExpression const&>(*lhs);
if (member.object().is_identifier() && static_cast<Identifier const&>(member.object()).string() == "let"sv)
syntax_error("For of statement may not start with let.");
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (has_annexB_for_in_init_extension)
syntax_error("Variable initializer not allowed in for..of", rule_start.position());
LibJS: Be more strict about the lhs of a for in/of loop This is not entirely correct as really Object- and ArrayExpressions are not allowed but that requires a bigger refactoring of for statement parsing.
2021-07-28 21:09:57 +00:00
}
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
auto rhs = parse_expression(is_in ? 0 : 2);
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
consume(TokenType::ParenClose);
LibJS: Disallow 'continue' & 'break' outside of their respective scopes 'continue' is no longer allowed outside of a loop, and an unlabeled 'break' is not longer allowed outside of a loop or switch statement. Labeled 'break' statements are still allowed everywhere, even if the label does not exist.
2020-10-07 19:21:15 +00:00
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
TemporaryChange break_change(m_state.in_break_context, true);
TemporaryChange continue_change(m_state.in_continue_context, true);
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
ScopePusher for_loop_scope = ScopePusher::for_loop_scope(*this, lhs);
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
auto body = parse_statement();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 10:44:56 +00:00
if (is_in)
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ForInStatement>({ m_source_code, rule_start.position(), position() }, move(for_declaration), move(rhs), move(body));
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 15:09:28 +00:00
if (is_for_await_loop == IsForAwaitLoop::Yes)
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ForAwaitOfStatement>({ m_source_code, rule_start.position(), position() }, move(for_declaration), move(rhs), move(body));
return create_ast_node<ForOfStatement>({ m_source_code, rule_start.position(), position() }, move(for_declaration), move(rhs), move(body));
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
}
LibJS: Implement (no-op) debugger statement
2020-04-30 16:26:27 +00:00
NonnullRefPtr<DebuggerStatement> Parser::parse_debugger_statement()
{
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
auto rule_start = push_start();
LibJS: Implement (no-op) debugger statement
2020-04-30 16:26:27 +00:00
consume(TokenType::Debugger);
consume_or_insert_semicolon();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<DebuggerStatement>({ m_source_code, rule_start.position(), position() });
LibJS: Implement (no-op) debugger statement
2020-04-30 16:26:27 +00:00
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
bool Parser::match(TokenType type) const
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
return m_state.current_token.type() == type;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
bool Parser::match_expression() const
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto type = m_state.current_token.type();
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 22:45:10 +00:00
if (type == TokenType::Import) {
auto lookahead_token = next_token();
return lookahead_token.type() == TokenType::Period || lookahead_token.type() == TokenType::ParenOpen;
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
return type == TokenType::BoolLiteral
|| type == TokenType::NumericLiteral
LibJS: Add BigInt
2020-06-06 00:14:10 +00:00
|| type == TokenType::BigIntLiteral
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::StringLiteral
LibJS: Add template literals Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-03 22:41:14 +00:00
|| type == TokenType::TemplateLiteralStart
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::NullLiteral
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
|| match_identifier()
LibJS: Replace the verify in private identifier with a syntax error Since sometimes expressions are parsed without checking we can hit this expression without it being followed by an 'in'.
2021-11-26 23:16:01 +00:00
|| type == TokenType::PrivateIdentifier
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
|| type == TokenType::Await
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::New
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
|| type == TokenType::Class
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::CurlyOpen
|| type == TokenType::BracketOpen
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
|| type == TokenType::ParenOpen
LibJS: Parse FunctionExpressions FunctionExpression is mostly like FunctionDeclaration, except the name is optional. Share the parsing logic in parse_function_node<NodeType>. This allows us to do nice things like: document.addEventListener("DOMContentLoaded", function() { alert("Hello friends!"); });
2020-03-19 10:52:56 +00:00
|| type == TokenType::Function
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
|| type == TokenType::Async
LibJS: Parse "this" as ThisExpression
2020-04-12 22:42:14 +00:00
|| type == TokenType::This
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
|| type == TokenType::Super
LibJS: Lex and parse regex literals, add RegExp objects This adds regex parsing/lexing, as well as a relatively empty RegExpObject. The purpose of this patch is to allow the engine to not get hung up on parsing regexes. This will aid in finding new syntax errors (say, from google or twitter) without having to replace all of their regexes first!
2020-06-03 23:05:49 +00:00
|| type == TokenType::RegexLiteral
LibJS: Force the lexer to parse a regex when expecting a statement
2021-07-29 21:28:28 +00:00
|| type == TokenType::Slash // Wrongly recognized regex by lexer
|| type == TokenType::SlashEquals // Wrongly recognized regex by lexer (/=a/ is a valid regex)
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
|| type == TokenType::Yield
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
|| match_unary_prefixed_expression();
}
bool Parser::match_unary_prefixed_expression() const
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto type = m_state.current_token.type();
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 18:45:51 +00:00
return type == TokenType::PlusPlus
|| type == TokenType::MinusMinus
|| type == TokenType::ExclamationMark
LibJS: Implement typeof operator
2020-03-17 19:33:32 +00:00
|| type == TokenType::Tilde
LibJS: Implement unary plus / minus
2020-04-02 16:58:39 +00:00
|| type == TokenType::Plus
|| type == TokenType::Minus
LibJS: Implement void operator
2020-04-15 16:55:03 +00:00
|| type == TokenType::Typeof
LibJS: Implement basic support for the "delete" operator It turns out "delete" is actually a unary op :) This patch implements deletion of object properties, it doesn't yet work for casually deleting properties from the global object. When deleting a property from an object, we switch that object to having a unique shape, no longer sharing shapes with others. Once an object has a unique shape, it no longer needs to care about shape transitions.
2020-04-26 11:53:40 +00:00
|| type == TokenType::Void
|| type == TokenType::Delete;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
bool Parser::match_secondary_expression(ForbiddenTokens forbidden) const
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto type = m_state.current_token.type();
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
if (!forbidden.allows(type))
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
return false;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
return type == TokenType::Plus
LibJS: Implement +=, -=, *=, and /= assignment operators
2020-03-12 12:09:15 +00:00
|| type == TokenType::PlusEquals
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::Minus
LibJS: Implement +=, -=, *=, and /= assignment operators
2020-03-12 12:09:15 +00:00
|| type == TokenType::MinusEquals
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::Asterisk
LibJS: Implement +=, -=, *=, and /= assignment operators
2020-03-12 12:09:15 +00:00
|| type == TokenType::AsteriskEquals
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::Slash
LibJS: Implement +=, -=, *=, and /= assignment operators
2020-03-12 12:09:15 +00:00
|| type == TokenType::SlashEquals
LibJS: Add basic support for modulo (%) in binary expressions
2020-04-04 19:17:34 +00:00
|| type == TokenType::Percent
LibJS: Implement modulo assignment operator (%=)
2020-05-04 22:07:05 +00:00
|| type == TokenType::PercentEquals
LibJS: Implement exponentiation (** operator)
2020-04-05 12:40:00 +00:00
|| type == TokenType::DoubleAsterisk
LibJS: Implement exponentiation assignment operator (**=)
2020-05-04 22:03:35 +00:00
|| type == TokenType::DoubleAsteriskEquals
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::Equals
LibJS: Parse === and !== binary operators
2020-03-12 12:11:33 +00:00
|| type == TokenType::EqualsEqualsEquals
|| type == TokenType::ExclamationMarkEqualsEquals
LibJS: Implement abstract equality and inequality
2020-03-15 22:23:38 +00:00
|| type == TokenType::EqualsEquals
|| type == TokenType::ExclamationMarkEquals
LibJS: Parse > and < binary operators
2020-03-12 12:10:27 +00:00
|| type == TokenType::GreaterThan
LibJS: Implement <= and >= binary operators
2020-03-12 12:07:08 +00:00
|| type == TokenType::GreaterThanEquals
LibJS: Parse > and < binary operators
2020-03-12 12:10:27 +00:00
|| type == TokenType::LessThan
LibJS: Implement <= and >= binary operators
2020-03-12 12:07:08 +00:00
|| type == TokenType::LessThanEquals
LibJS: Implement basic MemberExpression parsing At last we can parse "hello friends".length :^)
2020-03-12 12:05:57 +00:00
|| type == TokenType::ParenOpen
LibJS: Implement update expressions Note that currently only the non-prefixed variant is supported (i.e i++ not ++i), this variant returns the value of the argument before the update.
2020-03-12 11:45:45 +00:00
|| type == TokenType::Period
LibJS: Parse computed MemberExpressions MemberExpression comes in two flavors: computed: a[b] non-computed: a.b We can now parse both of the types. :^)
2020-03-20 19:51:03 +00:00
|| type == TokenType::BracketOpen
LibJS: Handle '++' and '--' more correctly within expression
2021-08-14 15:02:15 +00:00
|| (type == TokenType::PlusPlus && !m_state.current_token.trivia_contains_line_terminator())
|| (type == TokenType::MinusMinus && !m_state.current_token.trivia_contains_line_terminator())
LibJS: Implement 'in' operator
2020-04-23 15:06:01 +00:00
|| type == TokenType::In
LibJS: Implement ConditionalExpression (ternary "?:" operator)
2020-04-03 10:14:28 +00:00
|| type == TokenType::Instanceof
LibJS: Parse binary bitwise operators The Lexer and AST already have all the functionality required in place, so this is just updating Parser::match_secondary_expression() and Parser::parse_expression() to handle TokenType::{Ampersand,Pipe,Caret}, as well as adding some tests.
2020-04-03 12:02:31 +00:00
|| type == TokenType::QuestionMark
|| type == TokenType::Ampersand
LibJS: Implement bitwise assignment operators (&=, |=, ^=)
2020-05-04 21:34:45 +00:00
|| type == TokenType::AmpersandEquals
LibJS: Parse binary bitwise operators The Lexer and AST already have all the functionality required in place, so this is just updating Parser::match_secondary_expression() and Parser::parse_expression() to handle TokenType::{Ampersand,Pipe,Caret}, as well as adding some tests.
2020-04-03 12:02:31 +00:00
|| type == TokenType::Pipe
LibJS: Implement bitwise assignment operators (&=, |=, ^=)
2020-05-04 21:34:45 +00:00
|| type == TokenType::PipeEquals
LibJS: Fix logical expressions The current implementation of logical AND (&&) and logical OR (||) always returns a boolean - this is not correct.
2020-04-03 13:33:28 +00:00
|| type == TokenType::Caret
LibJS: Implement bitwise assignment operators (&=, |=, ^=)
2020-05-04 21:34:45 +00:00
|| type == TokenType::CaretEquals
LibJS: Implement bitwise left shift operator (<<)
2020-04-23 12:36:14 +00:00
|| type == TokenType::ShiftLeft
|| type == TokenType::ShiftLeftEquals
LibJS: Implement bitwise right shift operator (>>)
2020-04-23 12:45:19 +00:00
|| type == TokenType::ShiftRight
|| type == TokenType::ShiftRightEquals
LibJS: Implement bitwise unsigned right shift operator (>>>)
2020-04-23 14:43:10 +00:00
|| type == TokenType::UnsignedShiftRight
|| type == TokenType::UnsignedShiftRightEquals
LibJS: Fix logical expressions The current implementation of logical AND (&&) and logical OR (||) always returns a boolean - this is not correct.
2020-04-03 13:33:28 +00:00
|| type == TokenType::DoubleAmpersand
LibJS: Implement logical assignment operators (&&=, ||=, ??=) TC39 proposal, stage 4 as of 2020-07. https://tc39.es/proposal-logical-assignment/
2020-10-05 15:49:43 +00:00
|| type == TokenType::DoubleAmpersandEquals
LibJS: Implement nullish coalescing operator (??)
2020-04-17 23:49:11 +00:00
|| type == TokenType::DoublePipe
LibJS: Implement logical assignment operators (&&=, ||=, ??=) TC39 proposal, stage 4 as of 2020-07. https://tc39.es/proposal-logical-assignment/
2020-10-05 15:49:43 +00:00
|| type == TokenType::DoublePipeEquals
|| type == TokenType::DoubleQuestionMark
LibJS: Implement parsing and execution of optional chains
2021-09-14 02:26:31 +00:00
|| type == TokenType::DoubleQuestionMarkEquals
|| type == TokenType::QuestionMarkPeriod;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
bool Parser::match_statement() const
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto type = m_state.current_token.type();
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
return match_expression()
|| type == TokenType::Return
LibJS: Implement generator functions (only in bytecode mode)
2021-06-10 21:08:30 +00:00
|| type == TokenType::Yield
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::Do
|| type == TokenType::If
LibJS: Implement "throw" You can now throw an expression to the nearest catcher! :^) To support throwing arbitrary values, I added an Exception class that sits as a wrapper around whatever is thrown. In the future it will be a logical place to store a call stack.
2020-03-24 21:03:50 +00:00
|| type == TokenType::Throw
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::Try
|| type == TokenType::While
LibJS: Parse "with" statements :^)
2020-11-28 14:05:57 +00:00
|| type == TokenType::With
LibJS: Implement for statement
2020-03-12 12:12:12 +00:00
|| type == TokenType::For
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
|| type == TokenType::CurlyOpen
LibJS: Lexer and parser support for "switch" statements
2020-03-29 11:09:54 +00:00
|| type == TokenType::Switch
|| type == TokenType::Break
LibJS: Add support for "continue" inside "for" statements :^)
2020-04-04 22:22:42 +00:00
|| type == TokenType::Continue
LibJS: Implement (no-op) debugger statement
2020-04-30 16:26:27 +00:00
|| type == TokenType::Var
LibJS: Support empty statements We already skipped random semicolons in Parser::parse_program(), but now they are properly matched and parsed as empty statements - and thus recognized as a valid body of an if / else / while / ... statement.
2020-05-03 09:59:00 +00:00
|| type == TokenType::Debugger
|| type == TokenType::Semicolon;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
bool Parser::match_export_or_import() const
{
auto type = m_state.current_token.type();
return type == TokenType::Export
|| type == TokenType::Import;
}
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
bool Parser::match_declaration(AllowUsingDeclaration allow_using) const
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto type = m_state.current_token.type();
LibJS: Check that 'let' is followed by declaration before matching it Since 'let' is a valid variable name (in non-strict mode) let may not be the start of a declaration but just an identifier.
2021-07-29 00:03:16 +00:00
if (type == TokenType::Let && !m_state.strict_mode) {
return try_match_let_declaration();
}
LibJS: Don't match async \n function as an async function declaration
2021-11-30 14:52:51 +00:00
if (type == TokenType::Async) {
auto lookahead_token = next_token();
return lookahead_token.type() == TokenType::Function && !lookahead_token.trivia_contains_line_terminator();
}
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
if (allow_using == AllowUsingDeclaration::Yes && type == TokenType::Identifier && m_state.current_token.original_value() == "using"sv)
return try_match_using_declaration();
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
return type == TokenType::Function
|| type == TokenType::Class
|| type == TokenType::Const
LibJS: Don't match async \n function as an async function declaration
2021-11-30 14:52:51 +00:00
|| type == TokenType::Let;
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
}
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
Token Parser::next_token(size_t steps) const
LibJS: Check that 'let' is followed by declaration before matching it Since 'let' is a valid variable name (in non-strict mode) let may not be the start of a declaration but just an identifier.
2021-07-29 00:03:16 +00:00
{
LibJS: Do not save state for peeking at the next token from the lexer This saves having to save and load the parser state. This could give an incorrect token in some cases where the parser communicates to the lexer. However this is not applicable in any of the current usages and this would require one to parse the current token as normal which is exactly what you don't want to do in that scenario.
2021-10-07 22:38:24 +00:00
Lexer lookahead_lexer = m_state.lexer;
LibJS: Check that 'let' is followed by declaration before matching it Since 'let' is a valid variable name (in non-strict mode) let may not be the start of a declaration but just an identifier.
2021-07-29 00:03:16 +00:00
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
Token lookahead_token;
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
while (steps > 0) {
lookahead_token = lookahead_lexer.next();
steps--;
}
return lookahead_token;
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
}
LibJS: Check that 'let' is followed by declaration before matching it Since 'let' is a valid variable name (in non-strict mode) let may not be the start of a declaration but just an identifier.
2021-07-29 00:03:16 +00:00
LibJS: Do not save state for peeking at the next token from the lexer This saves having to save and load the parser state. This could give an incorrect token in some cases where the parser communicates to the lexer. However this is not applicable in any of the current usages and this would require one to parse the current token as normal which is exactly what you don't want to do in that scenario.
2021-10-07 22:38:24 +00:00
bool Parser::try_match_let_declaration() const
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
{
VERIFY(m_state.current_token.type() == TokenType::Let);
auto token_after = next_token();
LibJS: Check that 'let' is followed by declaration before matching it Since 'let' is a valid variable name (in non-strict mode) let may not be the start of a declaration but just an identifier.
2021-07-29 00:03:16 +00:00
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
if (token_after.is_identifier_name() && token_after.value() != "in"sv)
LibJS: Check that 'let' is followed by declaration before matching it Since 'let' is a valid variable name (in non-strict mode) let may not be the start of a declaration but just an identifier.
2021-07-29 00:03:16 +00:00
return true;
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 15:04:37 +00:00
if (token_after.type() == TokenType::CurlyOpen || token_after.type() == TokenType::BracketOpen)
LibJS: Check that 'let' is followed by declaration before matching it Since 'let' is a valid variable name (in non-strict mode) let may not be the start of a declaration but just an identifier.
2021-07-29 00:03:16 +00:00
return true;
return false;
}
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 21:09:57 +00:00
bool Parser::try_match_using_declaration() const
{
VERIFY(m_state.current_token.type() == TokenType::Identifier);
VERIFY(m_state.current_token.original_value() == "using"sv);
auto token_after = next_token();
if (token_after.trivia_contains_line_terminator())
return false;
return token_after.is_identifier_name();
}
LibJS: Do not save state for peeking at the next token from the lexer This saves having to save and load the parser state. This could give an incorrect token in some cases where the parser communicates to the lexer. However this is not applicable in any of the current usages and this would require one to parse the current token as normal which is exactly what you don't want to do in that scenario.
2021-10-07 22:38:24 +00:00
bool Parser::match_variable_declaration() const
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto type = m_state.current_token.type();
LibJS: Check that 'let' is followed by declaration before matching it Since 'let' is a valid variable name (in non-strict mode) let may not be the start of a declaration but just an identifier.
2021-07-29 00:03:16 +00:00
if (type == TokenType::Let && !m_state.strict_mode) {
return try_match_let_declaration();
}
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 22:30:07 +00:00
return type == TokenType::Var
|| type == TokenType::Let
|| type == TokenType::Const;
}
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
bool Parser::match_identifier() const
{
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
return token_is_identifier(m_state.current_token);
}
bool Parser::token_is_identifier(Token const& token) const
{
if (token.type() == TokenType::EscapedKeyword) {
if (token.value() == "let"sv)
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
return !m_state.strict_mode;
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
if (token.value() == "yield"sv)
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
return !m_state.strict_mode && !m_state.in_generator_function_context;
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
if (token.value() == "await"sv)
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
return m_program_type != Program::Type::Module && !m_state.await_expression_is_valid && !m_state.in_class_static_init_block;
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
return true;
}
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 00:45:29 +00:00
return token.type() == TokenType::Identifier
|| token.type() == TokenType::Async
|| (token.type() == TokenType::Let && !m_state.strict_mode)
|| (token.type() == TokenType::Await && m_program_type != Program::Type::Module && !m_state.await_expression_is_valid && !m_state.in_class_static_init_block)
|| (token.type() == TokenType::Yield && !m_state.in_generator_function_context && !m_state.strict_mode); // See note in Parser::parse_identifier().
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
}
LibJS: Allow reserved words as keys in object expressions.
2020-04-18 18:31:27 +00:00
bool Parser::match_identifier_name() const
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
return m_state.current_token.is_identifier_name();
LibJS: Allow reserved words as keys in object expressions.
2020-04-18 18:31:27 +00:00
}
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
bool Parser::match_property_key() const
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto type = m_state.current_token.type();
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 18:31:21 +00:00
return match_identifier_name()
|| type == TokenType::BracketOpen
|| type == TokenType::StringLiteral
|| type == TokenType::NumericLiteral
|| type == TokenType::BigIntLiteral;
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
bool Parser::done() const
{
return match(TokenType::Eof);
}
Token Parser::consume()
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
auto old_token = m_state.current_token;
m_state.current_token = m_state.lexer.next();
LibJS: Add an optimization to avoid needless arguments object creation This gives FunctionNode a "might need arguments object" boolean flag and sets it based on the simplest possible heuristic for this: if we encounter an identifier called "arguments" or "eval" up to the next (nested) function declaration or expression, we won't need an arguments object. Otherwise, we *might* need one - the final decision is made in the FunctionDeclarationInstantiation AO. Now, this is obviously not perfect. Even if you avoid eval, something like `foo.arguments` will still trigger a false positive - but it's a start and already massively cuts down on needlessly allocated objects, especially in real-world code that is often minified, and so a full "arguments" identifier will be an actual arguments object more often than not. To illustrate the actual impact of this change, here's the number of allocated arguments objects during a full test-js run: Before: - Unmapped arguments objects: 78765 - Mapped arguments objects: 2455 After: - Unmapped arguments objects: 18 - Mapped arguments objects: 37 This results in a ~5% speedup of test-js on my Linux host machine, and about 3.5% on i686 Serenity in QEMU (warm runs, average of 5). The following microbenchmark (calling an empty function 1M times) runs 25% faster on Linux and 45% on Serenity: function foo() {} for (var i = 0; i < 1_000_000; ++i) foo(); test262 reports no changes in either direction, apart from a speedup :^)
2021-10-05 07:44:58 +00:00
// NOTE: This is the bare minimum needed to decide whether we might need an arguments object
// in a function expression or declaration. ("might" because the AST implements some further
// conditions from the spec that rule out the need for allocating one)
if (old_token.type() == TokenType::Identifier && old_token.value().is_one_of("arguments"sv, "eval"sv))
m_state.function_might_need_arguments_object = true;
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 22:02:41 +00:00
return old_token;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
void Parser::consume_or_insert_semicolon()
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
// Semicolon was found and will be consumed
if (match(TokenType::Semicolon)) {
consume();
return;
}
// Insert semicolon if...
Everywhere: Fix typos Mostly in comments, but sprintf() now prints "August" instead of "Auguest" so that's something.
2020-10-02 13:59:28 +00:00
// ...token is preceded by one or more newlines
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (m_state.current_token.trivia_contains_line_terminator())
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
return;
LibJS: Fix semicolon insertion
2020-04-17 13:27:51 +00:00
// ...token is a closing curly brace
if (match(TokenType::CurlyClose))
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
return;
// ...token is eof
if (match(TokenType::Eof))
return;
// No rule for semicolon insertion applies -> syntax error
expected("Semicolon");
}
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
Token Parser::consume_identifier()
{
if (match(TokenType::Identifier))
return consume(TokenType::Identifier);
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
if (match(TokenType::EscapedKeyword))
return consume(TokenType::EscapedKeyword);
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
// Note that 'let' is not a reserved keyword, but our lexer considers it such
// As it's pretty nice to have that (for syntax highlighting and such), we'll
// special-case it here instead.
if (match(TokenType::Let)) {
if (m_state.strict_mode)
syntax_error("'let' is not allowed as an identifier in strict mode");
return consume();
}
LibJS: Treat yield as an identifier in more non-generator contexts And disallow some cases where we are in a generator context.
2021-07-28 23:45:55 +00:00
if (match(TokenType::Yield)) {
if (m_state.strict_mode || m_state.in_generator_function_context)
syntax_error("Identifier must not be a reserved word in strict mode ('yield')");
return consume();
}
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (match(TokenType::Await)) {
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 22:50:32 +00:00
if (m_program_type == Program::Type::Module || m_state.await_expression_is_valid || m_state.in_class_static_init_block)
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
syntax_error("Identifier must not be a reserved word in modules ('await')");
return consume();
}
if (match(TokenType::Async))
return consume();
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
expected("Identifier");
return consume();
}
// https://tc39.es/ecma262/#prod-IdentifierReference
Token Parser::consume_identifier_reference()
{
if (match(TokenType::Identifier))
return consume(TokenType::Identifier);
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
if (match(TokenType::EscapedKeyword)) {
auto name = m_state.current_token.value();
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
if (m_state.strict_mode && (name == "let"sv || name == "yield"sv))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("'{}' is not allowed as an identifier in strict mode", name));
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
if (m_program_type == Program::Type::Module && name == "await"sv)
syntax_error("'await' is not allowed as an identifier in module");
LibJS: Fix some small remaining issues with parsing unicode escapes Added a test to ensure the behavior stays the same. We now throw on a direct usage of an escaped keywords with a specific error to make it more clear to the user.
2021-08-21 09:27:20 +00:00
return consume();
}
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
// See note in Parser::parse_identifier().
if (match(TokenType::Let)) {
if (m_state.strict_mode)
syntax_error("'let' is not allowed as an identifier in strict mode");
return consume();
}
if (match(TokenType::Yield)) {
if (m_state.strict_mode)
syntax_error("Identifier reference may not be 'yield' in strict mode");
return consume();
}
if (match(TokenType::Await)) {
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
if (m_program_type == Program::Type::Module)
syntax_error("'await' is not allowed as an identifier in module");
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
return consume();
}
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 18:39:22 +00:00
if (match(TokenType::Async))
return consume();
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
expected(Token::name(TokenType::Identifier));
return consume();
}
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
Token Parser::consume(TokenType expected_type)
{
LibJS: Implement basic for..in and for..of loops
2020-04-21 18:21:26 +00:00
if (!match(expected_type)) {
LibJS: Implement automatic semicolon insertion
2020-04-17 13:05:58 +00:00
expected(Token::name(expected_type));
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Throw a syntax error when an identifier is a reserved word From the specification: It is a Syntax Error if StringValue of IdentifierName is the same String value as the StringValue of any ReservedWord except for yield or await. It is a Syntax Error if this phrase is contained in strict mode code and the StringValue of IdentifierName is: "implements", "interface", "let", "package", "private", "protected", "public", "static", or "yield".
2021-06-16 22:57:01 +00:00
auto token = consume();
if (expected_type == TokenType::Identifier) {
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
if (m_state.strict_mode && is_strict_reserved_word(token.value()))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Identifier must not be a reserved word in strict mode ('{}')", token.value()));
LibJS: Throw a syntax error when an identifier is a reserved word From the specification: It is a Syntax Error if StringValue of IdentifierName is the same String value as the StringValue of any ReservedWord except for yield or await. It is a Syntax Error if this phrase is contained in strict mode code and the StringValue of IdentifierName is: "implements", "interface", "let", "package", "private", "protected", "public", "static", or "yield".
2021-06-16 22:57:01 +00:00
}
return token;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Unprefixed octal numbers are a syntax error in strict mode
2020-10-19 17:01:28 +00:00
Token Parser::consume_and_validate_numeric_literal()
{
Everywhere: Pass AK::StringView by value
2021-11-10 23:55:02 +00:00
auto is_unprefixed_octal_number = [](StringView value) {
LibJS: Use the new is_ascii_foo() helpers from AK These constexpr helpers generate nicer code than the LibC ctype.h variants, so let's make use of them. :^)
2021-06-13 08:47:09 +00:00
return value.length() > 1 && value[0] == '0' && is_ascii_digit(value[1]);
LibJS: Unprefixed octal numbers are a syntax error in strict mode
2020-10-19 17:01:28 +00:00
};
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
auto literal_start = position();
LibJS: Unprefixed octal numbers are a syntax error in strict mode
2020-10-19 17:01:28 +00:00
auto token = consume(TokenType::NumericLiteral);
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (m_state.strict_mode && is_unprefixed_octal_number(token.value()))
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
syntax_error("Unprefixed octal number not allowed in strict mode", literal_start);
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
if (match_identifier_name() && m_state.current_token.trivia().is_empty())
LibJS: Disallow NumericLiteral immediately followed by Identifier From the spec: https://tc39.es/ecma262/#sec-literals-numeric-literals The SourceCharacter immediately following a NumericLiteral must not be an IdentifierStart or DecimalDigit. For example: 3in is an error and not the two input elements 3 and in.
2020-10-22 21:18:31 +00:00
syntax_error("Numeric literal must not be immediately followed by identifier");
LibJS: Unprefixed octal numbers are a syntax error in strict mode
2020-10-19 17:01:28 +00:00
return token;
}
Everywhere: Run clang-format
2022-04-01 17:58:27 +00:00
void Parser::expected(char const* what)
LibJS: Add Parser::syntax_error() helper Instead of having fprintf()s all over the place we can now use syntax_error("message") or syntax_error("message", line, column). This takes care of a consistent format, appending a newline and getting the line number and column of the current token if the last two params are omitted.
2020-04-29 20:37:51 +00:00
{
LibJS: Remove DeprecatedString usage from Token
2023-01-26 13:38:09 +00:00
auto message = m_state.current_token.message().to_deprecated_string();
LibJS: Use message from invalid token in syntax error
2020-10-26 20:19:36 +00:00
if (message.is_empty())
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
message = DeprecatedString::formatted("Unexpected token {}. Expected {}", m_state.current_token.name(), what);
LibJS: Use message from invalid token in syntax error
2020-10-26 20:19:36 +00:00
syntax_error(message);
LibJS: Add Parser::syntax_error() helper Instead of having fprintf()s all over the place we can now use syntax_error("message") or syntax_error("message", line, column). This takes care of a consistent format, appending a newline and getting the line number and column of the current token if the last two params are omitted.
2020-04-29 20:37:51 +00:00
}
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 17:15:22 +00:00
Position Parser::position() const
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
{
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
return {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
m_state.current_token.line_number(),
LibJS: Implement parsing and evaluation for AssignmentPatterns e.g. `[...foo] = bar` can now be evaluated :^)
2021-07-10 20:46:17 +00:00
m_state.current_token.line_column(),
m_state.current_token.offset(),
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
};
}
Everywhere: Run clang-format
2022-04-01 17:58:27 +00:00
bool Parser::try_parse_arrow_function_expression_failed_at_position(Position const& position) const
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 20:41:51 +00:00
{
auto it = m_token_memoizations.find(position);
if (it == m_token_memoizations.end())
return false;
return (*it).value.try_parse_arrow_function_expression_failed;
}
Everywhere: Run clang-format
2022-04-01 17:58:27 +00:00
void Parser::set_try_parse_arrow_function_expression_failed_at_position(Position const& position, bool failed)
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 20:41:51 +00:00
{
m_token_memoizations.set(position, { failed });
}
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
void Parser::syntax_error(DeprecatedString const& message, Optional<Position> position)
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
{
if (!position.has_value())
position = this->position();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
m_state.errors.append({ message, position });
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 13:24:43 +00:00
void Parser::save_state()
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
m_saved_state.append(m_state);
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 13:24:43 +00:00
}
void Parser::load_state()
{
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 19:42:32 +00:00
VERIFY(!m_saved_state.is_empty());
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 12:43:09 +00:00
m_state = m_saved_state.take_last();
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 13:24:43 +00:00
}
LibJS: Add Parser::syntax_error() helper Instead of having fprintf()s all over the place we can now use syntax_error("message") or syntax_error("message", line, column). This takes care of a consistent format, appending a newline and getting the line number and column of the current token if the last two params are omitted.
2020-04-29 20:37:51 +00:00
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 13:17:39 +00:00
void Parser::discard_saved_state()
{
m_saved_state.take_last();
}
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
void Parser::check_identifier_name_for_assignment_validity(DeprecatedFlyString const& name, bool force_strict)
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
{
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
// FIXME: this is now called from multiple places maybe the error message should be dynamic?
Userland: Make use of container version of any_of Problem: - New `any_of` implementation takes the entire container so the user does not need to pass explicit begin/end iterators. This is unused except is in tests. Solution: - Make use of the new and more user-friendly version where possible.
2021-07-25 20:21:27 +00:00
if (any_of(s_reserved_words, [&](auto& value) { return name == value; })) {
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
syntax_error("Binding pattern target may not be a reserved word");
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
} else if (m_state.strict_mode || force_strict) {
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
if (name.is_one_of("arguments"sv, "eval"sv))
syntax_error("Binding pattern target may not be called 'arguments' or 'eval' in strict mode");
LibJS: Be more strict about reserved and special identifiers
2021-07-11 23:27:35 +00:00
else if (is_strict_reserved_word(name))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Binding pattern target may not be called '{}' in strict mode", name));
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 11:04:55 +00:00
}
}
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
bool Parser::match_assert_clause() const
{
return !m_state.current_token.trivia_contains_line_terminator() && m_state.current_token.original_value() == "assert"sv;
}
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString Parser::consume_string_value()
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
{
VERIFY(match(TokenType::StringLiteral));
auto string_token = consume();
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString value = parse_string_literal(string_token)->value();
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// This also checks IsStringWellFormedUnicode which makes sure there is no unpaired surrogate
// Surrogates are at least 3 bytes
if (value.length() < 3)
return value;
Utf8View view { value.view().substring_view(value.length() - 3) };
VERIFY(view.length() <= 3);
auto codepoint = *view.begin();
if (Utf16View::is_high_surrogate(codepoint)) {
syntax_error("StringValue ending with unpaired high surrogate");
VERIFY(view.length() == 1);
}
return value;
}
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
// AssertClause, https://tc39.es/proposal-import-assertions/#prod-AssertClause
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
ModuleRequest Parser::parse_module_request()
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
{
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// Does not include the 'from' since that is not always required.
if (!match(TokenType::StringLiteral)) {
expected("ModuleSpecifier (string)");
return ModuleRequest { "!!invalid!!" };
}
ModuleRequest request { consume_string_value() };
if (!match_assert_clause())
return request;
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
VERIFY(m_state.current_token.original_value() == "assert"sv);
consume(TokenType::Identifier);
consume(TokenType::CurlyOpen);
while (!done() && !match(TokenType::CurlyClose)) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
DeprecatedString key;
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
if (match(TokenType::StringLiteral)) {
Everywhere: Rename to_{string => deprecated_string}() where applicable This will make it easier to support both string types at the same time while we convert code, and tracking down remaining uses. One big exception is Value::to_string() in LibJS, where the name is dictated by the ToString AO.
2022-12-06 01:12:49 +00:00
key = parse_string_literal(m_state.current_token)->value().to_deprecated_string();
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
consume();
} else if (match_identifier_name()) {
key = consume().value();
} else {
expected("IdentifierName or StringValue as AssertionKey");
consume();
}
consume(TokenType::Colon);
if (match(TokenType::StringLiteral)) {
for (auto& entries : request.assertions) {
if (entries.key == key)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Duplicate assertion clauses with name: {}", key));
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
}
Everywhere: Rename to_{string => deprecated_string}() where applicable This will make it easier to support both string types at the same time while we convert code, and tracking down remaining uses. One big exception is Value::to_string() in LibJS, where the name is dictated by the ToString AO.
2022-12-06 01:12:49 +00:00
request.add_assertion(move(key), parse_string_literal(m_state.current_token)->value().to_deprecated_string());
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
}
consume(TokenType::StringLiteral);
if (match(TokenType::Comma))
consume(TokenType::Comma);
else
break;
}
consume(TokenType::CurlyClose);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
return request;
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
}
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
static DeprecatedFlyString default_string_value = "default";
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
NonnullRefPtr<ImportStatement> Parser::parse_import_statement(Program& program)
{
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
// We use the extended syntax which adds:
// ImportDeclaration:
// import ImportClause FromClause [no LineTerminator here] AssertClause;
// import ModuleSpecifier [no LineTerminator here] AssertClause;
// From: https://tc39.es/proposal-import-assertions/#prod-ImportDeclaration
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
auto rule_start = push_start();
if (program.type() != Program::Type::Module)
syntax_error("Cannot use import statement outside a module");
consume(TokenType::Import);
if (match(TokenType::StringLiteral)) {
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// import ModuleSpecifier ;
auto module_request = parse_module_request();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ImportStatement>({ m_source_code, rule_start.position(), position() }, move(module_request));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
auto match_imported_binding = [&] {
return match_identifier() || match(TokenType::Yield) || match(TokenType::Await);
};
auto match_as = [&] {
LibJS: Allow Unicode escape sequences in identifiers For example, "property.br\u{64}wn" should resolve to "property.brown". To support this behavior, this commit changes the Token class to hold both the evaluated identifier name and a view into the original source for the unevaluated name. There are some contexts in which identifiers are not allowed to contain Unicode escape sequences; for example, export statements of the form "export {} from foo.js" forbid escapes in the identifier "from". The test file is added to .prettierignore because prettier will replace all escaped Unicode sequences with their unescaped value.
2021-08-18 20:34:25 +00:00
return match(TokenType::Identifier) && m_state.current_token.original_value() == "as"sv;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
};
bool continue_parsing = true;
struct ImportWithLocation {
LibJS: Move {Import,Export}Entry out of {Import,Export}Statement By making these be standalone instead of nested structs, we can forward declare them. This will allow us to stop including AST.h in some places.
2022-11-23 11:16:51 +00:00
ImportEntry entry;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
Position position;
};
Vector<ImportWithLocation> entries_with_location;
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// import ImportClause FromClause ;
// ImportClause :
// ImportedDefaultBinding
// NameSpaceImport
// NamedImports
// ImportedDefaultBinding , NameSpaceImport
// ImportedDefaultBinding , NamedImports
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
if (match_imported_binding()) {
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// ImportedDefaultBinding : ImportedBinding
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
auto id_position = position();
auto bound_name = consume().value();
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
entries_with_location.append({ { default_string_value, bound_name }, id_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
if (match(TokenType::Comma)) {
consume(TokenType::Comma);
} else {
continue_parsing = false;
}
}
if (!continue_parsing) {
// skip the rest
} else if (match(TokenType::Asterisk)) {
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// NameSpaceImport : * as ImportedBinding
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
consume(TokenType::Asterisk);
if (!match_as())
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Unexpected token: {}", m_state.current_token.name()));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
consume(TokenType::Identifier);
if (match_imported_binding()) {
auto namespace_position = position();
auto namespace_name = consume().value();
LibJS: Move {Import,Export}Entry out of {Import,Export}Statement By making these be standalone instead of nested structs, we can forward declare them. This will allow us to stop including AST.h in some places.
2022-11-23 11:16:51 +00:00
entries_with_location.append({ ImportEntry({}, namespace_name, true), namespace_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Unexpected token: {}", m_state.current_token.name()));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
} else if (match(TokenType::CurlyOpen)) {
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// NamedImports :
// { ImportSpecifier ,_opt } (repeated any amount of times)
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
consume(TokenType::CurlyOpen);
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
while (!done() && !match(TokenType::CurlyClose)) {
if (match_identifier_name()) {
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// ImportSpecifier : ImportedBinding
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
auto require_as = !match_imported_binding();
auto name_position = position();
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
auto name = consume().DeprecatedFlyString_value();
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
if (match_as()) {
consume(TokenType::Identifier);
auto alias_position = position();
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
auto alias = consume_identifier().DeprecatedFlyString_value();
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
check_identifier_name_for_assignment_validity(alias);
entries_with_location.append({ { name, alias }, alias_position });
} else if (require_as) {
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Unexpected reserved word '{}'", name));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else {
check_identifier_name_for_assignment_validity(name);
entries_with_location.append({ { name, name }, name_position });
}
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
} else if (match(TokenType::StringLiteral)) {
// ImportSpecifier : ModuleExportName as ImportedBinding
auto name = consume_string_value();
if (!match_as())
expected("as");
consume(TokenType::Identifier);
auto alias_position = position();
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
auto alias = consume_identifier().DeprecatedFlyString_value();
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
check_identifier_name_for_assignment_validity(alias);
entries_with_location.append({ { move(name), alias }, alias_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else {
expected("identifier");
break;
}
if (!match(TokenType::Comma))
break;
consume(TokenType::Comma);
}
consume(TokenType::CurlyClose);
} else {
expected("import clauses");
}
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 21:02:50 +00:00
auto from_statement = consume(TokenType::Identifier).original_value();
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
if (from_statement != "from"sv)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Expected 'from' got {}", from_statement));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
auto module_request = parse_module_request();
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
LibJS: Move {Import,Export}Entry out of {Import,Export}Statement By making these be standalone instead of nested structs, we can forward declare them. This will allow us to stop including AST.h in some places.
2022-11-23 11:16:51 +00:00
Vector<ImportEntry> entries;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
entries.ensure_capacity(entries_with_location.size());
for (auto& entry : entries_with_location) {
for (auto& import_statement : program.imports()) {
if (import_statement.has_bound_name(entry.entry.local_name))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Identifier '{}' already declared", entry.entry.local_name), entry.position);
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
for (auto& new_entry : entries) {
if (new_entry.local_name == entry.entry.local_name)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Identifier '{}' already declared", entry.entry.local_name), entry.position);
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
entries.append(move(entry.entry));
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ImportStatement>({ m_source_code, rule_start.position(), position() }, move(module_request), move(entries));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
NonnullRefPtr<ExportStatement> Parser::parse_export_statement(Program& program)
{
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 14:29:25 +00:00
// We use the extended syntax which adds:
// ExportDeclaration:
// export ExportFromClause FromClause [no LineTerminator here] AssertClause ;
// From: https://tc39.es/proposal-import-assertions/#prod-ExportDeclaration
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
auto rule_start = push_start();
if (program.type() != Program::Type::Module)
syntax_error("Cannot use export statement outside a module");
auto match_as = [&] {
LibJS: Allow Unicode escape sequences in identifiers For example, "property.br\u{64}wn" should resolve to "property.brown". To support this behavior, this commit changes the Token class to hold both the evaluated identifier name and a view into the original source for the unevaluated name. There are some contexts in which identifiers are not allowed to contain Unicode escape sequences; for example, export statements of the form "export {} from foo.js" forbid escapes in the identifier "from". The test file is added to .prettierignore because prettier will replace all escaped Unicode sequences with their unescaped value.
2021-08-18 20:34:25 +00:00
return match(TokenType::Identifier) && m_state.current_token.original_value() == "as"sv;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
};
auto match_from = [&] {
LibJS: Allow Unicode escape sequences in identifiers For example, "property.br\u{64}wn" should resolve to "property.brown". To support this behavior, this commit changes the Token class to hold both the evaluated identifier name and a view into the original source for the unevaluated name. There are some contexts in which identifiers are not allowed to contain Unicode escape sequences; for example, export statements of the form "export {} from foo.js" forbid escapes in the identifier "from". The test file is added to .prettierignore because prettier will replace all escaped Unicode sequences with their unescaped value.
2021-08-18 20:34:25 +00:00
return match(TokenType::Identifier) && m_state.current_token.original_value() == "from"sv;
};
auto match_default = [&] {
return match(TokenType::Default) && m_state.current_token.original_value() == "default"sv;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
};
consume(TokenType::Export);
struct EntryAndLocation {
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
ExportEntry entry;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
Position position;
};
Vector<EntryAndLocation> entries_with_location;
RefPtr<ASTNode> expression = {};
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
bool is_default = false;
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
ModuleRequest from_specifier;
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
LibJS: Allow Unicode escape sequences in identifiers For example, "property.br\u{64}wn" should resolve to "property.brown". To support this behavior, this commit changes the Token class to hold both the evaluated identifier name and a view into the original source for the unevaluated name. There are some contexts in which identifiers are not allowed to contain Unicode escape sequences; for example, export statements of the form "export {} from foo.js" forbid escapes in the identifier "from". The test file is added to .prettierignore because prettier will replace all escaped Unicode sequences with their unescaped value.
2021-08-18 20:34:25 +00:00
if (match_default()) {
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
is_default = true;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
auto default_position = position();
consume(TokenType::Default);
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
DeprecatedFlyString local_name;
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
auto lookahead_token = next_token();
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
enum class MatchesFunctionDeclaration {
Yes,
No,
WithoutName,
};
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// Note: For some reason the spec here has declaration which can have no name
// and the rest of the parser is just not setup for that. With these
// hacks below we get through most things but we should probably figure
// out a better solution. I have attempted to explain why/what these "hacks" do below.
// The summary is treat named declarations just as declarations and hack around unnamed
// declarations with expression see also SourceTextModule::initialize_environment.
// As far as I'm aware the only problem (which is a tricky one) is:
// `export default function() {}()`
// Since we parse this as an expression you are immediately allowed to call it
// which is incorrect and this should give a SyntaxError.
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
auto has_name = [&](Token const& token) {
if (token.type() != TokenType::ParenOpen)
return MatchesFunctionDeclaration::Yes;
return MatchesFunctionDeclaration::WithoutName;
};
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
auto match_function_declaration = [&] {
// Hack part 1.
// Match a function declaration with a name, since we have async and generator
// and asyncgenerator variants this is quite complicated.
auto current_type = m_state.current_token.type();
Lexer lookahead_lexer = m_state.lexer;
lookahead_lexer.next();
if (current_type == TokenType::Function) {
if (lookahead_token.type() == TokenType::Asterisk)
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
return has_name(lookahead_lexer.next()); // function * [name]
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
else
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
return has_name(lookahead_token); // function [name]
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
}
if (current_type == TokenType::Async) {
if (lookahead_token.type() != TokenType::Function)
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
return MatchesFunctionDeclaration::No;
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
if (lookahead_token.trivia_contains_line_terminator())
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
return MatchesFunctionDeclaration::No;
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
auto lookahead_two_token = lookahead_lexer.next();
if (lookahead_two_token.type() == TokenType::Asterisk)
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
return has_name(lookahead_lexer.next()); // async function * [name]
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
else
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
return has_name(lookahead_two_token); // async function [name]
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
}
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
return MatchesFunctionDeclaration::No;
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
};
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
if (auto matches_function = match_function_declaration(); matches_function != MatchesFunctionDeclaration::No) {
auto function_declaration = parse_function_node<FunctionDeclaration>(
(matches_function == MatchesFunctionDeclaration::WithoutName ? FunctionNodeParseOptions::HasDefaultExportName : 0)
| FunctionNodeParseOptions::CheckForFunctionAndName);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
m_state.current_scope_pusher->add_declaration(function_declaration);
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
if (matches_function == MatchesFunctionDeclaration::WithoutName)
local_name = ExportStatement::local_name_for_default;
else
local_name = function_declaration->name();
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
expression = move(function_declaration);
} else if (match(TokenType::Class) && lookahead_token.type() != TokenType::CurlyOpen && lookahead_token.type() != TokenType::Extends) {
// Hack part 2.
// Attempt to detect classes with names only as those are declarations,
// this actually seems to cover all cases already.
auto class_expression = parse_class_declaration();
m_state.current_scope_pusher->add_declaration(class_expression);
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
local_name = class_expression->name();
expression = move(class_expression);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else if (match_expression()) {
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
// Hack part 3.
// Even though the unnamed declarations look like expression we should
// not treat them as such and thus not consume a semicolon after them.
bool special_case_declaration_without_name = match(TokenType::Class) || match(TokenType::Function) || (match(TokenType::Async) && lookahead_token.type() == TokenType::Function && !lookahead_token.trivia_contains_line_terminator());
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
expression = parse_expression(2);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
if (!special_case_declaration_without_name)
consume_or_insert_semicolon();
if (is<ClassExpression>(*expression)) {
auto const& class_expression = static_cast<ClassExpression&>(*expression);
if (class_expression.has_name())
local_name = class_expression.name();
}
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else {
expected("Declaration or assignment expression");
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
local_name = "!!invalid!!";
}
if (local_name.is_null()) {
local_name = ExportStatement::local_name_for_default;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(default_string_value, move(local_name)), default_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else {
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
enum class FromSpecifier {
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
NotAllowed,
Optional,
Required
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
} check_for_from { FromSpecifier::NotAllowed };
AK+Everywhere: Rename FlyString to DeprecatedFlyString DeprecatedFlyString relies heavily on DeprecatedString's StringImpl, so let's rename it to A) match the name of DeprecatedString, B) write a new FlyString class that is tied to String.
2023-01-09 00:23:00 +00:00
auto parse_module_export_name = [&](bool lhs) -> DeprecatedFlyString {
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
// https://tc39.es/ecma262/#prod-ModuleExportName
// ModuleExportName :
// IdentifierName
// StringLiteral
if (match_identifier_name()) {
return consume().value();
}
if (match(TokenType::StringLiteral)) {
// It is a Syntax Error if ReferencedBindings of NamedExports contains any StringLiterals.
// Only for export { "a" as "b" }; // <-- no from
if (lhs)
check_for_from = FromSpecifier::Required;
return consume_string_value();
}
expected("ExportSpecifier (string or identifier)");
return {};
};
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
if (match(TokenType::Asterisk)) {
auto asterisk_position = position();
consume(TokenType::Asterisk);
if (match_as()) {
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
// * as ModuleExportName
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
consume(TokenType::Identifier);
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
auto namespace_position = position();
auto exported_name = parse_module_export_name(false);
entries_with_location.append({ ExportEntry::all_module_request(exported_name), namespace_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else {
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::all_but_default_entry(), asterisk_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
check_for_from = FromSpecifier::Required;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else if (match_declaration()) {
auto decl_position = position();
auto declaration = parse_declaration();
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-26 23:01:23 +00:00
m_state.current_scope_pusher->add_declaration(declaration);
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
if (is<FunctionDeclaration>(*declaration)) {
auto& func = static_cast<FunctionDeclaration&>(*declaration);
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(func.name(), func.name()), func.source_range().start });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else if (is<ClassDeclaration>(*declaration)) {
auto& class_declaration = static_cast<ClassDeclaration&>(*declaration);
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(class_declaration.name(), class_declaration.name()), class_declaration.source_range().start });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else {
VERIFY(is<VariableDeclaration>(*declaration));
auto& variables = static_cast<VariableDeclaration&>(*declaration);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
VERIFY(variables.is_lexical_declaration());
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
for (auto& decl : variables.declarations()) {
decl.target().visit(
[&](NonnullRefPtr<Identifier> const& identifier) {
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(identifier->string(), identifier->string()), identifier->source_range().start });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
},
[&](NonnullRefPtr<BindingPattern> const& binding) {
binding->for_each_bound_name([&](auto& name) {
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(name, name), decl_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
});
});
}
}
expression = declaration;
} else if (match(TokenType::Var)) {
auto variable_position = position();
auto variable_declaration = parse_variable_declaration();
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
m_state.current_scope_pusher->add_declaration(variable_declaration);
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
for (auto& decl : variable_declaration->declarations()) {
decl.target().visit(
[&](NonnullRefPtr<Identifier> const& identifier) {
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(identifier->string(), identifier->string()), identifier->source_range().start });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
},
[&](NonnullRefPtr<BindingPattern> const& binding) {
binding->for_each_bound_name([&](auto& name) {
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(name, name), variable_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
});
});
}
expression = variable_declaration;
} else if (match(TokenType::CurlyOpen)) {
consume(TokenType::CurlyOpen);
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
check_for_from = FromSpecifier::Optional;
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
LibJS: Handle empty named export This is an export which looks like `export {} from "module"`, and although it doesn't have any real export entries it should still add "module" to the required modules to load.
2022-08-29 20:12:25 +00:00
// FIXME: Even when empty should add module to requiredModules!
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
while (!done() && !match(TokenType::CurlyClose)) {
auto identifier_position = position();
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
auto identifier = parse_module_export_name(true);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
if (match_as()) {
consume(TokenType::Identifier);
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
auto export_name = parse_module_export_name(false);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(move(export_name), move(identifier)), identifier_position });
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
} else {
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
entries_with_location.append({ ExportEntry::named_export(identifier, identifier), identifier_position });
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
if (!match(TokenType::Comma))
break;
consume(TokenType::Comma);
}
LibJS: Handle empty named export This is an export which looks like `export {} from "module"`, and although it doesn't have any real export entries it should still add "module" to the required modules to load.
2022-08-29 20:12:25 +00:00
if (entries_with_location.is_empty()) {
// export {} from "module"; Since this will never be a
// duplicate we can give a slightly wrong location.
entries_with_location.append({ ExportEntry::empty_named_export(), position() });
}
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
consume(TokenType::CurlyClose);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 22:51:28 +00:00
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
} else {
syntax_error("Unexpected token 'export'", rule_start.position());
}
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
if (check_for_from != FromSpecifier::NotAllowed && match_from()) {
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
consume(TokenType::Identifier);
LibJS: Follow the spec with storing im- and export entries Because we can have arbitrary in- and export names with strings we can have '*' and '' which means using '*' as an indicating namespace imports failed / behaved incorrectly for string imports '*'. We now use more specific types to indicate these special states instead of these 'magic' string values. Do note that 'default' is not actually a magic string value but one specified by the spec. And you can in fact export the default value by doing: `export { 1 as default }`.
2022-01-27 00:54:47 +00:00
from_specifier = parse_module_request();
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
} else if (check_for_from == FromSpecifier::Required) {
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
expected("from");
}
LibJS: Allow full ModuleExportName in namespace This means we should accept a string after 'export * as '.
2022-08-27 23:40:51 +00:00
if (check_for_from != FromSpecifier::NotAllowed)
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
consume_or_insert_semicolon();
}
LibJS: Move {Import,Export}Entry out of {Import,Export}Statement By making these be standalone instead of nested structs, we can forward declare them. This will allow us to stop including AST.h in some places.
2022-11-23 11:16:51 +00:00
Vector<ExportEntry> entries;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
entries.ensure_capacity(entries_with_location.size());
for (auto& entry : entries_with_location) {
for (auto& export_statement : program.exports()) {
if (export_statement.has_export(entry.entry.export_name))
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Duplicate export with name: '{}'", entry.entry.export_name), entry.position);
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
for (auto& new_entry : entries) {
LibJS: Move {Import,Export}Entry out of {Import,Export}Statement By making these be standalone instead of nested structs, we can forward declare them. This will allow us to stop including AST.h in some places.
2022-11-23 11:16:51 +00:00
if (new_entry.kind != ExportEntry::Kind::EmptyNamedExport && new_entry.export_name == entry.entry.export_name)
AK+Everywhere: Rename String to DeprecatedString We have a new, improved string type coming up in AK (OOM aware, no null state), and while it's going to use UTF-8, the name UTF8String is a mouthful - so let's free up the String name by renaming the existing class. Making the old one have an annoying name will hopefully also help with quick adoption :^)
2022-12-04 18:02:33 +00:00
syntax_error(DeprecatedString::formatted("Duplicate export with name: '{}'", entry.entry.export_name), entry.position);
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
entries.append(move(entry.entry));
}
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 16:37:38 +00:00
return create_ast_node<ExportStatement>({ m_source_code, rule_start.position(), position() }, move(expression), move(entries), is_default, move(from_specifier));
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 15:42:30 +00:00
}
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-16 06:34:59 +00:00
Parser::ForbiddenTokens::ForbiddenTokens(std::initializer_list<TokenType> const& forbidden)
{
forbid_tokens(forbidden);
}
void Parser::ForbiddenTokens::forbid_tokens(std::initializer_list<TokenType> const& forbidden)
{
for (auto token : forbidden) {
switch (token) {
case TokenType::In:
m_forbid_in_token = true;
break;
case TokenType::DoubleAmpersand:
case TokenType::DoublePipe:
m_forbid_logical_tokens = true;
break;
case TokenType::DoubleQuestionMark:
m_forbid_coalesce_token = true;
break;
case TokenType::QuestionMarkPeriod:
m_forbid_question_mark_period = true;
break;
case TokenType::ParenOpen:
m_forbid_paren_open = true;
break;
case TokenType::Equals:
m_forbid_equals = true;
break;
default:
VERIFY_NOT_REACHED();
}
}
}
bool Parser::ForbiddenTokens::allows(TokenType token) const
{
switch (token) {
case TokenType::In:
return !m_forbid_in_token;
case TokenType::DoubleAmpersand:
case TokenType::DoublePipe:
return !m_forbid_logical_tokens;
case TokenType::DoubleQuestionMark:
return !m_forbid_coalesce_token;
case TokenType::QuestionMarkPeriod:
return !m_forbid_question_mark_period;
case TokenType::ParenOpen:
return !m_forbid_paren_open;
case TokenType::Equals:
return !m_forbid_equals;
default:
return true;
}
}
Parser::ForbiddenTokens Parser::ForbiddenTokens::merge(ForbiddenTokens other) const
{
ForbiddenTokens result = *this;
result.m_forbid_in_token |= other.m_forbid_in_token;
result.m_forbid_logical_tokens |= other.m_forbid_logical_tokens;
result.m_forbid_coalesce_token |= other.m_forbid_coalesce_token;
result.m_forbid_paren_open |= other.m_forbid_paren_open;
result.m_forbid_question_mark_period |= other.m_forbid_question_mark_period;
result.m_forbid_equals |= other.m_forbid_equals;
return result;
}
Parser::ForbiddenTokens Parser::ForbiddenTokens::forbid(std::initializer_list<TokenType> const& forbidden) const
{
ForbiddenTokens result = *this;
result.forbid_tokens(forbidden);
return result;
}
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-01 22:46:37 +00:00
template NonnullRefPtr<FunctionExpression> Parser::parse_function_node(u16, Optional<Position> const&);
template NonnullRefPtr<FunctionDeclaration> Parser::parse_function_node(u16, Optional<Position> const&);
LibJS: Explicitly instantiate Parser::parse_function_node Due to macOS visibility rules, this function did not end up being exported from liblagom-js.dylib, causing LagomWeb to fail to link.
2022-07-04 11:52:00 +00:00
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 18:27:43 +00:00
}
Reference in a new issue Copy permalink