JSSpecCompiler: Always treat trailing MemberAccess as punctuation

Due to the way expression parser is written, we need to resolve the
ambiguity between member access operators and dots used for punctuation
during lexing. The lexer uses a (totally bulletproof) heuristic to do
that: whenever '.' is followed by ' ' or '\n', it is considered a dot
and member access otherwise. While it works fine for prettified test
cases, non-prettified files often lack enter after a trailing dot
character. Since MemberAccess will always be invalid at that position,
explicitly treat trailing dot as a part of punctuation.

Meta/Lagom/Tools/CodeGenerators/JSSpecCompiler/Parser/Lexer.cpp

@@ -151,6 +151,10 @@ ParseErrorOr<TokenizeTreeResult> tokenize_tree(XML::Node const* node, bool allow
             },
             move(ignore_comments)));
     }
+
+    if (tokens.size() && tokens.last().type == TokenType::MemberAccess)
+        tokens.last().type = TokenType::Dot;
+
     return result;
 }
 

Meta/Lagom/Tools/CodeGenerators/JSSpecCompiler/Tests/spec-no-new-line-after-dot.xml

@@ -0,0 +1,19 @@
+<!DOCTYPE inline_dtd[<!ENTITY nbsp " ">]>
+<specification>
+   <emu-import>
+      <emu-clause id="1">
+         <h1><span class="secnum">1</span> The Celestial Object</h1>
+         <emu-clause id="1-1">
+            <h1><span class="secnum">1.1</span> Abstract Operations</h1>
+            <emu-clause id="1-1-1" aoid="Foo">
+               <h1><span class="secnum">1.1.1</span> Foo ( <var>a</var> )</h1>
+               <emu-alg>
+                  <ol>
+                     <li>Return <var>a</var>.<var>[[b]]</var>.</li>
+                  </ol>
+               </emu-alg>
+            </emu-clause>
+         </emu-clause>
+      </emu-clause>
+   </emu-import>
+</specification>

Meta/Lagom/Tools/CodeGenerators/JSSpecCompiler/Tests/spec-no-new-line-after-dot.xml.expectation

@@ -0,0 +1,8 @@
+===== AST after reference-resolving =====
+Foo(a):
+TreeList
+  ReturnNode
+    BinaryOperation MemberAccess
+      Var a
+      Slot b
+

Tests/JSSpecCompiler/test-runner.cpp

@@ -49,6 +49,7 @@ const Array regression_tests = {
     },
     TestDescription {
         .sources = {
+            "spec-no-new-line-after-dot.xml"sv,
             "spec-single-function-simple.xml"sv,
         },
         .flags = { dump_after_frontend },