Add tests and docs for ES2024 (#49)

README.md

@@ -129,7 +129,9 @@ Just like for StringLiteral, templates can also contain invalid escapes. `` `\u`
 
 _Spec: [RegularExpressionLiteral]_
 
-Regex literals may contain invalid regex syntax. They are still matched as regex literals.
+Regex literals may contain invalid regex syntax. They are still matched as regex literals. The specification even says:
+
+> The productions below [...] are used by the input element scanner to find the end of the regular expression literal. The source text [...] are subsequently parsed again using the more stringent ECMAScript Regular Expression grammar.
 
 If the ending `/` is missing, the token has `closed: false`. JavaScript regex literals cannot contain newlines (not even escaped ones), so unclosed regex literals simply end at the end of the line.
 
@@ -146,6 +148,7 @@ Examples:
 /a/Inva1id
 /+/
 /[/]\//
+/[\p{Decimal_Number}--[0-9]]/v
 ```
 
 ### MultiLineComment
@@ -434,7 +437,7 @@ All possible values in JSX children:
 
 The intention is to always support the latest ECMAScript version whose feature set has been finalized.
 
-Currently, ECMAScript 2023 is supported.
+Currently, ECMAScript 2024 is supported.
 
 #### Annex B and C (strict mode)
 

test/index.test.js

@@ -815,6 +815,11 @@ describe("Token", () => {
  match("/?foo/");
  match("/*foo/", false);
 
+ match("/[p{Decimal_Number}--[0-9]]/v");
+ match("/[[[a-z]--[b-d]]&&[0-2[q{a|bc|def}]]]/v");
+ match("/[/]/v"); // Valid token, invalid at later regex parsing stage.
+ match("/[[a]/]/v", "/[[a]/"); // This is why slash isn't valid in character class when nesting came to play.
+
  match("/a/d");
  match("/a/g");
  match("/a/m");