refactor(ecma/lexer): split lexer of parser

[bvanjoi]

We are not ensure each token is correct during parse js/ts code, so let's split parse/lexer into single crates and this crate can ensure lexer is correctly

[changeset-bot]

🦋 Changeset detected

Latest commit: 6b0e4fc

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[codspeed-hq]

CodSpeed Performance Report

Merging #10377 will improve performances by 3.05%

_{Comparing bvanjoi:rm-token-contexts (6b0e4fc) with main (773d19c)}

Summary

⚡ 1 improvements
✅ 140 untouched benchmarks
🆕 11 new benchmarks
⁉️ 11 dropped benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
🆕	es/lexer/angular	N/A	12.2 ms	N/A
🆕	es/lexer/backbone	N/A	1.7 ms	N/A
🆕	es/lexer/cal-com	N/A	33.8 ms	N/A
🆕	es/lexer/colors	N/A	44.2 µs	N/A
🆕	es/lexer/jquery	N/A	8.8 ms	N/A
🆕	es/lexer/jquery mobile	N/A	13.6 ms	N/A
🆕	es/lexer/mootools	N/A	7.1 ms	N/A
🆕	es/lexer/three	N/A	43.2 ms	N/A
🆕	es/lexer/typescript	N/A	236.2 ms	N/A
🆕	es/lexer/underscore	N/A	1.4 ms	N/A
🆕	es/lexer/yui	N/A	7.2 ms	N/A
⚡	es/lints/libs/d3	28.6 ms	27.8 ms	+3.05%
⁉️	es/lexer/angular	12.2 ms	N/A	N/A
⁉️	es/lexer/backbone	1.7 ms	N/A	N/A
⁉️	es/lexer/cal-com	33.8 ms	N/A	N/A
⁉️	es/lexer/colors	44.2 µs	N/A	N/A
⁉️	es/lexer/jquery	8.8 ms	N/A	N/A
⁉️	es/lexer/jquery mobile	13.6 ms	N/A	N/A
⁉️	es/lexer/mootools	7.1 ms	N/A	N/A
⁉️	es/lexer/three	43.2 ms	N/A	N/A
...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

[bvanjoi]

It's surprising that these failing test cases parse correctly in the playground environment, yet were passing in our earlier test suite.

For instance, the failing test case in test262::error_reporting::fail::0ff3826356c94f67.js containing ({function} = 0) was parsed successfully but generated no error reports before.

[kdy1]

CI failed, but I guess it's #10378, right? Please feel free to fix CI in any direction - except removing the whole tests.

I agree that it's a problem, but I just didn't bother to think about the solution deeply.

[bvanjoi]

Please feel free to fix CI in any direction

The issue occurs because swc_ecma_lexer doesn't receive the feature="verify" flag. Moving is_reserved_word to swc_ecma_parser resolves this.

[kdy1]

You can do ```toml [features] verify = ["swc_ecma_lexer/verify"] ``` from the swc_ecma_parser/Cargo.toml

…

Message ID: ***@***.***>

[bvanjoi]

Appreciate the clarification! The design pattern makes sense now.

[kdy1]

Thanks!