Integrate Test262 into WPT #229

jcscottiii · 2025-11-04T01:20:21Z

jcscottiii · 2025-11-04T16:29:32Z

rfcs/test262_integration.md

+
+## Alternatives Considered
+
+### Using `--test-types=test262`


We should move to this instead.

If we do use a separate test type internally (not testharness.js) then I agree, this would be the natural way.

I don't think the internal implementation details are the important thing. From the point of view of a consumer test262 tests are just another kind of test we can run, just like crashtests are basically a special kind of reftest, but we expose them as a separate test type because that makes more sense to users.

jcscottiii · 2025-11-04T16:38:08Z

Should protect the WPT source from landing manual changes to that repo since they won't be synced back upstream.

gsnedders · 2025-11-04T16:45:48Z

Can we open web-platform-tests/wpt@master...o-:wpt:test262-upstream as a draft PR?

jgraham · 2025-11-07T09:23:39Z

rfcs/test262_integration.md

+
+## Summary
+
+This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT. Due to its large size (>50,000 files), tests will be fetched on-demand rather than vendored into the repository. Execution will be opt-in via a new `--test262` flag. This provides a unified way to run this conformance suite while protecting the WPT repository from bloat and performance degradation.


I strongly prefer just vendoring the test262 files directly.

Although that number is large it's about a 50% increase on the current number of files in the repo, so it's not unmanagable.

Doing the vendoring directly means that there's no novel requirements for wpt consumers who want to run these tests (e.g. in vendor repos), apart from selecting the new test type.

I'd also propose a path like third_party/test262 so that it's at least clear that this is not code that lives in wpt. I do think we're likely to run into cases where people are surprised when they make changes to this directory and they aren't upstreamed; at least the third_party convention provides a clue that this might not work.

It looks like a shallow clone of Test262 is only 16M in the .git/objects/ directory, so I think that's about how much WPT would grow, which seems OK.

rfcs/test262_integration.md

jgraham · 2025-11-07T09:26:43Z

rfcs/test262_integration.md

+
+### 1. Source Management
+
+The WPT repository will contain a reference to the official `tc39/test262` repository (e.g., via a `.gitmodules` entry). This reference specifies the expected local path for the Test262 source and a pinned commit SHA.


I think we should try and be specific and not use "e.g." here.

I don't think we should use git submodules, or .gitmodules, but just use our own format e.g. a file like third_party/vendor.toml with a format like:

[test262] source = <git url> commit = <sha1>

I went ahead and changed the language to specify that we would be copying the files into the directory instead. Similar to the wasm/core directory.

jgraham · 2025-11-07T09:29:45Z

rfcs/test262_integration.md

+WPT's manifest generation and serving logic will be extended to support the on-demand nature of Test262 tests.
+
+*   **Manifest Generation:** The `wpt manifest` command will recognize the Test262 source directory and traverse its `test` directory to discover tests.
+*   **Metadata Parsing:** A new, Test262-specific parser will read the YAML frontmatter from each `.js` file to extract metadata (e.g., ES features, negative test expectations, execution flags).


To be clear: where will this end up? In MANIFEST.json? Or is the proposal to have a new special manifest format just for test262? Ideally I'd like to avoid that, but the RFC isn't providing enough information to understand the tradeoffs.

Same MANIFEST.json.

I did try it out with this PR: web-platform-tests/wpt#55997

The uncompressed manifest.json file is 36M in size now.

jgraham · 2025-11-07T09:32:35Z

rfcs/test262_integration.md

+
+### 3. Acquiring Test262 Source Code
+
+The Test262 source code is acquired on-demand. `wpt` can handle this automatically for users in a Git environment, while providing a manual path for CI systems.


As already mentioned I don't think this is the correct approach.

I think we should have a CI job that fetches the latest commit of test262, copies that over to the destination path and then opens a PR with the update. The CI should run some kind of smoketests to ensure that the update hasn't broken test262 support. Possibly a bot should be trusted to auto-merge these PRs (although that means we have to trust the test262 repository).

I changed it to this suggested approach.

gsnedders · 2025-11-07T21:52:33Z

See also: https://github.com/web-platform-tests/wpt/issues/8308

jcscottiii · 2025-11-13T01:32:06Z

@jgraham @gsnedders @foolip I updated the RFC. Also. I have updated the WIP PR too web-platform-tests/wpt#55997 so you can see what happens

foolip

Looks good, just a few comments!

foolip · 2025-11-24T16:04:40Z

rfcs/test262_integration.md

+## Implementation Considerations & Risks
+
+*   **Manifest Generation Performance:** Including the 50,000+ Test262 tests in the manifest by default will increase the time required to run `wpt manifest`. This performance cost is accepted to ensure a simpler user experience, where a single, default manifest contains all tests.
+*   **Repository Size:** Vendoring Test262 will significantly increase WPT repository size (~50,000 files), a manageable increase.


Maybe note that vendors that don't want it can exclude this directory using their existing mechanisms to exclude parts of WPT?

This would add the requirement that WPT has to work even with the directory deleted, should we test that in CI somehow?

Is there anyone who is known to not want these tests and can't simply ignore them? I feel like we're worrying a lot about people potentially being annoyed by the size and/or runtime increases, but without any specific examples of people who are actually worried by those things.

Not sure. But I just updated the RFC to have it run by default.

foolip · 2025-11-24T16:13:20Z

rfcs/test262_integration.md

+
+The current dynamic approach requires two disk I/O operations per test: one for the `wptserve` handler to parse YAML frontmatter, and a second for the browser to fetch the `.js` content. Storing metadata in `MANIFEST.json` would eliminate the first disk I/O and YAML parsing, effectively halving disk I/O operations per test during runtime.
+
+However, this optimization is deferred for initial implementation simplicity. Metadata will be parsed dynamically by `wptserve` handlers at runtime.


FWIW, I think it's a good thing to defer work until the tests are running when possible. Less time spent building the manifest makes ./wpt run feel faster as it starts running tests faster. (But total execution time should be the goal I think, and I'm sure it's sometimes faster to do the work once up front.)

rfcs/test262_integration.md

jgraham

Basically this seems fine to me, but ideally we'd change a couple of things:

Move the tests into a third_party directory with a manifest to indicate the state of the vendoring.
Make test262 tests behave like other tests as much as possible rather than making them a special case. In particular make them run by default as with every other test type.

rfcs/test262_integration.md

jgraham · 2025-11-25T10:32:04Z

rfcs/test262_integration.md

+
+*   **Manifest Generation:** `wpt manifest` will, by default, recognize `tc39/test262`, traverse its `test` directory, and add discovered tests to `MANIFEST.json`.
+*   **Metadata Parsing:** A new Test262-specific parser will read YAML frontmatter from `.js` files to extract metadata (e.g., ES features, negative expectations, execution flags). Unlike `wasm/core` tests, which are pre-processed, Test262 metadata will be used by `wptserve` handlers at runtime.
+*   **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use parsed metadata to construct the execution environment. Test modes will use distinct URL conventions based on file extensions (e.g., `path/to/test.js` served as `path/to/test.test262.html` or `path/to/test.test262-module.html`).


I don't understand why we need test262 as flags in the test URL. I think we can wire up the handlers so they only apply in the relevant test262 directories, and use a simpler mapping from source file to test URL.

That's a very good point about simplifying the URL mapping.

My main reason for using the explicit .test262.html extension in the current proof-of-concept was to handle the smoke tests. Since they live in infrastructure/test262/, a global rule based on the file extension was the most straightforward way to ensure they were picked up by the correct wptserve handler.

If we switched to a handler that only activates for paths inside third_party/test262/, it wouldn't find the smoke tests. We could explicitly add a second check for infrastructure/test262/, but hardcoding special-case paths like that into the server logic feels a bit brittle.

Another alternative could be to make the handler apply to any path that contains a test262 subdirectory. That would cover both cases neatly.

Let me know your thoughts.

rfcs/test262_integration.md

jgraham · 2025-11-25T10:39:19Z

rfcs/test262_integration.md

+## Implementation Considerations & Risks
+
+*   **Manifest Generation Performance:** Including the 50,000+ Test262 tests in the manifest by default will increase the time required to run `wpt manifest`. This performance cost is accepted to ensure a simpler user experience, where a single, default manifest contains all tests.
+*   **Repository Size:** Vendoring Test262 will significantly increase WPT repository size (~50,000 files), a manageable increase.


Is there anyone who is known to not want these tests and can't simply ignore them? I feel like we're worrying a lot about people potentially being annoyed by the size and/or runtime increases, but without any specific examples of people who are actually worried by those things.

jcscottiii added 4 commits November 4, 2025 01:18

Integrate Test262 into WPT

0b4d345

updates

b2bd357

minor edits

1d618f8

remove fluff

4ac9407

jcscottiii marked this pull request as ready for review November 4, 2025 15:31

jcscottiii commented Nov 4, 2025

View reviewed changes

jgraham requested changes Nov 7, 2025

View reviewed changes

jcscottiii added 2 commits November 12, 2025 16:47

address feedback

0e196b0

edits

b19bb88

foolip approved these changes Nov 24, 2025

View reviewed changes

jgraham reviewed Nov 25, 2025

View reviewed changes

Address feedback

efd080a


		## Summary

		This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT. Due to its large size (>50,000 files), tests will be fetched on-demand rather than vendored into the repository. Execution will be opt-in via a new `--test262` flag. This provides a unified way to run this conformance suite while protecting the WPT repository from bloat and performance degradation.


		### 1. Source Management

		The WPT repository will contain a reference to the official `tc39/test262` repository (e.g., via a `.gitmodules` entry). This reference specifies the expected local path for the Test262 source and a pinned commit SHA.


		### 3. Acquiring Test262 Source Code

		The Test262 source code is acquired on-demand. `wpt` can handle this automatically for users in a Git environment, while providing a manual path for CI systems.


		The current dynamic approach requires two disk I/O operations per test: one for the `wptserve` handler to parse YAML frontmatter, and a second for the browser to fetch the `.js` content. Storing metadata in `MANIFEST.json` would eliminate the first disk I/O and YAML parsing, effectively halving disk I/O operations per test during runtime.

		However, this optimization is deferred for initial implementation simplicity. Metadata will be parsed dynamically by `wptserve` handlers at runtime.

Integrate Test262 into WPT #229

Are you sure you want to change the base?

Integrate Test262 into WPT #229

Conversation

jcscottiii commented Nov 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcscottiii commented Nov 4, 2025

Uh oh!

gsnedders commented Nov 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gsnedders commented Nov 7, 2025

Uh oh!

jcscottiii commented Nov 13, 2025

Uh oh!

foolip left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jgraham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants