test(shared-tree): Remove the report output path to let the benchmark tool to post it to pipeline

[chentong7]

Remove memory usage report specific output path to let the benchmark tool to generate report on a default path and it will be published to pipeline artifacts. Benchmarks always need the --perfMode flag to execute as such (and not as correctness unit tests). This missing one was an oversight from the past. Adding it.

[Copilot]

Pull Request Overview

This PR removes a specific output directory configuration for memory usage benchmarks to allow the benchmark tool to use its default path and publish results to pipeline artifacts automatically.

• Removes the hardcoded reportDir reporter option from the Mocha configuration
• Allows the benchmark tool to handle report output path management internally

[alexvy86]

Chatted offline; the change looks fine as far as I can tell but we'd like to see a successful run of the e2e tests pipeline with it before merging so I asked @chentong7 to do a manual run form a test/ branch.

[chentong7]

Still waiting for Build - client packages run: https://dev.azure.com/fluidframework/internal/_build/results?buildId=347013&view=results

[chentong7]

Finally, benchmark pipeline run: https://dev.azure.com/fluidframework/internal/_build/results?buildId=347037&view=results

[alexvy86]

For reference: benchmarks always need the --perfMode flag to execute as such (and not as correctness unit tests). This missing one was an oversight from the past.

[chentong7]

Include it in the PR description

[alexvy86]

@chentong7 I believe you confirmed we can use a much lower timeout, right?

[chentong7]

And it timed out at the end. Instead of making a flaky test, slow test is more acceptable.
Error: Timeout of 80000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[alexvy86]

80000 or 8000? If we have individual tests that are really taking 80s when run as benchmarks, we might need to rethink our strategy for them :/

[chentong7]

Actually, it's worse than 80s. Each test case goes 30s makes the total test set to 90s, and some old test cases just running forever.

Overall summary

status  suite name                                                                          # of passed tests  total time (s)
------  ----------------------------------------------------------------------------------  -----------------  --------------
    ✔   SharedTree table APIs memory usage Column Insertion                                        3 out of 3            90.1
    ✔   SharedTree table APIs memory usage Row Insertion                                           3 out of 3            90.1
    ✔   SharedTree table APIs memory usage Column and Row Insertion                                2 out of 2            60.1
    ✔   SharedTree table APIs memory usage Column Insertion                                        3 out of 3            90.1
    ✔   SharedTree table APIs memory usage Row Insertion                                           3 out of 3            85.2
    ✔   SharedTree table APIs memory usage Column and Row Insertion                                2 out of 2            38.9
    ✔   SharedTree table APIs memory usage Column Removal                                          2 out of 2            60.1
    ✔   SharedTree table APIs memory usage Row Removal                                             3 out of 3            82.8
    ✔   SharedTree table APIs memory usage Column and Row Removal                                  2 out of 2            60.1
    ✔   SharedTree table APIs memory usage Insert a column and a row and remove right away         1 out of 1            30.1
    ✔   SharedTree table APIs memory usage Cell Value Setting                                      3 out of 3            89.0
    ✔   SharedTree table APIs memory usage Column Insertion                                        3 out of 3            71.9
    ✔   SharedTree table APIs memory usage Row Insertion                                           3 out of 3            68.8
    ✔   SharedTree table APIs memory usage Column and Row Insertion                                2 out of 2            41.2
    ✔   SharedTree table APIs memory usage Column Insertion                                        3 out of 3            70.7
    ✔   SharedTree table APIs memory usage Row Insertion                                           3 out of 3            70.8
    ✔   SharedTree table APIs memory usage Column and Row Insertion                                2 out of 2            39.8
    ✔   SharedTree table APIs memory usage Column Removal                                          2 out of 2            45.1
    ✔   SharedTree table APIs memory usage Row Removal                                             3 out of 3            73.2
    ✔   SharedTree table APIs memory usage Column and Row Removal                                  2 out of 2            47.9
    ✔   SharedTree table APIs memory usage Insert a column and a row and remove right away         1 out of 1            16.6
    ✔   SharedTree table APIs memory usage Cell Value Setting                                      3 out of 3            68.4
    ✔   SharedTree table APIs memory usage Column Removal                                          2 out of 2            43.0
    ✔   SharedTree table APIs memory usage Row Removal                                             3 out of 3            70.8
    ✔   SharedTree table APIs memory usage Column and Row Removal                                  2 out of 2            47.1
    ✔   SharedTree table APIs memory usage Insert a column and a row and remove right away         1 out of 1            20.9
    ✔   SharedTree table APIs memory usage Cell Value Setting                                      3 out of 3            69.5
    ✔   SharedTree memory usage Forest memory usage Array of monomorphic leaves                    8 out of 8           840.6
    ✔   SharedTree memory usage Forest memory usage Array of polymorphic leaves                    8 out of 8           837.8
    ✔   SharedTree memory usage Forest memory usage Array of deep monomorphic leaves               8 out of 8         1,016.4
    !   SharedTree memory usage                                                                   4 out of 10           120.8
        total                                                                                    93 out of 99         4,458.2

This @fluidframework-tree test step itself just goes over 1.5h

[alexvy86]

Pending confirmation from a manual run of the pipeline, this looks good to me. Just update the timeout to the lower value that you tested with.

[chentong7]

Run memory usage tests - @fluidframework/tree pipeline step goes over 1h: https://dev.azure.com/fluidframework/internal/_build/results?buildId=347037&view=logs&j=12bb9097-5cdf-5833-9679-3c0d8d960191&t=9d111fa8-556a-5c73-2a6e-ed547ec5ff4f

[chentong7]

Run memory usage tests - @fluidframework/tree pipeline step goes over 1.5h: https://dev.azure.com/fluidframework/internal/_build/results?buildId=347069&view=logs&j=12bb9097-5cdf-5833-9679-3c0d8d960191&t=0e0d0841-03e8-5a8c-1611-e689df253089

[alexvy86]

@chentong7 how do local runs look? How long do they take to complete?

[chentong7]

@chentong7 how do local runs look? How long do they take to complete?

Yes, each test case takes 30s to complete. I put my local timeout to 5mins and it last ~1h for 10 times test cases and hours(like before bed) for 100 times.

I disable the old tests first because some old tests is just forever long:
✔ SharedTree memory usage Forest memory usage Array of deep monomorphic leaves 8 out of 8 1,016.4
1016.4s? 16mins?

[chentong7]

Sync-up with Josh offline, we shouldn't enable this long running test package in pipeline. Pipeline jobs can be run in parallel. Open a task to modify this pipeline to multi-job mode then come back to it to enable tree package in benchmark pipeline.