Skip to content

[CSR-791] cypress-cloud is not compatible with cypress 13.x #179

@anlambert

Description

@anlambert

Before opening, please confirm:

  • I have searched for duplicate or closed issues and discussions.
  • I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
  • I acknowledge that I will attach a full debug log, otherwise the issue will be closed with no response.

Environment information

# Put output below this line

System:
    OS: Linux 6.1 Debian GNU/Linux 12 (bookworm) 12 (bookworm)
    CPU: (8) x64 Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz
    Memory: 18.94 GB / 31.16 GB
    Container: Yes
    Shell: 5.2.15 - /bin/bash
  Binaries:
    Node: 18.16.0 - /usr/bin/node
    Yarn: 1.22.19 - /usr/bin/yarn
    npm: 9.5.1 - /usr/bin/npm
  Browsers:
    Chrome: 114.0.5735.90
    Chromium: 116.0.5845.140
  npmPackages:
    eslint-config-custom: latest => 0.0.0 
    prettier: latest => 3.0.2 
    turbo: ^1.10.13 => 1.10.13 
  npmGlobalPackages:
    azurite: 3.23.0
    corepack: 0.17.0
    gulp-cli: 2.3.0
    npm: 9.5.1
    samfirm: 0.2.0

Describe the bug

Due to this cypress commit that introduces some breaking API changes in cypress 13.x, I noticed two issues when using cypress-cloud with it:

  1. Executed cypress tests are always reported as failing whatever their real statuses are.

See sample cypress-cloud output put below:

$ yarn run cypress-cloud run --parallel --record --key somekey --cloud-debug true --ci-build-id hello-cypress36
yarn run v1.22.19
$ /home/anlambert/swh/swh-environment/swh-web/node_modules/.bin/cypress-cloud run --parallel --record --key somekey --cloud-debug true --ci-build-id hello-cypress36
 WARNING  Failed to load config file, falling back to the default config. Attempted locations: [
  'file:///home/anlambert/swh/swh-environment/swh-web/currents.config.js',
  'file:///home/anlambert/swh/swh-environment/swh-web/currents.config.cjs',
  'file:///home/anlambert/swh/swh-environment/swh-web/currents.config.mjs'
]

Copyright (C) 2023 Currents Software Inc https://currents.dev
This is free software, and you are welcome to redistribute it under certain
conditions. This program comes with no warranty. Parts of this program are MIT
licensed. Refer to the license for details
https://github.com/currents-dev/cypress-cloud/blob/main/LICENSE.md

Discovered 1 spec files
Tags: false; Group: false; Parallel: true; Batch Size: 3
Connecting to cloud orchestration service...
🎥 Run URL: http://localhost:8080/run/aff6e1ec70f400b22c4deeb5b83332e7

====================================================================================================

Running: cypress/e2e/home.cy.js (1/1)

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.1.0                                                                         │
  │ Browser:        Electron 106 (headless)                                                        │
  │ Node Version:   v18.16.0 (/usr/bin/node)                                                       │
  │ Specs:          1 found (home.cy.js)                                                           │
  │ Searched:       cypress/e2e/home.cy.js                                                         │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  home.cy.js                                                                      (1 of 1)


  Home Page Tests
    ✓ should have focus on search form after page load
    ✓ should display positive stats for each category
    ✓ should display null counters and hide history graphs when storage is empty
    ✓ should hide counters when data is missing
    ✓ should redirect to search page when submitting search form


  5 passing (6s)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        5                                                                                │
  │ Passing:      5                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     6 seconds                                                                        │
  │ Spec Ran:     home.cy.js                                                                       │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  home.cy.js                               00:06        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:06        5        5        -        -        -  


  Reporting results and artifacts in background...  


====================================================================================================


  Cloud Run Finished  

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │     Spec                                         Tests   Passing   Failing   Pending   Skipped │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖   cypress/e2e/home.cy.js               0ms         1         -         1         -         - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖   1 of 1 failed                        0ms         1         -         1         -         - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

🏁 Recorded Run: http://localhost:8080/run/aff6e1ec70f400b22c4deeb5b83332e7

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

After reading cypress-cloud code, it seems that issue can be fixed using the following diff:

diff --git a/packages/cypress-cloud/lib/results/results.ts b/packages/cypress-cloud/lib/results/results.ts
index f9380de..635f0b4 100644
--- a/packages/cypress-cloud/lib/results/results.ts
+++ b/packages/cypress-cloud/lib/results/results.ts
@@ -12,11 +12,6 @@ import { ConfigState } from "../state";
 
 const debug = Debug("currents:results");
 
-export const isSuccessResult = (
-  result: CypressResult
-): result is CypressCommandLine.CypressRunResult => {
-  return result.status === "finished";
-};
 
 export const getScreenshotsSummary = (
   tests: CypressCommandLine.TestResult[] = []
@@ -282,9 +277,6 @@ export function getCypressRunResultForSpec(
   spec: string,
   cypressResult: CypressResult
 ): CypressCommandLine.CypressRunResult | undefined {
-  if (!isSuccessResult(cypressResult)) {
-    return;
-  }
 
   const run = cypressResult.runs.find((r) => r.spec.relative === spec);
   if (!run) {
  1. Screenshots fail to be processed.

Once the fix above applied, there is an error when cypress-cloud attempts to process test screenshots, see output below:

$ yarn run cypress-cloud run --parallel --record --key somekey --cloud-debug true --ci-build-id hello-cypress38
yarn run v1.22.19
$ /home/anlambert/swh/swh-environment/swh-web/node_modules/.bin/cypress-cloud run --parallel --record --key somekey --cloud-debug true --ci-build-id hello-cypress38
 WARNING  Failed to load config file, falling back to the default config. Attempted locations: [
  'file:///home/anlambert/swh/swh-environment/swh-web/currents.config.js',
  'file:///home/anlambert/swh/swh-environment/swh-web/currents.config.cjs',
  'file:///home/anlambert/swh/swh-environment/swh-web/currents.config.mjs'
]

Copyright (C) 2023 Currents Software Inc https://currents.dev
This is free software, and you are welcome to redistribute it under certain
conditions. This program comes with no warranty. Parts of this program are MIT
licensed. Refer to the license for details
https://github.com/currents-dev/cypress-cloud/blob/main/LICENSE.md

Discovered 1 spec files
Tags: false; Group: false; Parallel: true; Batch Size: 3
Connecting to cloud orchestration service...
🎥 Run URL: http://localhost:8080/run/8daacfecbd761c3d663e7e4bd2646eae

====================================================================================================

Running: cypress/e2e/home.cy.js (1/1)

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.1.0                                                                         │
  │ Browser:        Electron 106 (headless)                                                        │
  │ Node Version:   v18.16.0 (/usr/bin/node)                                                       │
  │ Specs:          1 found (home.cy.js)                                                           │
  │ Searched:       cypress/e2e/home.cy.js                                                         │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  home.cy.js                                                                      (1 of 1)


  Home Page Tests
    ✓ should have focus on search form after page load
    ✓ should display positive stats for each category
    ✓ should display null counters and hide history graphs when storage is empty
    ✓ should hide counters when data is missing
    ✓ should redirect to search page when submitting search form


  5 passing (6s)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        5                                                                                │
  │ Passing:      5                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     6 seconds                                                                        │
  │ Spec Ran:     home.cy.js                                                                       │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  home.cy.js                               00:06        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:06        5        5        -        -        -  


  Reporting results and artifacts in background...  

 ERROR  TypeError: Cannot read properties of undefined (reading 'flatMap')
    at /home/anlambert/swh/swh-environment/swh-web/node_modules/lib/results/results.ts:26:21
    at Array.flatMap (<anonymous>)
    at /home/anlambert/swh/swh-environment/swh-web/node_modules/lib/results/results.ts:25:19
    at Array.flatMap (<anonymous>)
    at getScreenshotsSummary (/home/anlambert/swh/swh-environment/swh-web/node_modules/lib/results/results.ts:24:16)
    at getInstanceResultPayload (/home/anlambert/swh/swh-environment/swh-web/node_modules/lib/results/results.ts:62:18)
    at getReportResultsTask (/home/anlambert/swh/swh-environment/swh-web/node_modules/lib/results/uploadResults.ts:22:13)
    at createReportTask (/home/anlambert/swh/swh-environment/swh-web/node_modules/lib/runner/reportTask.ts:26:3)
    at /home/anlambert/swh/swh-environment/swh-web/node_modules/lib/runner/runner.ts:55:7
    at Array.forEach (<anonymous>)
    at runTillDone (/home/anlambert/swh/swh-environment/swh-web/node_modules/lib/runner/runner.ts:54:7)


====================================================================================================


  Cloud Run Finished  

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │     Spec                                         Tests   Passing   Failing   Pending   Skipped │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔   undefined                           6.2s         5         5         -         -         - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔   All specs passed!                   6.2s         5         5         -         -         - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

🏁 Recorded Run: http://localhost:8080/run/8daacfecbd761c3d663e7e4bd2646eae

Done in 13.82s.

The fix for this one is not straightforward to me.

Expected behavior

cypress 13.x should be supported.

Command and Setup

Just run any cypress-cloud session with cypress 13.x and the reported errors will show up.

Full log and debug output

cypress_cloud_debug_logs.txt

Metadata

Metadata

Assignees

Labels

wontfixThis will not be worked on

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions