ruby-dev-builder - debug builds

[MSP-Greg]

Currently, if any build fails, the Ruby head builds are not updated.

1. Recently a Ruby commit (with no test code) was done that omitted some methods. The methods were not covered in current testing. A subsequent commit added them, along with tests. The CI here passed on the first commit, but has since failed (due to macos-13, debug), so the second commit isn't available. This caused a failure in Puma's CI, and possibly in other repos.

2. Given that the failures are intermittent, maybe use --retry in test-all?

# Test
- run: make test-spec MSPECOPT=-j
- run: make test-all TESTS="-j4"
  if: matrix.name != 'debug'
- run: make test-all TESTS="-j4 --retry"
  if: matrix.name == 'debug'

Below is a list of recent failures here (retrieved with the GitHub API). I deleted Ubuntu 20.04 failures.

CRuby Dev Builds 88 total

Run CRuby Dev Builds  No 1965  2025-04-30T08:40:30Z id: 14739297053
    build (macos-13, debug)

Run CRuby Dev Builds  No 1964  2025-04-28T19:27:19Z id: 14715775725
    build (macos-13, debug)

Run CRuby Dev Builds  No 1961  2025-04-25T19:27:57Z id: 14671706313
    build (macos-13, debug)

Run CRuby Dev Builds  No 1958  2025-04-23T19:27:15Z id: 14626093925
    build (macos-13, debug)

Run CRuby Dev Builds  No 1957  2025-04-22T19:28:24Z id: 14602626260
    build (ubuntu-22.04, debug)

Run CRuby Dev Builds  No 1956  2025-04-21T19:27:30Z id: 14579559702
    build (ubuntu-22.04, debug)
    build (macos-13, debug)

Run CRuby Dev Builds  No 1954  2025-04-19T19:27:52Z id: 14552049204
    build (ubuntu-22.04, debug)

Run CRuby Dev Builds  No 1953  2025-04-18T19:27:03Z id: 14540342164
    build (macos-13, debug)

Run CRuby Dev Builds  No 1952  2025-04-17T19:27:38Z id: 14522957100
    build (macos-13, debug)

Run CRuby Dev Builds  No 1950  2025-04-15T19:26:25Z id: 14477389082
    build (macos-13, debug)

Run CRuby Dev Builds  No 1946  2025-04-11T19:28:30Z id: 14410269899
    build (macos-13, debug)

Run CRuby Dev Builds  No 1943  2025-04-08T19:29:26Z id: 14341581526
    build (macos-13, debug)

Run CRuby Dev Builds  No 1936  2025-04-01T19:28:43Z id: 14203939510
    build (macos-13, debug)

Run CRuby Dev Builds  No 1932  2025-03-28T19:28:07Z id: 14136110406
    build (macos-13, debug)

Run CRuby Dev Builds  No 1913  2025-03-09T19:25:56Z id: 13751596904
    build (ubuntu-24.04, asan)

Run CRuby Dev Builds  No 1893  2025-02-19T19:28:44Z id: 13420382636
    build (ubuntu-22.04-arm, debug)
    build (ubuntu-24.04-arm, debug)
    build (ubuntu-24.04, asan)

Run CRuby Dev Builds  No 1892  2025-02-18T19:30:06Z id: 13398184752
    build (ubuntu-24.04, debug)
    build (macos-13, debug)
    build (macos-14, debug)
    build (ubuntu-24.04, asan)

Run CRuby Dev Builds  No 1891  2025-02-18T11:43:45Z id: 13389366694
    build (ubuntu-24.04, debug)
    build (ubuntu-24.04-arm, debug)
    build (macos-13, debug)
    build (macos-14, debug)

Run CRuby Dev Builds  No 1885  2025-02-14T19:25:49Z id: 13335729098
    build (ubuntu-24.04-arm, head)

Run CRuby Dev Builds  No 1882  2025-02-11T19:28:35Z id: 13270541756
    build (ubuntu-24.04-arm, head)
    build (ubuntu-24.04-arm, debug)

[eregon]

1. the second commit isn't available. This caused a failure in Puma's CI, and possibly in other repos.

Right, I think this is unavoidable for head builds, things can be broken yet not fail ruby/ruby tests.
It's probably still better to not ship newer builds in the case of another more serious breakage than ship partial builds which would then even fail the download on some platforms or use an older version which would be very confusing.
IOW, I don't see any improvement possible there (alternatives all seem to have far worse trade offs), but feel free to suggest something.

2. Given that the failures are intermittent, maybe use --retry in test-all?

I think that's useful, could you make a PR with that?
I also just pushed a increase of the timeout.

[MSP-Greg]

I think that's useful, could you make a PR with that?

I will. But, now that you mention 'timeout', some of the debug failures were SEGV or timeouts, and --retry won't help with those failures...

[eregon]

For SEGV no (unless in a subprocess) but for timeouts I think --retry would retry those tests