GitHub App: use checks API to report PR status

[flying-sheep]

What's the problem this feature will solve?

This is about GitHub PR runs. The in_progress part makes two things a little nicer:

1. When a commit comes in triggering a RTD job, it’s unclear if RTD is functional, as it doesn’t update the check run between queued and completed (also pending looks very similar to queued)
2. when restarting a run, it still displays as failure until it finishes again instead of as in_progress again

Describe the solution you'd like

GitHub check runs have a few state updates, including in_progress: https://docs.github.com/en/rest/checks/runs?apiVersion=2022-11-28#update-a-check-run

status string

The current status of the check run. Only GitHub Actions can set a status of waiting, pending, or requested.

Can be one of: queued, in_progress, completed, (waiting, requested, pending)

It would be great if you could update the check run with in_progress once it has started/restarted

Part 2: cancelled

There’s also a conclusion field, which you can use to display manually cancelled runs as such instead of lumping them together with regular failure ones:

conclusion string

Required if you provide completed_at or a status of completed. The final conclusion of the check. Note: Providing conclusion will automatically set the status parameter to completed. You cannot change a check run conclusion to stale, only GitHub can set this.

Can be one of: action_required, cancelled, failure, neutral, success, skipped, (stale), timed_out

[stsewd]

Hi, we use commit statuses (https://docs.github.com/en/rest/commits/statuses?apiVersion=2022-11-28), not commit checks, I think RTD is good using commit statuses, as commit checks are from what I understand are for linters or similar that report errors/warnings to a specific file/line.

when restarting a run, it still displays as failure until it finishes again instead of as in_progress again

That's probably a bug that we should fix.

[flying-sheep]

Well, the page you linked says

If you are developing a GitHub App and want to provide more detailed information about an external service, you may want to use the REST API to manage checks. For more information, see REST API endpoints for checks.

… which is what this issue is about. E.g. I’m using check runs for a benchmark server and it’s quite straightforward to use: You create a check run, store its ID together with the rest of job metadata, and then update it while the job runs and if/when it’s restarted.

Features that check runs have over commit statuses:

1. As said above: richer status and conclusion, while commit statuses only know the states “error, failure, pending, or success”, so while you could set it back to pending when restarting, it really helps to see at a glance from the icon if the status is queued or in_progress and if the conclusion is cancelled or failure
2. You can send a little summary that shows up in the “Checks” tab (example) which would e.g. allow to link both the preview page and the build page, instead of only having access to one link.
3. Yes, it has line annotations for linters/coverage/…, but you don’t need to send them at all.

Only caveat:

Write permission for the REST API to interact with checks is only available to GitHub Apps. OAuth apps and authenticated users can view check runs and check suites, but they are not able to create them. If you aren't building a GitHub App, you might be interested in using the REST API to interact with commit statuses.

[stsewd]

We are in the process of replacing our GH OAuth app for a GH app (#11942).

You can send a little summary that shows up in the “Checks” tab (example) which would e.g. allow to link both the preview page and the build page, instead of only having access to one link.

This feature looks interesting to integrate with our file tree diff feature (https://docs.readthedocs.com/platform/stable/visual-diff.html), as opposed to create a comment in the PR.

The Checks API only looks for pushes in the repository where the check suite or check run were created. Pushes to a branch in a forked repository are not detected and return an empty pull_requests array.

I assume that means that it doesn't work on PRs from forked repos?

[flying-sheep]

I assume that means that it doesn't work on PRs from forked repos?

IIRC it does work for these. I don’t know what that quote refers to 🤷

[flying-sheep]

Hi, you probably want to ask for this permission while the app in in preview mode, otherwise everyone has to accept “new permissions” when you switch the app from statuses to checks.

You don’t have to implement it immediately, just have the app require checks permission in addition to commit statuses permissions. You can always drop the statuses permission when it becomes superfluous without requiring user interaction.

[stsewd]

@flying-sheep thanks for the suggestion! We are asking for this permission now.