Evaluate OH capacity at monitoring async actions

[simonrosenberg]

Typical async actions that OH must be good at:

• OH must poll github CIs
• OH must poll gcloud kube deployments

Typical errors:

• actions is run, agent checks initial state but doesn't pause itself to make sure it works end-to-end
• agent struggles at finding the error logs

What should be done:

1. Find a way to evaluate how good OH is at those tasks - We should add an integration test for this behavior
2. Improve OH's system prompt for these type of interactive tasks if any failures

[xingyaoww]

Another thing we might want to consider would be to improve the SDK so it is possible for the agent to:

1. Register a callback event (e.g., wake me up when this GitHub CI / wake me up when this command finishes execute)
2. and we wake up the agent .run() when certain conditions are met

[enyst]

Completely agree with this issue, I was going to post one myself. Reference to a previous experiment and discussion:

• codex-cli with gpt-5-codex

Please note that it wasn't even codex-max, nor 5.1, it was codex-5. Something was working very well, and it would be worth to replicate and maybe check out codex-cli code, which is open source, to see if we can learn something.

[enyst]

On another note, LLMs know gh very well. If we don't have gh in the image, maybe we could add it? I expect it's very small.

[xingyaoww]

@OpenHands please clone OpenHands/skills library, and update https://github.com/OpenHands/skills/blob/main/skills/github.md about usage of gh run watch https://cli.github.com/manual/gh_run_watch

We should add one line of text to tell the agent if user asks you to check github status, first try to use gh run watch, only fallback to basic API if it is not successful.

[openhands-ai]

I'm on it! xingyaoww can track my progress at all-hands.dev

[openhands-ai]

Summary of Work Completed

I have successfully completed the requested update to the OpenHands/skills repository:

✅ Requirements Checklist:

1. Cloned OpenHands/skills library - Repository cloned to /workspace/project/skills
2. Updated skills/github.md - Added guidance about gh run watch usage
3. Added one line instruction - Agents are now instructed to prioritize gh run watch for GitHub Actions monitoring

📝 Changes Made:

Added a single line to the <IMPORTANT> section in skills/github.md:

If the user asks you to check GitHub Actions status, first try to use `gh run watch` (https://cli.github.com/manual/gh_run_watch) to monitor workflow runs, and only fallback to basic API calls if it is not successful.

🔗 Pull Request:

• PR Link: Add gh run watch guidance for GitHub Actions monitoring skills#3
• Branch: openhands/update-github-doc-gh-run-watch
• Status: Draft PR (ready for review)

This change addresses issue #1285 by providing agents with better guidance for monitoring asynchronous GitHub Actions workflows using the gh CLI tool, which is well-known to LLMs and specifically designed for watching workflow runs.

_{View full conversation}