[TEST] Add two-stage auto PR review with Claude (comment-only, no merge)#3801
[TEST] Add two-stage auto PR review with Claude (comment-only, no merge)#3801sekyondaMeta wants to merge 7 commits intomainfrom
Conversation
- Stage 1 (claude-pr-review.yml): Captures PR number on PR open, no AI/secrets - Stage 2 (claude-pr-review-run.yml): Runs Claude review in protected bedrock environment with script-generated facts section and COMMENT-only output - Harden claude-code.yml with --allowedTools Skill (matches pytorch main repo) - Update pr-review skill: SECURITY block, COMMENT-only policy, advisory labels Security: Claude cannot merge, approve, push, or execute commands. Reviews are advisory COMMENT-only. Script-generated facts provide injection-resistant anchor.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3801
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 0f11b29 with merge base 05c08f2 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
…ermission - Remove lintrunner install + run (already handled by lintrunner.yml workflow) - Remove issues:write permission (only PR comments needed, not issue writes) - Keep id-token:write (required for AWS OIDC → Bedrock auth)
…al git diff, collapsible output - Checkout at exact PR head SHA so Claude reviews the actual changed code - Switch from gh pr diff (GitHub API) to local git diff - Workflow assembles final comment (facts + Claude analysis + footer) so Claude never touches the facts section (prompt injection defense) - Claude produces only its analysis, workflow posts via gh pr comment - Update SKILL.md output format to collapsible <details> blocks - Add CI auto-review mode instructions to SKILL.md - Preserve tutorials-specific fact checks (card entry, thumbnail, deps)
| ## CI Environment (GitHub Actions) | ||
|
|
||
| This section applies when Claude is running inside the GitHub Actions workflow (`claude-code.yml`). | ||
| This section applies when Claude is running inside the GitHub Actions workflow (`claude-code.yml` or `claude-pr-review-run.yml`). |
There was a problem hiding this comment.
nit: specifying workflows prob not necessary
| Claude is invoked when a user mentions `@claude` in a PR comment or PR review comment. The triggering comment is passed as the prompt. Respond directly to what the user asked — do not perform unrequested actions. | ||
| Claude is invoked in two ways: | ||
| 1. **Auto-review**: Triggered automatically when a PR is opened or updated (via `claude-pr-review-run.yml`). The PR number and script-generated facts are passed as the prompt. | ||
| 2. **On-demand**: Triggered when a user mentions `@claude` in a PR comment (via `claude-code.yml`). The triggering comment is passed as the prompt. Respond directly to what the user asked — do not perform unrequested actions. | ||
|
|
There was a problem hiding this comment.
Does claude realy need to be told this information about how it's triggered? As a general rule of thumb, by minimizing the info in the skill to what's truly needed, you maximize your changes of the non-deterministic claude following the instructions you actually care about
| permissions: | ||
| actions: read | ||
| contents: read | ||
| pull-requests: write |
There was a problem hiding this comment.
This can be made even more secure by changing this permission to be read-only and setting up a second job (in this same file) that reads the output of this review job, has the pull-requests: write permission, and actually posts the review.
That way, the claude workflow never has access to a token with write access to anything
Example:
This job has read only permissions and invokes claude:
https://github.com/meta-pytorch/pytorch-gha-infra/pull/963/changes#diff-d4ef0e033f22c303442ef48d5fe915fab23ee38929dec0e84fbcd8c8581a99c0R14-R17
And this job (same file) has write permissions and reads the output from the first job to figure out what to write
https://github.com/meta-pytorch/pytorch-gha-infra/pull/963/changes#diff-d4ef0e033f22c303442ef48d5fe915fab23ee38929dec0e84fbcd8c8581a99c0R154-R156
| claude_args: | | ||
| --model global.anthropic.claude-sonnet-4-5-20250929-v1:0 | ||
| --allowedTools "Skill,Read,Glob,Grep" | ||
| prompt: | |
There was a problem hiding this comment.
Consider having the output be a json formatted output (example of how to enforce this).
This gives you the benefit of a much more consistent output format, with the cost of needing to add hooks to validation that claude is outputting the data correctly. If the schema you want to use is simple enough, then there's also some built in claude params that will handle that validation for you.
2 participants
Security: Claude cannot merge, approve, push, or execute commands. Reviews are advisory COMMENT-only. Script-generated facts provide injection-resistant anchor.