feat: parallelized integration tests

[Weves]

Description

[Provide a brief description of the changes in this PR]

How Has This Been Tested?

[Describe the tests you ran to verify your changes]

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
internal-search	Ready	Preview	Comment	Sep 10, 2025 5:56pm

[greptile-apps]

Greptile Summary

This PR introduces parallelization to the Model Integration Tests (MIT) workflow by implementing a matrix strategy in GitHub Actions. The key changes include:

1. Adding a new job discover-test-dirs that dynamically identifies test directories
2. Implementing a matrix strategy to run tests in parallel
3. Modifying the log artifact collection to be test-directory specific

These changes maintain the existing test functionality while potentially reducing the overall execution time significantly. The parallelization is implemented in a way that maintains isolation between test suites and improves debugging capabilities through directory-specific logging.

Confidence score: 4/5

1. This PR is safe to merge with standard review attention
2. The changes are well-structured, maintain existing functionality, and follow GitHub Actions best practices
3. The .github/workflows/pr-mit-integration-tests.yml file needs careful review, particularly around the matrix strategy implementation and log collection modifications

_{1 file reviewed, 1 comment
Edit PR Review Bot Settings | Greptile}

[justin-tahara]

@greptileai

[greptile-apps]

Greptile Summary

This PR transforms the MIT integration test workflow from a sequential, monolithic approach to a parallelized matrix-based system. The key architectural changes include:

Dynamic Test Discovery: The workflow now automatically discovers test directories using find commands and builds a JSON matrix of test paths, eliminating the need to manually maintain test configurations.

Parallelized Build Process: The original single build step is split into separate parallel jobs (prepare-build, build-backend-image, build-model-server-image, build-integration-image) that can execute simultaneously, reducing overall build time.

Infrastructure Modernization: The workflow migrates from GitHub-hosted runners to Blacksmith runners and introduces AWS ECR for Docker image storage. Images are built once, pushed to ECR, then pulled by each matrix job for test execution.

Matrix-Based Test Execution: Tests now run in parallel across discovered test directories using GitHub Actions' matrix strategy, with each test directory executing on its own runner instance.

Enhanced Reliability: Retry logic is added using nick-fields/retry@v3 to handle test flakiness, and the workflow includes improved error handling and logging.

The custom Docker build action is also extended with new optional parameters (outputs, provenance, build-args) to support the more sophisticated build requirements of the parallelized system. This change integrates with the existing codebase by maintaining the same test execution patterns while dramatically improving performance through concurrent execution.

Confidence score: 4/5

• This PR introduces significant architectural improvements but adds infrastructure complexity that requires careful validation
• Score reflects well-structured parallelization approach but concerns about AWS dependencies and runner coordination
• Pay close attention to the test discovery logic and matrix configuration in the workflow files

_{2 files reviewed, no comments}

_{Edit Code Review Bot Settings | Greptile}

[justin-tahara]

Leaving some initial comments and questions. I think if it looks good we should merge and start testing the other tests as well so that we reduce CI time across the board. Thanks for doing this Chris!

[justin-tahara]

Are we saying we need a larger instance because of how much CPU cores are needed for building the image?

[Weves]

I think we used to use 32cpu-linux-x64, so this is smaller. Not sure if we actually need this large of an instance though 🤔

[justin-tahara]

How does this ECR Registry get updated with our backend images? Do we need to do this for all images and move them over to ECR?

[Weves]

we also build them + push them in this action. For our public images, we can still use dockerhub.

It's nicer imo to just use one container repo across the board, but dockerhub seems to be throttling us for these high bandwidth use cases :(

[justin-tahara]

Thanks for addressing the comments!