Skip to content

fix: reducing space consumption in GitHub runners #3900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
May 8, 2025

Conversation

germa89
Copy link
Collaborator

@germa89 germa89 commented May 7, 2025

Description

Last month we have seen a few errors regarding the runner running out of space.

image

Hence, let's reduce the disk footprint by removing non used files and using moving instead of copying on the logging steps.

Related link: actions/runner-images#10386

Issue linked

Related to #3751

Checklist

Summary by Sourcery

Optimize GitHub runner disk space usage by reducing file operations and cleaning up temporary files

Bug Fixes:

  • Reduce disk space usage in GitHub runners by minimizing file duplication

CI:

  • Add steps to free up disk space before and after test runs
  • Add flag to exclude logs from passed tests in pytest arguments

Chores:

  • Replace file copy operations with move operations to reduce disk space consumption

@germa89 germa89 requested a review from a team as a code owner May 7, 2025 10:00
@germa89 germa89 requested review from clatapie and pyansys-ci-bot May 7, 2025 10:00
@ansys-reviewer-bot
Copy link
Contributor

Thanks for opening a Pull Request. If you want to perform a review write a comment saying:

@ansys-reviewer-bot review

Copy link

sourcery-ai bot commented May 7, 2025

Reviewer's Guide

To address GitHub runner space limitations, this PR implements changes across CI scripts and workflows. Log collection scripts (.sh files in "/.ci/") now utilize mv commands instead of cp to handle log files, preventing duplication. The "test-local.yml" workflow incorporates pre-test cleanup by removing the CodeQL directory and post-package installation cleanup via apt-get clean. Both testing workflows ("test-local.yml" and "test-remote.yml") now use a Pytest argument to suppress logs for passed tests, further reducing disk usage. The "test-local.yml" workflow also adds steps to output disk space information before and after tests.

File-Level Changes

Change Details Files
Switched from copying to moving files in log collection scripts to reduce disk footprint.
  • Replaced cp (copy) operations with mv (move) for various log, output, error, crash, and profiling files.
  • Ensured target directory for profiling files is created before moving its contents.
/.ci/collect_mapdl_logs_remote.sh
/.ci/collect_mapdl_logs_locals.sh
Implemented disk space saving measures and logging adjustments in GitHub workflows.
  • Added a pre-test step to delete the "/__t/CodeQL" directory.
  • Modified apt install commands to include apt-get clean for removing cached package files.
  • Added the "--report-log-exclude-logs-on-passed-tests" argument to PYTEST_ARGUMENTS to reduce log output.
  • Integrated steps to display disk usage (df -h) before and after tests.
/.github/workflows/test-local.yml
/.github/workflows/test-remote.yml
Documented the fix in the changelog.
  • Added a new changelog entry detailing the reduction in GitHub runner space consumption.
doc/changelog.d/3900.fixed.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@germa89 germa89 self-assigned this May 7, 2025
Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @germa89 - I've reviewed your changes - here's some feedback:

  • The duplicated 'Free Disk Space Before Build' step at the job's end might be redundant if the runner environment is ephemeral.
  • Consider reducing the verbosity of the 'Show space consumption' step if its detailed output is primarily for initial debugging rather than routine monitoring.
  • The PR description mentions 'using moving instead of copying' to reduce disk footprint, but this change doesn't appear in the provided diff.
Here's what I looked at during the review
  • 🟢 General issues: all looks good
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@github-actions github-actions bot added CI/CD Related with CICD, Github Actions, etc maintenance General maintenance of the repo (libraries, cicd, etc) bug Issue, problem or error in PyMAPDL labels May 7, 2025
Copy link

codecov bot commented May 7, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.28%. Comparing base (1b1b6b4) to head (a17d39f).
Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3900      +/-   ##
==========================================
- Coverage   88.29%   88.28%   -0.02%     
==========================================
  Files         187      187              
  Lines       14830    14830              
==========================================
- Hits        13094    13092       -2     
- Misses       1736     1738       +2     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@germa89
Copy link
Collaborator Author

germa89 commented May 8, 2025

@sourcery-ai review

@germa89
Copy link
Collaborator Author

germa89 commented May 8, 2025

@pyansys-ci-bot LGTM

@germa89 germa89 enabled auto-merge (squash) May 8, 2025 11:32
Copy link
Contributor

@pyansys-ci-bot pyansys-ci-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approving this PR because germa89 said so in here 😬

LGTM

Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @germa89 - I've reviewed your changes - here's some feedback:

  • Consider adding df -h logging to test-remote.yml for consistent space monitoring across workflows.
  • Consider making log collection failures an error if these artifacts are critical for diagnostics, instead of only echoing a message.
Here's what I looked at during the review
  • 🟡 General issues: 2 issues found
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@germa89 germa89 changed the title fix: reducing space consumption in github runners fix: reducing space consumption in GitHub runners May 8, 2025
@germa89 germa89 merged commit fab53e1 into main May 8, 2025
85 of 88 checks passed
@germa89 germa89 deleted the fix/reducing-space-consumption-in-github-runners branch May 8, 2025 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue, problem or error in PyMAPDL CI/CD Related with CICD, Github Actions, etc maintenance General maintenance of the repo (libraries, cicd, etc)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants