Skip to content

Commit fab53e1

Browse files
germa89pyansys-ci-bot
germa89
and
pyansys-ci-bot
authored
fix: reducing space consumption in github runners (#3900)
* feat: add disk space usage reporting and cleanup steps in local testing workflow * feat: add disk space cleanup step before build in local testing workflow * chore: adding changelog file 3900.fixed.md [dependabot-skip] * feat: enhance disk space reporting in local testing workflow * fix: update disk usage reporting to show current directory size * fix: improve disk usage reporting with error handling in local testing workflow * fix: remove redundant disk usage checks and streamline reporting in local testing workflow * fix: replace cp with mv for log file handling in local and remote scripts * fix: update log file handling to create directories before moving files in local and remote scripts * fix: update tar command to remove files after compression in log collection scripts * fix: update pytest arguments to exclude logs on passed tests in local and remote workflows * fix: add disk usage checks for runner directories in local testing workflow * fix: add disk usage check for CodeQL directory in local testing workflow * fix: remove CodeQL directory after disk usage check in local testing workflow * fix: enhance disk usage reporting and cleanup for CodeQL directory in local testing workflow * fix: streamline disk usage reporting and cleanup for CodeQL directory in local testing workflow --------- Co-authored-by: pyansys-ci-bot <92810346+pyansys-ci-bot@users.noreply.github.com>
1 parent a269e75 commit fab53e1

File tree

5 files changed

+58
-27
lines changed

5 files changed

+58
-27
lines changed

‎.ci/collect_mapdl_logs_locals.sh

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,18 @@
22
mkdir "$LOG_NAMES" && echo "Successfully generated directory $LOG_NAMES"
33

44
echo "Copying the log files..."
5-
cp *.log ./"$LOG_NAMES"/ || echo "No log files could be found"
6-
cp *apdl.out ./"$LOG_NAMES"/ || echo "No APDL log files could be found"
7-
cp *pymapdl.apdl ./"$LOG_NAMES"/ || echo "No PYMAPDL APDL log files could be found"
5+
mv *.log ./"$LOG_NAMES"/ || echo "No log files could be found"
6+
mv *apdl.out ./"$LOG_NAMES"/ || echo "No APDL log files could be found"
7+
mv *pymapdl.apdl ./"$LOG_NAMES"/ || echo "No PYMAPDL APDL log files could be found"
88

99
echo "Copying the profiling files..."
10-
cp -r prof ./"$LOG_NAMES"/prof || echo "No profile files could be found"
10+
mkdir -p ./"$LOG_NAMES"/prof
11+
mv prof/* ./"$LOG_NAMES"/prof || echo "No profile files could be found"
1112

1213
echo "Copying the JSONL files..."
13-
cp *.jsonl ./"$LOG_NAMES"/ || echo "No JSONL files could be found"
14+
mv *.jsonl ./"$LOG_NAMES"/ || echo "No JSONL files could be found"
1415

1516
ls -la ./"$LOG_NAMES"
1617

1718
echo "Tar files..."
18-
tar cvzf ./"$LOG_NAMES".tgz ./"$LOG_NAMES" || echo "Failed to compress"
19+
tar --remove-files -cvzf ./"$LOG_NAMES".tgz ./"$LOG_NAMES" || echo "Failed to compress"

‎.ci/collect_mapdl_logs_remote.sh

Lines changed: 14 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,43 +3,41 @@ if [[ $MAPDL_VERSION == *"ubuntu"* ]] ; then
33
echo "It is an ubuntu based image"
44
export FILE=/jobs/file
55
export WDIR='/jobs/'
6-
76
else
87
echo "It is a CentOS based image"
98
export FILE=file
109
export WDIR=""
11-
1210
fi;
1311

14-
1512
mkdir "$LOG_NAMES" && echo "Successfully generated directory $LOG_NAMES"
1613

1714
####
1815
echo "Collecting MAPDL logs..."
1916

2017
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "mkdir -p /mapdl_logs && echo 'Successfully created directory inside docker container'") || echo "Failed to create a directory inside docker container for logs."
21-
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "if compgen -G '$FILE*.out' > /dev/null ;then cp -f /file*.out /mapdl_logs && echo 'Successfully copied out files.'; fi") || echo "Failed to copy the 'out' files into a local file"
22-
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "if compgen -G '$FILE*.err' > /dev/null ;then cp -f /file*.err /mapdl_logs && echo 'Successfully copied err files.'; fi") || echo "Failed to copy the 'err' files into a local file"
23-
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "if compgen -G '$FILE*.log' > /dev/null ;then cp -f /file*.log /mapdl_logs && echo 'Successfully copied log files.'; fi") || echo "Failed to copy the 'log' files into a local file"
24-
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "if compgen -G '$WDIR*.crash' > /dev/null ;then cp -f /*.crash /mapdl_logs && echo 'Successfully copied crash files.'; fi") || echo "Failed to copy the 'crash' files into a local file"
18+
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "if compgen -G '$FILE*.out' > /dev/null ;then mv -f /file*.out /mapdl_logs && echo 'Successfully moved out files.'; fi") || echo "Failed to move the 'out' files into a local file"
19+
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "if compgen -G '$FILE*.err' > /dev/null ;then mv -f /file*.err /mapdl_logs && echo 'Successfully moved err files.'; fi") || echo "Failed to move the 'err' files into a local file"
20+
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "if compgen -G '$FILE*.log' > /dev/null ;then mv -f /file*.log /mapdl_logs && echo 'Successfully moved log files.'; fi") || echo "Failed to move the 'log' files into a local file"
21+
(docker exec "$MAPDL_INSTANCE" /bin/bash -c "if compgen -G '$WDIR*.crash' > /dev/null ;then mv -f $WDIR*.crash /mapdl_logs && echo 'Successfully moved crash files.'; fi") || echo "Failed to move the 'crash' files into a local file"
2522

2623
docker cp "$MAPDL_INSTANCE":/mapdl_logs/. ./"$LOG_NAMES"/. || echo "Failed to copy the 'log-build-docs' files into a local directory"
2724

2825
####
2926
echo "Collecting local build logs..."
3027

3128
echo "Collecting docker run log..."
32-
cp log.txt ./"$LOG_NAMES"/log.txt || echo "MAPDL run docker log not found."
29+
mv log.txt ./"$LOG_NAMES"/log.txt || echo "MAPDL run docker log not found."
3330

34-
echo "Copying docker launch log..."
35-
cp mapdl_launch_0.log ./"$LOG_NAMES"/mapdl_launch_0.log || echo "MAPDL launch docker log not found."
36-
cp mapdl_launch_1.log ./"$LOG_NAMES"/mapdl_launch_1.log || echo "MAPDL launch docker log not found."
31+
echo "Moving docker launch log..."
32+
mv mapdl_launch_0.log ./"$LOG_NAMES"/mapdl_launch_0.log || echo "MAPDL launch docker log not found."
33+
mv mapdl_launch_1.log ./"$LOG_NAMES"/mapdl_launch_1.log || echo "MAPDL launch docker log not found."
3734

38-
echo "Copying the profiling files..."
39-
cp -r prof ./"$LOG_NAMES"/prof || echo "No profile files could be found"
35+
echo "Moving the profiling files..."
36+
mkdir -p ./"$LOG_NAMES"/prof
37+
mv prof/* ./"$LOG_NAMES"/prof || echo "No profile files could be found"
4038

41-
echo "Copying the JSONL files..."
42-
cp *.jsonl ./"$LOG_NAMES"/ || echo "No JSONL files could be found"
39+
echo "Moving the JSONL files..."
40+
mv *.jsonl ./"$LOG_NAMES"/ || echo "No JSONL files could be found"
4341

4442
echo "Collecting file structure..."
4543
ls -R > ./"$LOG_NAMES"/files_structure.txt || echo "Failed to copy file structure to a file"
@@ -48,4 +46,4 @@ echo "Collecting docker file structure..."
4846
docker exec "$MAPDL_INSTANCE" /bin/bash -c "ls -R" > ./"$LOG_NAMES"/docker_files_structure.txt || echo "Failed to copy the docker structure into a local file"
4947

5048
echo "Tar files..."
51-
tar cvzf ./"$LOG_NAMES".tgz ./"$LOG_NAMES" || echo "Failed to compress"
49+
tar --remove-files -cvzf ./"$LOG_NAMES".tgz ./"$LOG_NAMES" || echo "Failed to compress"

‎.github/workflows/test-local.yml

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ jobs:
120120
TESTING_MINIMAL: ${{ inputs.testing-minimal }}
121121
P_SCHEMA: "/ansys_inc/v241/ansys/ac4/schema"
122122
PYTEST_TIMEOUT: 120 # seconds. Limit the duration for each unit test
123-
PYTEST_ARGUMENTS: '-vvv -rxXsa --color=yes --durations=30 --random-order --random-order-bucket=class --maxfail=10 --reruns 3 --reruns-delay 4 --cov=ansys.mapdl.core --cov-report=html --timeout=180 --profile-svg --profile'
123+
PYTEST_ARGUMENTS: '-vvv -rxXsa --color=yes --durations=30 --random-order --random-order-bucket=class --maxfail=10 --reruns 3 --reruns-delay 4 --cov=ansys.mapdl.core --cov-report=html --timeout=180 --profile-svg --profile --report-log-exclude-logs-on-passed-tests'
124124
OMPI_ALLOW_RUN_AS_ROOT: 1
125125
OMPI_ALLOW_RUN_AS_ROOT_CONFIRM: 1
126126

@@ -132,6 +132,15 @@ jobs:
132132
password: ${{ secrets.token }}
133133

134134
steps:
135+
- name: "Freeing some space and show space consumption (pre-test)"
136+
shell: bash
137+
run: |
138+
echo "Deleting CodeQL..."
139+
rm -rf /__t/CodeQL || echo "CodeQL not found"
140+
141+
echo "Disk space:"
142+
df -h
143+
135144
- name: "Install Git and checkout project"
136145
uses: actions/checkout@v4.2.2
137146
with:
@@ -162,13 +171,13 @@ jobs:
162171
shell: bash
163172
if: inputs.testing-minimal == true
164173
run: |
165-
apt-get update && apt install -y libgomp1 graphviz
174+
apt-get update && apt install -y libgomp1 graphviz && apt-get clean
166175
167176
- name: "Installing OS packages"
168177
shell: bash
169178
if: inputs.testing-minimal == false
170179
run: |
171-
apt-get update && apt install -y libgl1-mesa-glx xvfb libgomp1 graphviz
180+
apt-get update && apt install -y libgl1-mesa-glx xvfb libgomp1 graphviz && apt-get clean
172181
173182
- name: "Setup Python"
174183
uses: actions/setup-python@v5
@@ -289,3 +298,9 @@ jobs:
289298
name: ${{ inputs.file-name }}.xml
290299
path: ./${{ inputs.file-name }}.xml
291300

301+
- name: "Show space consumption (post-test)"
302+
if: always()
303+
shell: bash
304+
run: |
305+
echo "Disk space:"
306+
df -h

‎.github/workflows/test-remote.yml

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,10 +63,19 @@ jobs:
6363
PYMAPDL_DB_PORT2: 21003 # default won't work on GitHub runners
6464
DPF_PORT: 21004
6565
DPF_START_SERVER: False
66-
PYTEST_ARGUMENTS: '-vvv -rxXsa --color=yes --durations=30 --random-order --random-order-bucket=class --maxfail=10 --reruns 3 --reruns-delay 4 --cov=ansys.mapdl.core --cov-report=html --timeout=180 --profile-svg --profile'
66+
PYTEST_ARGUMENTS: '-vvv -rxXsa --color=yes --durations=30 --random-order --random-order-bucket=class --maxfail=10 --reruns 3 --reruns-delay 4 --cov=ansys.mapdl.core --cov-report=html --timeout=180 --profile-svg --profile --report-log-exclude-logs-on-passed-tests'
6767
MAPDL_PACKAGE: ghcr.io/ansys/mapdl
6868

6969
steps:
70+
- name: "Freeing some space and show space consumption (pre-test)"
71+
shell: bash
72+
run: |
73+
echo "Deleting CodeQL..."
74+
rm -rf /__t/CodeQL || echo "CodeQL not found"
75+
76+
echo "Disk space:"
77+
df -h
78+
7079
- name: "Install Git and checkout project"
7180
uses: actions/checkout@v4.2.2
7281

@@ -263,4 +272,11 @@ jobs:
263272
LOG_NAMES: logs-${{ inputs.file-name }}
264273
shell: bash
265274
run: |
266-
.ci/display_logs_remote.sh
275+
.ci/display_logs_remote.sh
276+
277+
- name: "Show space consumption (post-test)"
278+
if: always()
279+
shell: bash
280+
run: |
281+
echo "Disk space:"
282+
df -h

‎doc/changelog.d/3900.fixed.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
fix: reducing space consumption in GitHub runners

0 commit comments

Comments
 (0)