Skip to content
Open
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
185 changes: 185 additions & 0 deletions .github/workflows/main-ci-pipeline.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
name: CI/CD Pipeline

on:
push:
branches:
- main
pull_request:
types:
- opened
- reopened
- synchronize

jobs:

build-job:
name: Build
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3

- name: Set up CMake
uses: lukka/get-cmake@v3.31.0

# Run the setup script to install dependencies
- name: Run setup script
run: |
chmod +x build-utils/pipe_setup.sh
./build-utils/pipe_setup.sh

- name: Build project
run: |
cmake -S . -B cmake-build-debug -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DENABLE_COVERAGE=ON -DCOVERAGE_DIR_NAME="coverage"
cmake --build cmake-build-debug

- name: Upload build artifacts
uses: actions/upload-artifact@v3
with:
name: build-artifacts
path: |
cmake-build-debug
retention-days: 1

test-job:
name: Test
runs-on: ubuntu-latest
needs: build-job
steps:
- name: Checkout code
uses: actions/checkout@v3

- name: Set up CMake
uses: lukka/get-cmake@v3.31.0

# Run the setup script to install dependencies
- name: Run setup script
run: |
chmod +x build-utils/pipe_setup.sh
./build-utils/pipe_setup.sh

- name: Download build artifacts
uses: actions/download-artifact@v3
with:
name: build-artifacts
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be unused at the moment. Are you trying to share something between jobs, or is it just a leftover?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is one of the main issues I am having right now. I am trying to share the cmake-build-debug dir created in the build job so we do not have to build the project again in every job. The dir is needed in the clang-tidy and the test job. However, this does not seem to work the way I want it to as the dir apparently cannot be found in the test job for example:

Run ctest --test-dir cmake-build-debug/test --rerun-failed --output-on-failure Failed to change working directory to "/home/runner/work/surviving-sarntal/surviving-sarntal/cmake-build-debug/test" : No such file or directory Internal ctest changing into directory: /home/runner/work/surviving-sarntal/surviving-sarntal/cmake-build-debug/test Error: Process completed with exit code 1.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent of the direction you take next, I think that uploading/downloading artifacts is not intended for this use case.

I think this could be much easier if you indeed ran everything inside a container, because then you could write into a persistent directory (depends a bit on how you run the container).

Another alternative would be to not split multiple jobs, but just have one job with multiple steps.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. The persistent directory is why we will try with docker container next (see comment below) 👍


- name: Run tests
run: |
ctest --test-dir cmake-build-debug/test --rerun-failed --output-on-failure
cmake --build cmake-build-debug --target coverage
COVERAGE=$(lcov --summary cmake-build-debug/../coverage/coverage.info | awk '/lines/ {print $2}')
echo "Line coverage $COVERAGE"

# Remove percentage symbol and compare numeric value
COVERAGE_VALUE=$(echo "$COVERAGE" | sed 's/%//')
if [ "$COVERAGE_VALUE" -lt 80 ]; then
echo "Coverage is below 80%! Failing pipeline."
# TODO once the test coverage is high enough, we should fail the pipeline here
# exit 1
else
echo "Coverage is sufficient: $COVERAGE_VALUE%"
fi
Comment on lines +71 to +79
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could also store the coverage value in a file and check if the PR increases or decreases the coverage. This would be easier to track than eventually (and cumulatively) dropping below a threshold.

Or run lcov on both branches.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a great idea! I will tackle this once the pipeline is up and running.


- name: Upload coverage report
uses: actions/upload-artifact@v3
with:
name: coverage-report
path: coverage/report
retention-days: 1

clang-format:
name: Clang Format
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3

- name: Set up CMake
uses: lukka/get-cmake@v3.31.0

# Run the setup script to install dependencies
- name: Run setup script
run: |
chmod +x build-utils/pipe_setup.sh
./build-utils/pipe_setup.sh

- name: Debug LLVM Binaries
run: |
sudo find /usr /lib /opt -name clang-format-18 -o -name clang-tidy-18
dpkg-query -L clang-format-18 || echo "clang-format-18 not found in package files"
llvm-config --bindir || echo "llvm-config not installed or misconfigured"
echo $PATH
Comment on lines +103 to +108
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the clang-format setup looks too complicated. Do you know that you can get any clang-format version from pip?

https://pypi.org/project/clang-format/

You could also use a predefined clang-format action or, even better, use a predefined pre-commit hook. See an example in https://github.com/precice/precice/blob/develop/.pre-commit-config.yaml

Copy link
Collaborator Author

@bjoheimer bjoheimer Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The setup was originally not that complicated. Those are just remnants of my efforts to debug the pipeline and will be removed.
I did not know that I could install clang via pip and I will try this tmrw.
I do have pre-commit hooks in place running clang-format and clang-tidy locally on the changed files. However we agreed on running the two in the pipeline as well on all our src files as there are some errors clang-tidy only catches when looking at all files.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could still run pre-commit on all files in the CI (pre-commit run -a).
The advantage of integrating pre-commit in the CI is that you avoid duplication and get the same checks that a developer should also get locally.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is smart. Originally, we had a very weak runner which is why we split up the pre-commit linting into clang-format and clang-tidy. clang-tidy needs a build which took forever on our old runner, so we performed clang-format first so that the pipe "fails fast". However, now performance is no longer an issue, so it is sensible to reuse the pre-commit checks for the reasons you mentioned.



- name: Run clang-format
run: |
clang-format -style=file $(find src/ test/ -type f \( -name "*.cpp" -o -name "*.hpp" -o -name "*.h" \)) --Werror --dry-run

clang-tidy:
name: Clang Tidy
runs-on: ubuntu-latest
needs: [build-job, test-job]
steps:
- name: Checkout code
uses: actions/checkout@v3

- name: Set up CMake
uses: lukka/get-cmake@v3.31.0

# Run the setup script to install dependencies
- name: Run setup script
run: |
chmod +x build-utils/pipe_setup.sh
./build-utils/pipe_setup.sh

- name: Check clang-tidy version
run: |
# Check and install clang-tidy version 18 if needed
if command -v clang-tidy &> /dev/null; then
INSTALLED_VERSION=$(clang-tidy --version | grep -oP '(?<=version )\d+' | head -n 1) # Extracts major version after "version"
if [ "$INSTALLED_VERSION" -lt 18 ]; then
echo "Clang-tidy version $INSTALLED_VERSION is less than 18. Installing clang-tidy version 18..."
sudo apt-get install -y clang-tidy-18
sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-18 100
else
echo "Clang-tidy version $INSTALLED_VERSION is sufficient."
fi
else
echo "Clang-tidy is not installed. Installing clang-tidy version 18..."
sudo apt-get install -y clang-tidy-18
sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-18 100
fi

- name: Download build artifacts
uses: actions/download-artifact@v3
with:
name: build-artifacts
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above: Do you need this?

Copy link
Collaborator Author

@bjoheimer bjoheimer Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, if it does what I assume it does, I need the compile_commands.json file which is outputted to cmake-build-debug during the build.


- name: Run clang-tidy
run: |
clang-tidy -p=cmake-build-debug/ $(find src/ test/ -type f \( -name "*.cpp" -o -name "*.hpp" -o -name "*.h" \)) --warnings-as-errors=* --header-filter="^((src|test)/.*)$" --use-color

run-exe:
name: Run Executable
runs-on: ubuntu-latest
needs: build-job
steps:
- name: Checkout code
uses: actions/checkout@v3

- name: Set up CMake
uses: lukka/get-cmake@v3.31.0

# Run the setup script to install dependencies
- name: Run setup script
run: |
chmod +x build-utils/pipe_setup.sh
./build-utils/pipe_setup.sh

- name: Download build artifacts
uses: actions/download-artifact@v3
with:
name: build-artifacts

- name: Run Executable
run: ./build-utils/run_check.sh
56 changes: 56 additions & 0 deletions build-utils/pipe_setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/bin/bash

# Update the package list
sudo apt-get update

# Install required dependencies
sudo apt-get install -y \
build-essential \
cmake \
xorg-dev \
libsdl2-dev \
clang-tidy \
clang-format \
git \
pre-commit \
python3 \
lcov

sudo apt-get install -y software-properties-common wget build-essential cmake xorg-dev libsdl2-dev git pre-commit python3 lcov

# Add the official LLVM APT repository for the latest Clang versions
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 18 # Install LLVM version 18

# Verify installation paths
CLANG_FORMAT_PATH=$(find /usr /lib /opt -name clang-format-18 | head -n 1)
CLANG_TIDY_PATH=$(find /usr /lib /opt -name clang-tidy-18 | head -n 1)

# Update alternatives to point to LLVM tools
if [ -f "$CLANG_FORMAT_PATH" ]; then
echo "Setting up clang-format version 18..."
sudo update-alternatives --install /usr/bin/clang-format clang-format "$CLANG_FORMAT_PATH" 100
else
echo "Error: clang-format-18 binary not found at $CLANG_FORMAT_PATH"
fi

if [ -f "$CLANG_TIDY_PATH" ]; then
echo "Setting up clang-tidy version 18..."
sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy "$CLANG_TIDY_PATH" 100
else
echo "Error: clang-tidy-18 binary not found at $CLANG_TIDY_PATH"
fi

# Echo installed software versions
echo "Installed software versions:"
echo "--------------------------------"
echo "gcc version: $(gcc --version | head -n 1)"
echo "cmake version: $(cmake --version | head -n 1)"
echo "clang-tidy version: $(clang-tidy --version | head -n 1)"
echo "clang-format version: $(clang-format --version | head -n 1)"
echo "git version: $(git --version)"
echo "pre-commit version: $(pre-commit --version)"
echo "python3 version: $(python3 --version)"
echo "lcov version: $(lcov --version | head -n 1)"
echo "--------------------------------"
Loading