Skip to content

Commit e53afec

Browse files
PaperMtnPaperMtn
authored
Merge pull request #14 from PaperMtn/release/3.1.0
Release/3.1.0
2 parents d06cd2d + f4f2ca3 commit e53afec

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+4024
-1403
lines changed

‎.github/workflows/docker_build_test.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
name: Build and Test Docker Image
2+
3+
on:
4+
push:
5+
6+
env:
7+
TEST_TAG: papermountain/gitlab-watchman:test
8+
9+
jobs:
10+
docker:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- name: Set up Docker Buildx
14+
uses: docker/setup-buildx-action@v3
15+
16+
- name: Build
17+
uses: docker/build-push-action@v6
18+
with:
19+
load: true
20+
tags: ${{ env.TEST_TAG }}
21+
22+
- name: Inspect
23+
run: |
24+
docker image inspect ${{ env.TEST_TAG }}
25+
26+
- name: Test
27+
run: |
28+
docker run --rm ${{ env.TEST_TAG }} --version
29+
docker run --rm ${{ env.TEST_TAG }} --help

.github/workflows/dockerpublish.yml renamed to .github/workflows/docker_publish.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
name: ci
1+
name: Publish Docker Image
22

33
on:
44
push:
5-
branches: [ master ]
5+
branches: [ master, main ]
66

77
jobs:
88
build:

‎.github/workflows/python_package.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
name: Test Python Package
2+
3+
on:
4+
push:
5+
branches: [ develop, feature/**, release/**, hotfix/** ]
6+
pull_request:
7+
branches: [ develop, feature/**, release/**, hotfix/** ]
8+
9+
jobs:
10+
build-ubuntu:
11+
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
python-version: ["3.10", "3.11", "3.12", "3.13"]
16+
17+
steps:
18+
- uses: actions/checkout@v2
19+
- name: Set up Python ${{ matrix.python-version }}
20+
uses: actions/setup-python@v2
21+
with:
22+
python-version: ${{ matrix.python-version }}
23+
- name: Install dependencies
24+
run: |
25+
pip install poetry
26+
poetry install
27+
- name: Test setup & install
28+
run: |
29+
poetry build
30+
python3 -m pip install dist/*.whl
31+
- name: Test run
32+
run: |
33+
gitlab-watchman --version
34+
gitlab-watchman --help

‎.github/workflows/python_publish.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: Poetry Publish
2+
3+
on:
4+
push:
5+
branches: [ master, main ]
6+
7+
jobs:
8+
deploy:
9+
10+
runs-on: ubuntu-latest
11+
12+
steps:
13+
- uses: actions/checkout@v2
14+
- name: Set up Python
15+
uses: actions/setup-python@v2
16+
with:
17+
python-version: '3.12'
18+
- name: Install dependencies
19+
run: |
20+
pip install poetry
21+
poetry install
22+
poetry config pypi-token.pypi "${{ secrets.PYPI_TOKEN }}"
23+
- name: Publish package
24+
run: poetry publish --build

‎.github/workflows/python_run_tests.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: Run Unit Test via Pytest
2+
3+
on:
4+
push:
5+
6+
jobs:
7+
build:
8+
runs-on: ubuntu-latest
9+
strategy:
10+
matrix:
11+
python-version: ["3.10", "3.11", "3.12", "3.13"]
12+
13+
steps:
14+
- uses: actions/checkout@v3
15+
- name: Set up Python ${{ matrix.python-version }}
16+
uses: actions/setup-python@v4
17+
with:
18+
python-version: ${{ matrix.python-version }}
19+
- name: Install dependencies
20+
run: |
21+
python -m pip install --upgrade pip
22+
pip install poetry
23+
poetry install --with dev
24+
- name: Analysing the code with pylint
25+
run: |
26+
poetry run pylint $(git ls-files '*.py')
27+
continue-on-error: true
28+
- name: Test with pytest
29+
run: |
30+
poetry run coverage run -m pytest -v -s
31+
- name: Generate Coverage Report
32+
run: |
33+
poetry run coverage report -m

‎.github/workflows/pythonpackage.yml‎

Lines changed: 0 additions & 46 deletions
This file was deleted.

‎.github/workflows/pythonpublish.yml‎

Lines changed: 0 additions & 30 deletions
This file was deleted.

‎CHANGELOG.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,28 @@
1+
## [3.1.0] - 2024-11-18
2+
### Added
3+
- Signatures now loaded into memory instead of being saved to disk. This allows for running on read-only filesystems.
4+
- Ability to disable signatures by their ID in the watchman.conf config file.
5+
- These signatures will not be used when running Slack Watchman
6+
- Signature IDs for each signature can be found in the Watchman Signatures repository
7+
- Tests for Docker build
8+
- Enhanced deduplication of findings
9+
- The same match should not be returned multiple times within the same scope. E.g. if a token is found in a commit, it should not be returned multiple times in the same commit.
10+
- All dates are now converted and logged in UTC
11+
- Unit tests added for models and utils
12+
13+
### Changed
14+
- Package management and deployment moved to Poetry
15+
- Docker build process improved using multi-stage builds. The Dockerfile now doesn't contain any unnecessary files, and is much smaller.
16+
- Refactor to separate GitLab client and Watchman processing into modules
17+
- Refactor to implement [python-gitlab](https://python-gitlab.readthedocs.io/) library for GitLab API calls, instead of the custom client used previously.
18+
- This change gives more efficient and easier to read code, is more reliable, and also allows for enhancements to be added more easily in the future.
19+
20+
### Fixed
21+
- Error when searching wiki-blobs
22+
- There would often be failures when trying to find projects or groups associated with blobs. This is now fixed by adding logic to check if the blob is associated with a project or group, and get the correct information accordingly.
23+
- URL encoding for wiki-blobs where the URL contains special characters
24+
- Error when enumerating pages when there is no `X-Total-Pages` header
25+
126
## [3.0.0] - 2023-05-15
227
This major version release brings multiple updates to GitLab Watchman in usability, functionality and behind the scenes improvements.
328
### Added

‎Dockerfile‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,20 @@
11
# syntax=docker/dockerfile:1
2+
FROM python:3.12-slim-bullseye AS builder
3+
WORKDIR /opt/gitlab-watchman
4+
COPY . .
5+
RUN pip install poetry
6+
RUN poetry config virtualenvs.create false && \
7+
poetry install --no-dev && \
8+
poetry build
29

3-
FROM python:3.10
4-
COPY . /opt/gitlab-watchman
10+
FROM python:3.12-slim-bullseye
511
WORKDIR /opt/gitlab-watchman
6-
ENV PYTHONPATH=/opt/gitlab-watchman GITLAB_WATCHMAN_TOKEN="" GITLAB_WATCHMAN_URL=""
7-
RUN pip3 install -r requirements.txt build && \
8-
chmod -R 700 . && \
9-
python3 -m build && \
10-
python3 -m pip install dist/*.whl
12+
COPY --from=builder /opt/gitlab-watchman/dist/*.whl /opt/gitlab-watchman/dist/
13+
COPY --from=builder /opt/gitlab-watchman/pyproject.toml /opt/gitlab-watchman/poetry.lock /opt/gitlab-watchman/
14+
ENV PYTHONPATH=/opt/gitlab-watchman \
15+
GITLAB_WATCHMAN_TOKEN="" \
16+
GITLAB_WATCHMAN_URL=""
17+
RUN pip install dist/*.whl && \
18+
chmod -R 700 .
1119
STOPSIGNAL SIGINT
12-
WORKDIR /opt/gitlab-watchman
1320
ENTRYPOINT ["gitlab-watchman"]

‎README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,18 @@ GitLab Watchman can enumerate potentially useful information from a GitLab insta
5555
### Signatures
5656
GitLab Watchman uses custom YAML signatures to detect matches in GitLab. These signatures are pulled from the central [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures). Slack Watchman automatically updates its signature base at runtime to ensure its using the latest signatures to detect secrets.
5757

58+
#### Suppressing Signatures
59+
You can define signatures that you want to disable when running GitLab Watchman by adding their IDs to the `disabled_signatures` section of the `watchman.conf` file. For example:
60+
61+
```yaml
62+
gitlab_watchman:
63+
disabled_signatures:
64+
- tokens_generic_bearer_tokens
65+
- tokens_generic_access_tokens
66+
```
67+
68+
You can find the ID of a signature in the individual YAML files in [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures).
69+
5870
### Logging
5971
6072
GitLab Watchman gives the following logging options:
@@ -106,6 +118,16 @@ You also need to provide the URL of your GitLab instance.
106118
#### Providing token & URL
107119
GitLab Watchman will get the GitLab token and URL from the environment variables `GITLAB_WATCHMAN_TOKEN` and `GITLAB_WATCHMAN_URL`.
108120

121+
### watchman.conf file
122+
Configuration options can be passed in a file named `watchman.conf` which must be stored in your home directory. The file should follow the YAML format, and should look like below:
123+
```yaml
124+
gitlab_watchman:
125+
disabled_signatures:
126+
- tokens_generic_bearer_tokens
127+
- tokens_generic_access_tokens
128+
```
129+
GitLab Watchman will look for this file at runtime, and use the configuration options from here.
130+
109131
## Installation
110132
You can install the latest stable version via pip:
111133

0 commit comments

Comments
 (0)