Build our own mongodb image#2366
Conversation
Hello benzekrimaha,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Incorrect fix versionThe
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
Please check the |
Incorrect fix versionThe
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
Please check the |
5964658 to
8d6260f
Compare
| runs-on: ubuntu-24.04 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: |
There was a problem hiding this comment.
no need for multiple mongodb builds : we have multiple zenko branches, and just one mongodb per zenko branch
There was a problem hiding this comment.
both mongo 8.0 and 8.2 are still committed : this is not needed
69ce420 to
d94b364
Compare
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
| if [[ "${IMAGE_NAME}" == "mongodb-sharded" ]]; then | ||
| MAJOR_MINOR=$(echo "${VERSION}" | cut -d. -f1-2) | ||
| CONTEXT="solution-base/images/mongodb-sharded/${MAJOR_MINOR}/debian-12" | ||
| else | ||
| MAJOR=$(echo "${VERSION}" | cut -d. -f1) | ||
| CONTEXT="solution-base/images/mongodb-sharded/${IMAGE_NAME}/${MAJOR}/debian-12" | ||
| fi |
There was a problem hiding this comment.
we don't need to store multiple versions of either of these components in the repo : so we can choose to organize in the exact way we want, and keep this much simpler.
→ choose in which folder we store them, in a consistent fashion, e.g. either store each in a folder which has the same name as the image ; or just /<major_minor> if we want to avoid mismatch in case of upgrade
There was a problem hiding this comment.
I removed the version-based folder branching and standardized the Mongo image layout to a single consistent structure per image, which simplifies context resolution and avoids mismatch risks during upgrades.
| local deps_key=$1 | ||
| local prefix=$2 |
There was a problem hiding this comment.
make deps_key and prefix match, for simplicity?
| cp -r /tmp/bitnami-containers/bitnami/mongodb-sharded/8.2/debian-12/prebuildfs solution-base/images/mongodb-sharded/8.2/debian-12/ | ||
| cp -r /tmp/bitnami-containers/bitnami/mongodb-sharded/8.2/debian-12/rootfs solution-base/images/mongodb-sharded/8.2/debian-12/ |
There was a problem hiding this comment.
not correct, this would keep "removed" files
There was a problem hiding this comment.
Agreed and fixed. I removed copy-based vendoring instructions because they do not handle upstream deletions safely.
| # Repeat for other images as needed | ||
|
|
||
| rm -rf /tmp/bitnami-containers | ||
| git diff # review changes |
There was a problem hiding this comment.
"copying" is intrinsically dangerous and complex, as it
- automatically discards all changes that could have been made to the charts
- starts from scratch each time, without considering the history: i.e. the common ancestor between the "old" and "new" code
These problems are already solved by Git, so may be best to use Git here.
In particular, git subtree [1] allows to easily merge a "remote" repository in a remote one. This kind of approach would mean we don't copy and diff, but instead perform a proper git merge.
The drawback is that this command is probably not as common for everyone ; but the specific usage can be documented here, and/or wrapped in a makefile (similar to what is done for mongo chart)
Going in to more details, there is an extra subtlety here because we don't want the full repo, but only a subset. There are solutions all over the internet, but it requires a few more commands: c.f. [2]
# Add the remote repository
$ git remote add bitnami-containers https://github.com/bitnami/containers.git
$ git fetch bitnami-containers
# Create a branch containing only the subfolders of the remote repo we are interested in
$ git checkout bitnami-containers/main
$ git subtree split -b bitnami-upstream -P bitnami/mongodb-sharded/8.2/debian-12 -P bitnami/mongodb-exporter/0/debian-12
# Back to our "dev" branch, merge the subtree
$ git checkout -b improvement/ZENKO-xxx
# First time: just "add" the subtree, without committing every change
$ git subtree add -P solution-base/images bitnami-upstream --squash
# Next times: merge upstream, to consider both our changes & bitnamis. Not sure we will want to squash changes (if we see only relevant changes, may be nice to see them individually in history), but not the most important thing anyway
$ git subtree merge -P solution-base/images bitnami-upstream --squash
Note:
- git subtree is part of git "contribs", but not installed by default. So need to install it, c.f. [3]
$ cd $(brew --prefix)/share/git-core/contrib/subtree $ make $ make prefix=$(brew --prefix)/opt/git install - since bitnami repo is large,
git subtree splitcan be quite slow if the whole history was retrieved. It may be much faster if the whole repository is not fetched, not sure if that works though... - The sub-path is making git subtree more complex to build the upstream branch or falling back to advanced git commands instead should be considered, c.f. [4] : i.e. basically we use git read-tree the first time, and followup merge upstream with git diff | git apply.
[1] https://manpages.debian.org/testing/git-man/git-subtree.1.en.html
[2] https://medium.com/@icheko/use-a-subfolder-from-a-remote-repo-in-another-repo-using-git-subtree-98046f33ca40
[3] https://codeengineered.com/blog/how-to-install-git-subtree/
[4] https://stackoverflow.com/questions/23937436/add-subdirectory-of-remote-repo-with-git-subtree
There was a problem hiding this comment.
Just asked Claude, seems to automate quite well (though the code will need to be reworked to also handle the list of containers to sync in the makefile/build script)
BITNAMI_REMOTE := bitnami-containers
BITNAMI_REPO := https://github.com/bitnami/containers.git
create-remote:
@git remote get-url $(BITNAMI_REMOTE) >/dev/null 2>&1 || git remote add $(BITNAMI_REMOTE) $(BITNAMI_REPO)
# Usage: make vendor-sync CHART=mongodb-sharded PREFIX=solution-base/mongodb/charts/mongodb-sharded
vendor-sync: create-remote
git fetch $(BITNAMI_REMOTE) main --depth=1
git subtree split --prefix=bitnami/$(CONTAINER) $(BITNAMI_REMOTE)/main -b vendor/$(CONTAINER)
git subtree merge --prefix=solution-base/images/$(CONTAINER) vendor/$(CONTAINER) --squash
# First-time import variant
vendor-add: create-remote
git fetch $(BITNAMI_REMOTE) main --depth=1
git subtree split --prefix=bitnami/$(CHART) $(BITNAMI_REMOTE)/main -b vendor/$(CONTAINER)
git subtree add --prefix=solution-base/images/$(CONTAINER) vendor/$(CONTAINER) --squash
Then for any container:
make vendor-add CONTAINER=mongodb-sharded
There was a problem hiding this comment.
to clarify the intent here : however we do it, we will make changes to the image, but we need to manage it.
Such in (git) history we need to have something like:
* Initial mport from upstream commit XXXX
* Our own tweaks [for exemple: modify makefile to take mongodb version in argument]
* More tweaks [for exemple: bump baseline image hash]
* ...
* Merge upstream commit YYY
* ...
So we need to make sure we actually "merge" upstream with our own changes.
For the charts, we ended using patch files - works, but not very efficient: but we need to do something in this case as well...
There was a problem hiding this comment.
I switched documentation and workflow to a Git-based vendoring approach (git subtree) so upstream updates are merged properly and history is preserved.
| 1. Update the Dockerfile under the version directory (e.g. | ||
| `8.0/debian-12/Dockerfile`): | ||
| - `COMPONENTS+=("mongodb-8.0.14-0-linux-...")` (update component version) | ||
| - `MONGODB_TARBALL="mongodb-linux-aarch64-ubuntu2204-8.0.14.tgz"` (arm64) | ||
| - `org.opencontainers.image.version="8.0.14"` (label) | ||
| - `APP_VERSION="8.0.14"` (env) |
There was a problem hiding this comment.
should be a single ARG (e.g. "MONGODB_VERSION") in the Dockerfile.....whose value we retrieve from the deps.yaml, and is passed to docker build
There was a problem hiding this comment.
The Dockerfile now uses one MONGODB_VERSION build arg, and CI passes its value from deps, so version updates are centralized and consistent.
|
|
||
| ### Bumping the MongoDB version in the image | ||
|
|
||
| 1. Edit the Dockerfile for the target major line |
There was a problem hiding this comment.
redundant with the doc in images/mongodb-shared/README.md
either keep both here, or just use a "pointer" to the other file here
(and you see the cost of redundancy: you forgot to update this one to match your latest changes)
There was a problem hiding this comment.
I removed redundant step-by-step image bump content and left a pointer to the canonical mongo image readme to avoid drift.
| - 'w/**' | ||
| - main | ||
| paths: | ||
| - 'solution-base/images/mongodb-sharded/**' |
There was a problem hiding this comment.
also depends on deps.yaml, and possibly other build scripts
There was a problem hiding this comment.
I removed the separate path-filtered Mongo workflow and moved Mongo image builds into the main pipeline, so trigger drift on deps/build script changes is no longer an issue
There was a problem hiding this comment.
why use another mongodb workflow?
- since builds are cached and the tag is computed, it does not cost anything to rebuild
- path filtering can be tricky and cause unexpected results
- it duplicates trigger conditions, which can easily get out of sync and create inconsistencies (or bugs) in the jobs actually executed
- it makes it harder to see what's going on overall (when opening a PR/...)
- it prevents using the just built image in the same build, as there is no dependency: so the mongodb image used may not have been built
There was a problem hiding this comment.
Mongo build is now part of the main CI flow with proper job dependencies, so tests use images built in the same run and overall CI behavior is easier to follow.
| TREE_HASH=$(git rev-parse HEAD:"${CONTEXT}") | ||
| SHORT_HASH=${TREE_HASH:0:8} | ||
|
|
||
| TAGS="${{ env.REGISTRY }}/${IMAGE}:${VERSION}" |
There was a problem hiding this comment.
this tags is a moving tag (may change on every commit), should probably not be used...
There was a problem hiding this comment.
The pipeline now builds and consumes immutable tree-hash-tagged images for Mongo components instead of relying on floating semver tags.
| SHORT_HASH=${TREE_HASH:0:8} | ||
|
|
||
| TAGS="${{ env.REGISTRY }}/${IMAGE}:${VERSION}" | ||
| TAGS="${TAGS},${{ env.REGISTRY }}/${IMAGE}:${VERSION}-${SHORT_HASH}" |
There was a problem hiding this comment.
the tag with the "tree-hash" that we built MUST be included in the solution-base ISO (do not depend on a floating tag!) and used when deploying mongo in tests
c.f. how this is done for kafka images (which also have such a tree hash)
There was a problem hiding this comment.
I aligned Mongo handling with the Kafka pattern: the tree-hash image tag is propagated into solution-base build inputs and test deployment paths, so both ISO content and tests use the exact built image.
c452ddf to
f38303d
Compare
francoisferrand
left a comment
There was a problem hiding this comment.
not sure about the target branch now, checking with product.
Either way this PR will need to be rebased
| BITNAMI_REPO := https://github.com/bitnami/containers.git | ||
|
|
||
| # Required: | ||
| # - CONTAINER: destination path under solution-base/images (example: mongodb-sharded/debian-12) |
There was a problem hiding this comment.
CONTAINER is not an argument:
- we know exactly which values are possible and MUST be used
- merging/... should be done at the exact same point, we don't want to mix-and-match different versions of exporter/mongo/os-shell
- git fetch and subtree split are somewhat costly: we should avoid doing them multiple times
typically this means there should be multiple rules : creating remote, fetching/updating "vendor" branch (split subtree), then one rule for each image
There was a problem hiding this comment.
Addressed. I removed free-form CONTAINER/BITNAMI_PREFIX usage and moved all allowed values/ref mappings into fixed Makefile rules.
| @test -n "$(CONTAINER)" || (echo "CONTAINER is required"; exit 1) | ||
| @test -n "$(BITNAMI_PREFIX)" || (echo "BITNAMI_PREFIX is required"; exit 1) | ||
| git fetch $(BITNAMI_REMOTE) main --depth=1 | ||
| git subtree split --prefix=$(BITNAMI_PREFIX) $(BITNAMI_REMOTE)/main -b vendor/$(CONTAINER) |
There was a problem hiding this comment.
this does not work today : main does not contain bitnami/mongodb-sharded/8.0 → should we have something else instead of main ?
- may be ok once we move to 8.2, but not just now...
- how does it work when we move from 8.0 to 8.2 ?
There was a problem hiding this comment.
mongodb-sharded is now split from a pinned upstream commit that still contains bitnami/mongodb-sharded/8.0/debian-12, while mongodb-exporter and os-shell are still split from upstream main. This is now explicit in Makefile.
|
|
||
| # Required: | ||
| # - CONTAINER: destination path under solution-base/images (example: mongodb-sharded/debian-12) | ||
| # - BITNAMI_PREFIX: source path in bitnami/containers repository |
There was a problem hiding this comment.
same BITNAMI_PREFIX is not an input, the goal of a makefile is to set these values instead of relying on some external knowledge...
There was a problem hiding this comment.
Prefixes and refs are now defined in the Makefile itself, not passed from CLI arguments. See Makefile.
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 |
| ### Re-vendoring upstream Bitnami scripts | ||
|
|
||
| When upstream Bitnami script changes need to be pulled in, | ||
| compare `bitnami/containers` and copy the updated `prebuildfs/` / `rootfs/` directories |
There was a problem hiding this comment.
| compare `bitnami/containers` and copy the updated `prebuildfs/` / `rootfs/` directories | |
| compare `bitnami/containers` and merge upstream changes to `prebuildfs/` / `rootfs/` directories |
There was a problem hiding this comment.
already fixed upstream, should be dropped
e3eb031 to
41113a3
Compare
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 |
| BITNAMI_REMOTE := bitnami-containers | ||
| BITNAMI_REPO := https://github.com/bitnami/containers.git | ||
|
|
||
| BITNAMI_MONGODB_SHARDED_REF := 48a109547d39cd8cf8a5d4058d832ecb5844829e |
There was a problem hiding this comment.
kind of unexpected to have just a commit hash here : I know this is needed because the 8.0 images have been removed from the upstream repo since then, but worth adding a comment here to explain this is the "latest" commit on main which has the 8.0 image
There was a problem hiding this comment.
I added a comment explaining that the pinned hash is the latest upstream commit known to still contain bitnami/mongodb-sharded/8.0/debian-12.
| tag: 8.0.13-debian-12-r0 | ||
| sourceRegistry: ghcr.io | ||
| image: scality/zenko/mongodb-sharded | ||
| tag: "8.0.13" |
There was a problem hiding this comment.
nit: we are removing the debian-12 "information" from the image tag. Is it because we consider it useless/redundant, or just to simplify script?
There was a problem hiding this comment.
Intentional. We keep distro information in the source layout and Dockerfile base digest, and use clean semantic app version tags in deps.yaml for consistency with build args and immutable tree-hash tagging. If preferred, we can restore distro suffixes, but current flow does not rely on them.
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v5 |
There was a problem hiding this comment.
same for a few others below
|
@benzekrimaha This needs to be rebased on 2.14 and change target branch to 2.14 |
c75dad0 to
f9af93c
Compare
f9af93c to
897ef87
Compare
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
f3b4967 to
9bae24f
Compare
2d49647 to
21c8055
Compare
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
6485054 to
26e42e5
Compare
Fork the Bitnami mongodb-sharded image sources into the Zenko repository so we no longer depend on Bitnami publishing versioned mongodb-sharded images. Vendored under solution-base/images are mongodb-sharded, mongodb-exporter, and os-shell sources, with Git-based vendoring documentation for future upstream syncs. Issue: ZENKO-5110
Build the vendored mongodb-sharded, mongodb-exporter, and os-shell images inside the end2end workflow and publish immutable tree-hash tags to ghcr.io/scality/zenko/. Issue: ZENKO-5110
Point Zenko MongoDB dependencies to the vendored GHCR images and make both kind deployments and ISO manifests consume the immutable tree-hash tags produced by CI. Issue: ZENKO-5110
Restore mongodump and bsondump in the custom mongodb-sharded image so artifact archival keeps working with the vendored image. Issue: ZENKO-5110
26e42e5 to
fc4409d
Compare
|
/approve |
Build failedThe build for commit did not succeed in branch improvement/ZENKO-5110-build-our-own-bitnami-image The following options are set: approve |
Issue: ZENKO-5110