Use mapstructure to (de-)serialize internal metadata; move metadata processing from specific stores to general place by felixfontein · Pull Request #2120 · getsops/sops

felixfontein · 2026-03-23T21:42:31Z

Right now, metadata is handled quite strangely when flattened (by going through JSON), and all stores somehow have to deal with metadata. This PR cleans this up as follows:

Use mapstructure to convert to/from metadata and coerce types.
Convert metadata to/from sops.TreeBranch, and provide three options for handling flattening (none; keep sops toplevel map; flatten completely).
Properly handle errors during flattening and unflattening. Right now you can panic SOPS and overwrite values by playing around with flattened metadata.
Remove old public APIs for dealing with internal metadata and (un-)flattening.

While this is a breaking change, I think this is still OK:

The APIs that are broken are internal ones. To from our promise, they're not public and it's not a breaking change.
They should only be used by store implementations, which are all part of SOPS itself. This means that even though some other programs use parts of SOPS as a library that aren't public, it is unlikely that they directly use these parts of SOPS that are changed by this PR.

This is related to #1401, #1338, #1046, #1009, which were earlier attempts to introduce mapstructure.

felixfontein · 2026-05-03T10:03:36Z

Rebased to resolve conflict.

sabre1041

LGTM

Signed-off-by: Felix Fontein <felix@fontein.de>

Co-authored-by: Hidde Beydals <hiddeco@users.noreply.github.com> Signed-off-by: Felix Fontein <felix@fontein.de>

Signed-off-by: Felix Fontein <felix@fontein.de>

felixfontein · 2026-05-07T20:04:17Z

(Rebased against current main.)

Lower-case the doc comment on `metadataFromInternal` to match the unexported function name, wrap it to ~80 columns, and terminate it with a period. Replace `i -= 1` with `i--` in the `MetadataFlattenFull` branch of `ExtractMetadata` to match the incrementor style used in the loop body of the other branch. Signed-off-by: Hidde Beydals <hidde@hhh.computer>

The previous implementation pre-allocated `result` to `len(slice)` and wrote each non-comment value at its source index. When the input contained comments, the slots at those indices were left as `nil`, producing arrays with embedded nil holes that downstream `mapstructure` decoding would silently accept. Switch to an `append`-based compaction loop, mirroring `flattenDescendArray` in `flatten.go`, so that comments are dropped cleanly and indices on the output remain contiguous. No metadata array carries comments today, so this is a latent fix rather than a behaviour change for any current input. Signed-off-by: Hidde Beydals <hidde@hhh.computer>

The dotenv package re-declared `SopsPrefix` with the same value as `stores.SopsPrefix`. Its only consumer was the package-local `HasSopsTopLevelKey`. Use `stores.SopsPrefix` directly so there is one declaration of the prefix, and remove the duplicate doc comment which also carried over a `metadatada` typo previously fixed in `metadata.go` (`04318bd2`). Signed-off-by: Hidde Beydals <hidde@hhh.computer>

`EncodeNewLines` and `DecodeNewLines` escaped or unescaped `\n` for every string value in a metadata map, and were applied to the flattened metadata before it was handed to the INI/dotenv writers. After the move to a single `SerializeMetadata` pipeline they are no longer called: each store handles newline escaping at the file boundary itself. Signed-off-by: Hidde Beydals <hidde@hhh.computer>

`convert` previously returned a misleading "Incomplete list" error when called on an empty `node`, because the indices branch ran with `minValue == math.MaxInt`. Short-circuit the all-empty case to return an empty `sops.TreeBranch` instead, which is the natural identity for `unflattenTreeBranch(sops.TreeBranch{})`. Signed-off-by: Hidde Beydals <hidde@hhh.computer>

Signed-off-by: Felix Fontein <felix@fontein.de>

felixfontein · 2026-05-08T05:33:11Z

@hiddeco @sabre1041 thanks a lot for reviewing and improving the PR!

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [getsops/sops](https://github.com/getsops/sops) | minor | `v3.12.2` → `v3.13.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>getsops/sops (getsops/sops)</summary> ### [`v3.13.0`](https://github.com/getsops/sops/releases/tag/v3.13.0) [Compare Source](getsops/sops@v3.12.2...v3.13.0) #### Installation To install `sops`, download one of the pre-built binaries provided for your platform from the artifacts attached to this release. For instance, if you are using Linux on an AMD64 architecture: ```shell # Download the binary curl -LO https://github.com/getsops/sops/releases/download/v3.13.0/sops-v3.13.0.linux.amd64 # Move the binary in to your PATH mv sops-v3.13.0.linux.amd64 /usr/local/bin/sops # Make the binary executable chmod +x /usr/local/bin/sops ``` ##### Verify checksums file signature The checksums file provided within the artifacts attached to this release is signed using [Cosign](https://docs.sigstore.dev/cosign/overview/) with GitHub OIDC. To validate the signature of this file, run the following commands: ```shell # Download the checksums file, certificate and signature curl -LO https://github.com/getsops/sops/releases/download/v3.13.0/sops-v3.13.0.checksums.txt curl -LO https://github.com/getsops/sops/releases/download/v3.13.0/sops-v3.13.0.checksums.sigstore.json # Verify the checksums file cosign verify-blob sops-v3.13.0.checksums.txt \ --bundle sops-v3.13.0.checksums.sigstore.json \ --certificate-identity-regexp=https://github.com/getsops \ --certificate-oidc-issuer=https://token.actions.githubusercontent.com ``` ##### Verify binary integrity To verify the integrity of the downloaded binary, you can utilize the checksums file after having validated its signature: ```shell # Verify the binary using the checksums file sha256sum -c sops-v3.13.0.checksums.txt --ignore-missing ``` ##### Verify artifact provenance The [SLSA provenance](https://slsa.dev/provenance/v0.2) of the binaries, packages, and SBOMs can be found within the artifacts associated with this release. It is presented through an [in-toto](https://in-toto.io/) link metadata file named `sops-v3.13.0.intoto.jsonl`. To verify the provenance of an artifact, you can utilize the [`slsa-verifier`](https://github.com/slsa-framework/slsa-verifier#artifacts) tool: ```shell # Download the metadata file curl -LO https://github.com/getsops/sops/releases/download/v3.13.0/sops-v3.13.0.intoto.jsonl # Verify the provenance of the artifact slsa-verifier verify-artifact <artifact> \ --provenance-path sops-v3.13.0.intoto.jsonl \ --source-uri github.com/getsops/sops \ --source-tag v3.13.0 ``` #### Container Images The `sops` binaries are also available as container images, based on Debian (slim) and Alpine Linux. The Debian-based container images include any dependencies which may be required to make use of certain key services, such as GnuPG, AWS KMS, Azure Key Vault, and Google Cloud KMS. The Alpine-based container images are smaller in size, but do not include these dependencies. These container images are available for the following architectures: `linux/amd64` and `linux/arm64`. ##### GitHub Container Registry - `ghcr.io/getsops/sops:v3.13.0` - `ghcr.io/getsops/sops:v3.13.0-alpine` ##### Quay.io - `quay.io/getsops/sops:v3.13.0` - `quay.io/getsops/sops:v3.13.0-alpine` ##### Verify container image signature The container images are signed using [Cosign](https://docs.sigstore.dev/cosign/overview/) with GitHub OIDC. To validate the signature of an image, run the following command: ```shell cosign verify ghcr.io/getsops/sops:v3.13.0 \ --certificate-identity-regexp=https://github.com/getsops \ --certificate-oidc-issuer=https://token.actions.githubusercontent.com \ -o text ``` ##### Verify container image provenance The container images include [SLSA provenance](https://slsa.dev/provenance/v0.2) attestations. For more information around the verification of this, please refer to the [`slsa-verifier` documentation](https://github.com/slsa-framework/slsa-verifier#containers). #### Software Bill of Materials The Software Bill of Materials (SBOM) for each binary is accessible within the artifacts enclosed with this release. It is presented as an [SPDX](https://spdx.dev/) JSON file, formatted as `<binary>.spdx.sbom.json`. #### What's Changed - build(deps): Bump google.golang.org/grpc from 1.79.1 to 1.79.3 by [@dependabot](https://github.com/dependabot)\[bot] in [#2110](getsops/sops#2110) - build(deps): Bump the ci group with 3 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#2117](getsops/sops#2117) - build(deps): Bump the ci group with 3 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#2126](getsops/sops#2126) - build(deps): Bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 by [@dependabot](https://github.com/dependabot)\[bot] in [#2133](getsops/sops#2133) - build(deps): Bump docker/login-action from 4.0.0 to 4.1.0 in the ci group by [@dependabot](https://github.com/dependabot)\[bot] in [#2139](getsops/sops#2139) - Improve error message for top-level arrays by [@kushal9897](https://github.com/kushal9897) in [#2138](getsops/sops#2138) - build(deps): Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.96.0 to 1.97.3 by [@dependabot](https://github.com/dependabot)\[bot] in [#2142](getsops/sops#2142) - docs: add get key permission requirement for Azure Key Vault when version is omitted by [@Hanashiko](https://github.com/Hanashiko) in [#2122](getsops/sops#2122) - build(deps): Bump actions/upload-artifact from 7.0.0 to 7.0.1 in the ci group by [@dependabot](https://github.com/dependabot)\[bot] in [#2149](getsops/sops#2149) - Use prefixed and truncated key hash instead of location as GPG agent cache key by [@felixfontein](https://github.com/felixfontein) in [#2145](getsops/sops#2145) - Drop support for Go 1.24 by [@felixfontein](https://github.com/felixfontein) in [#2141](getsops/sops#2141) - build(deps): Bump go.opentelemetry.io/otel/sdk from 1.40.0 to 1.43.0 by [@dependabot](https://github.com/dependabot)\[bot] in [#2146](getsops/sops#2146) - build(deps): Bump the go group across 1 directory with 21 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#2144](getsops/sops#2144) - Add SOPS\_GCP\_KMS\_ENDPOINT and SOPS\_GCP\_KMS\_UNIVERSE\_DOMAIN env vars by [@DnR-iData](https://github.com/DnR-iData) in [#2114](getsops/sops#2114) - Preserve YAML inline comments through encrypt/edit roundtrip by [@knowald](https://github.com/knowald) in [#2131](getsops/sops#2131) - Update release for cosign v3 by [@sabre1041](https://github.com/sabre1041) in [#2082](getsops/sops#2082) - build(deps): Bump the ci group with 3 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#2151](getsops/sops#2151) - build(deps): Bump the go group with 12 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#2152](getsops/sops#2152) - Update broken link in README by [@manute](https://github.com/manute) in [#2153](getsops/sops#2153) - sops exec-file: make sure to set GID to correct value instead of UID by [@felixfontein](https://github.com/felixfontein) in [#2154](getsops/sops#2154) - sops exec-file: improve error handling, reject non-local paths in `--filename` by [@felixfontein](https://github.com/felixfontein) in [#2155](getsops/sops#2155) - Obtain `--indent` global flag value and pass it to store configs by [@skirino](https://github.com/skirino) in [#2156](getsops/sops#2156) - build(deps): Bump the go group with 3 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#2159](getsops/sops#2159) - build(deps): Bump goreleaser/goreleaser-action from 7.1.0 to 7.2.1 in the ci group by [@dependabot](https://github.com/dependabot)\[bot] in [#2158](getsops/sops#2158) - Update all Go dependencies with 'go get -t -u ./...' by [@felixfontein](https://github.com/felixfontein) in [#2165](getsops/sops#2165) - Allow to have space-separated keys in SOPS\_AGE\_KEY by [@felixfontein](https://github.com/felixfontein) in [#2086](getsops/sops#2086) - HC Vault: add allowlist support for acceptable HC Vault URLs by [@felixfontein](https://github.com/felixfontein) in [#2164](getsops/sops#2164) - build(deps): Bump the go group with 5 updates by [@dependabot](https://github.com/dependabot)\[bot] in [#2171](getsops/sops#2171) - build(deps): Bump github/codeql-action from 4.35.2 to 4.35.3 in the ci group by [@dependabot](https://github.com/dependabot)\[bot] in [#2173](getsops/sops#2173) - Use mapstructure to (de-)serialize internal metadata; move metadata processing from specific stores to general place by [@felixfontein](https://github.com/felixfontein) in [#2120](getsops/sops#2120) - Release 3.13.0 by [@felixfontein](https://github.com/felixfontein) in [#2176](getsops/sops#2176) #### New Contributors - [@kushal9897](https://github.com/kushal9897) made their first contribution in [#2138](getsops/sops#2138) - [@Hanashiko](https://github.com/Hanashiko) made their first contribution in [#2122](getsops/sops#2122) - [@DnR-iData](https://github.com/DnR-iData) made their first contribution in [#2114](getsops/sops#2114) - [@knowald](https://github.com/knowald) made their first contribution in [#2131](getsops/sops#2131) - [@manute](https://github.com/manute) made their first contribution in [#2153](getsops/sops#2153) - [@skirino](https://github.com/skirino) made their first contribution in [#2156](getsops/sops#2156) **Full Changelog**: <getsops/sops@v3.12.2...v3.13.0> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).

felixfontein requested a review from a team March 23, 2026 21:52

felixfontein force-pushed the mapstructure branch 3 times, most recently from 97fea11 to 0f3238f Compare April 21, 2026 18:49

hiddeco reviewed Apr 21, 2026

View reviewed changes

felixfontein requested a review from hiddeco April 21, 2026 20:19

felixfontein force-pushed the mapstructure branch from 2513d1c to f7e7a7d Compare April 29, 2026 18:23

felixfontein requested a review from a team April 29, 2026 19:05

felixfontein force-pushed the mapstructure branch from f7e7a7d to c4f3a2d Compare May 3, 2026 10:03

felixfontein force-pushed the mapstructure branch from c4f3a2d to e9e3a72 Compare May 3, 2026 18:08

sabre1041 approved these changes May 7, 2026

View reviewed changes

felixfontein and others added 11 commits May 7, 2026 22:01

Use mapstructure for parsing.

84b0ad2

Signed-off-by: Felix Fontein <felix@fontein.de>

Add common code to encode and decode metadata.

3ae1575

Signed-off-by: Felix Fontein <felix@fontein.de>

Remove old flattening code.

4187477

Signed-off-by: Felix Fontein <felix@fontein.de>

Remove old (de-)serialization.

26cf045

Signed-off-by: Felix Fontein <felix@fontein.de>

Make internal metadata type private.

3def883

Signed-off-by: Felix Fontein <felix@fontein.de>

Remove code unrelated to (un-)flattening to other file.

ff1266b

Signed-off-by: Felix Fontein <felix@fontein.de>

Better separation.

a587e03

Signed-off-by: Felix Fontein <felix@fontein.de>

Also move tests.

a07f6d3

Signed-off-by: Felix Fontein <felix@fontein.de>

Replace unflattening algorithm.

de89957

Signed-off-by: Felix Fontein <felix@fontein.de>

Apply first suggestions from Hidde's review.

f42a9c5

Co-authored-by: Hidde Beydals <hiddeco@users.noreply.github.com> Signed-off-by: Felix Fontein <felix@fontein.de>

Fix age key in tests.

f14527d

Signed-off-by: Felix Fontein <felix@fontein.de>

felixfontein force-pushed the mapstructure branch from e9e3a72 to f14527d Compare May 7, 2026 20:01

hiddeco added 3 commits May 7, 2026 22:29

hiddeco approved these changes May 7, 2026

View reviewed changes

felixfontein commented May 7, 2026

View reviewed changes

Comment thread stores/metadata.go Outdated

Comment thread stores/metadata.go Outdated

felixfontein commented May 7, 2026

View reviewed changes

Comment thread stores/metadata.go Outdated

hiddeco added 2 commits May 7, 2026 23:00

hiddeco force-pushed the mapstructure branch from 46520b8 to 9061d01 Compare May 7, 2026 21:08

Re-add some of Hidde's comments.

a2ab197

Signed-off-by: Felix Fontein <felix@fontein.de>

felixfontein merged commit da120c7 into getsops:main May 8, 2026
15 checks passed

felixfontein deleted the mapstructure branch May 8, 2026 05:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use mapstructure to (de-)serialize internal metadata; move metadata processing from specific stores to general place#2120

Use mapstructure to (de-)serialize internal metadata; move metadata processing from specific stores to general place#2120
felixfontein merged 17 commits into
getsops:mainfrom
felixfontein:mapstructure

felixfontein commented Mar 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

felixfontein commented May 3, 2026

Uh oh!

sabre1041 left a comment

Uh oh!

felixfontein commented May 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

felixfontein commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

felixfontein commented Mar 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

felixfontein commented May 3, 2026

Uh oh!

sabre1041 left a comment

Choose a reason for hiding this comment

Uh oh!

felixfontein commented May 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

felixfontein commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants