Add support for generating signed sysexts

[danzatt]

Sign published sysexts

Use systemd-repart for creating signed DDI sysexts. The keys are expected to be stored in GitHub secrets SYSEXT_PRIVATE_KEY and SYSEXT_CERTIFICATE.

Note/TODO: this needs additional changes in the GH repo setup, there needs to be a signing key and certificate in GH secrets. Additionally, the signing certificate should be published as a release, so that consumers can use it to verify the sysexts. This can be automated through new GH actions, so that the signing key is would be generated and uploaded right in GH actions.

How to use

The following butane config can be used for testing (pulls wasmedge sysext as an example, puts signing certificate into /etc/verity.d and forces sysext signature verification using service dropin). For the sysext verification to work, you need to recompile kernel with CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG (implemented in a separate PR flatcar/scripts#3162).

Note: it uses releases from my fork of sysext-bakery

variant: flatcar
version: 1.0.0

storage:
  files:
    - path: /opt/extensions/wasmedge-0.14.1-x86-64.raw
      mode: 0420
      contents:
        source: https://github.com/danzatt/sysext-bakery/releases/download/wasmedge-0.14.1/wasmedge-0.14.1-x86-64.raw
    - path: /etc/sysupdate.wasmedge.d/wasmedge.conf
      contents:
        source: https://extensions.flatcar.org/extensions/wasmedge.conf
    - path: /etc/verity.d/sysext-bakery.crt
      mode: 0420
      contents:
        source: https://github.com/danzatt/sysext-bakery/releases/download/signing-key/sysext.crt
  links:
    - target: /opt/extensions/wasmedge-0.14.1-x86-64.raw
      path: /etc/extensions/wasmedge.raw
      hard: false
systemd:
  units:
    - name: systemd-sysext.service
      enabled: true
      dropins:
        - name: force-signature.conf
          contents: |
            [Service]
            TimeoutStartSec=10
            ExecStart=
            ExecStart=systemd-sysext refresh --image-policy="root=verity+signed+absent:usr=verity+signed+absent"
            ExecReload=
            ExecReload=systemd-sysext refresh --image-policy="root=verity+signed+absent:usr=verity+signed+absent"
            [Unit]
            JobRunningTimeoutSec=5

Testing done

[Describe the testing you have done before submitting this PR. Please include both the commands you issued as well as the output you got.]

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[pothos]

I think we should make signing opt-in, so wrap the the direct invocations in one if branch?

[pothos]

This could also reuse the exported repart flag options env var.
FYI: I also noticed problems with erofs and have filed a PR to fix this and if merged earlier you might need to adapt the settings when rebasing here #176

[danzatt]

I think we should make signing opt-in, so wrap the the direct invocations in one if branch?

I've just modified the PR to include both raw squashfs and signed DDI in the release (see releases in my fork).

[pothos]

I think we should publish the cert as release artifact - and maybe include it in the sysupdate definition?

[danzatt]

Yes, we definitely should publish it as a release. However, I'm not so sure about updating it using sysupdate. If I understand correctly, sysupdate cannot cryptographically verify the integrity of the update (besides using HTTPS). So it could potentially be used to inject a malicious certificate during sysupdate. Maybe we could ship up-to date sysext bakery certificate in OS image (e.g. in /opt) and the user would just symlink the cert to /etc/verity.d to trust it. This way the bakery cert would be updated with OS update (that is verified using Flatcar release key).

[pothos]

Sysupdate has an option to verify with GPG (which is recommended) but anyway, this then would have to be part of the Ignition config. I think for now we can also provision the cert via Ignition. If the cert doesn't change it sounds ok to have it in the OS but if it's expiring every now and then, having it in the OS might be more complex for managing the lifecycle?

[jepio]

does systemd-repart support pkcs11 tokens? that would be more secure than making the private key accessible to github actions workers

[danzatt]

Yes, it does. I've even tested it with Keyvault PKCS11 token. I'll try to figure out how to use keyvault from github actions.

[danzatt]

I've just reworked the PR to use PKCS11 token. I've also included script to set up Azure keyvault and managed identity for GiHub Action.

[danzatt]

I've just updated the PR. It now:

• builds both signed DDI and unsigned raw image (as before)
  • there are two sysupdate files, one for the signed and one for the raw file
• the bakery CI will now run inside a container based on Ubuntu 24.10
  • Github runner only offers 24.04, which has too old systemd-repart (without PKCS support)
  • the docker image will contain prebuild Keyvault PKCS11 token
  • I had to pull off some tricks to make it work, because bakery itself runs containers (i.e. Docker-in-Docker)
• sysexts are signed using Azure Keyvault via PKCS11 token
  • there is a script (lib/setup_azure_keyvault.sh) to set up the Keyvault, managed identity for GitHub Actions and permissions

[chewi]

Are you sure we actually need Azure CLI? This is normally only needed when signing manually. In CI, you're more likely to use a managed identity or a token set in an environment variable. We currently used a managed identity for signing the boot binaries in CI. You may see errors about the az command not being found, but that's only because it tries that before other mechanisms.

[pothos]

Suggested change

	# Note: We didn't chown to root:root, meaning that the file ownership is left as is
	# Note: We didn't chown to root:root, meaning that the file ownership is left as is with btrfs

[pothos]

• I had to pull off some tricks to make it work, because bakery itself runs containers (i.e. Docker-in-Docker)

Maybe unpriv. Podman could help here because it can be nested many times (with the right invocation options).

[pothos]

Suggested change

	SYSTEMD_REPART_MKFS_OPTIONS_EROFS="-zlz4hc,12 -C65536 -Efragments,ztailpacking -Uclear --all-root"
	SYSTEMD_REPART_MKFS_OPTIONS_EROFS="-zlz4hc,12 -C65536 -Efragments,ztailpacking -U00000000-0000-0000-0000-000000000000 --all-root"

From #181

[danzatt]

Are you sure we actually need Azure CLI? This is normally only needed when signing manually. In CI, you're more likely to use a managed identity or a token set in an environment variable. We currently used a managed identity for signing the boot binaries in CI. You may see errors about the az command not being found, but that's only because it tries that before other mechanisms.

@chewi We need it because the job isn't running on an Azure VM with attached managed identity (like in our CI). As the job is running in GitHub it's not possible to directly assign an Azure identity to the job/worker. Instead I configure the managed identity to trust this github repo to provide federated identity to Azure (I basically followed this guide https://learn.microsoft.com/en-us/azure/developer/github/connect-from-azure-openid-connect). Azure then gives the github action a one-time token to login into the managed identity. The PKCS11 token than uses az to authenticate (using the AzureCliCredential).

[chewi]

I figured managed identities wouldn't be possible, but I thought environment variables might be. They are, but service principal secrets are not recommended. However! I chatted with Copilot, and it has told me that if azure-keyvault-pkcs11 adds support for workload identities (one extra line), then it should work without Azure CLI. The only clunky part is you have to write the token out to a file first. It admits this part could be better, but it's still preferable over installing Azure CLI.

Azure::Identity::ChainedTokenCredential::Sources{
    std::make_shared<Azure::Identity::EnvironmentCredential>(),
    std::make_shared<Azure::Identity::WorkloadIdentityCredential>(),
    getClientSecretCredential(),
    std::make_shared<Azure::Identity::AzureCliCredential>(),
    std::make_shared<Azure::Identity::ManagedIdentityCredential>()});

permissions:
  id-token: write
  contents: read

jobs:
  authenticate:
    runs-on: ubuntu-latest
    steps:
      - name: Request OIDC token and save to file
        id: oidc-token
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const token = await github.actions.getIDToken();
            fs.writeFileSync('/tmp/azure-oidc-token', token);
            return '/tmp/azure-oidc-token';
      - name: Set AZURE_FEDERATED_TOKEN_FILE
        run: echo "AZURE_FEDERATED_TOKEN_FILE=${{ steps.oidc-token.outputs.result }}" >> $GITHUB_ENV

I'm also wondering whether you really need a Ubuntu image for this. The Flatcar SDK already has everything needed... except Docker? I'm confused about why that is needed though.