HashiCorp Vault authentication renewal

[mafa73]

I have a problem when using Vault through the io.quarkiverse.vault:quarkus-vault dependency in Quarkus 3.22.1

The application starts up fine and I can retrieve a secret, but after about 1 hour it stops working and I get the error:
Failed to set Authorization Bearer io.quarkus.vault.client.VaultClientException: VAULT [SYS (wrapping)] Unwrap' at path 'https://****/v1/sys/wrapping/unwrap' with status 400 errors: wrapping token is not valid or does not exist; this means that the token has already expired (if so you can increase the ttl on the wrapping token) or has been consumed by somebody else (potentially indicating that the wrapping token has been stolen) wrapping token is not valid or does not exist

I'm using the VaultKVSecretReactiveEngine to fetch secrets

Config used:
quarkus.vault.url=${vault.addr}
quarkus.vault.kv-secret-engine-version=1
quarkus.vault.authentication.client-token-wrapping-token=${wrapped.vault.token}

I know the ttl for the vault token is 2 hours.

I'm not sure if this is a bug or if I don't understand. Shouldn't a request be made to /v1/auth/token/renew-self for renewal?

Am I suppose to renew programatically?

Granted this is not my area of expertise so any guidance is appreciated

[quarkus-bot[bot]]

/cc @vsevel (vault)

[vsevel]

was it something that used to work before? I would be surprised.
I believe the issue here is that you are using a wrapping token.
it seems that the token you get from unwrapping the wrapped token expires, and upon renew, we try to unwrap the wrapped token again, which by definition is not allowed.
have you tried looking at the unwrapped token, and tried to renew it? if this is working, then may be there is something in the vault extension that is not done correctly.

[vsevel]

I tried reproducing the issue.
I started a dev vault with docker, logged in through the CLI with the root token, and created a new token valid for a minute:

$ docker run --rm --cap-add=IPC_LOCK -e VAULT_ADDR=http://localhost:8200 -p 8200:8200 -d --name=dev-vault vault:1.13.3
$ export VAULT_TOKEN=<root token>
$ export VAULT_ADDR=http://localhost:8200
$ vault token create -ttl=1m
Key                  Value
---                  -----
token                hvs.6ZxfPBEITyjuOQpNpLeReLTx
token_accessor       pLsBjFgR1Ng8hw3Qdg258GHX
token_duration       1m
token_renewable      true
token_policies       ["root"]
identity_policies    []
policies             ["root"]

then I configured my app with:

quarkus.http.port=18080
quarkus.tls.trust-all=true
quarkus.log.category."io.quarkus.vault".level=TRACE
quarkus.log.min-level=TRACE
quarkus.vault.url=http://localhost:8200
quarkus.vault.log-confidentiality-level=low
quarkus.vault.renew-grace-period=20S
# root token: hvs.i1I9EjhLAfdbbuzsZnDMtdaI
quarkus.vault.authentication.client-token=hvs.6ZxfPBEITyjuOQpNpLeReLTx

and finally in GreetingResource:

    @Inject
    VaultKVSecretEngine secretEngine;
    
    @Scheduled(every = "10S")
    public void scheduled() {
        log.info("read secrets: " + getSecrets());
    }

I start the application, it works for a little while, and after a minute or so, I start getting errors.
it means that the static token does not get renewed at all, and that is not even considering a wrapping token, which could not improve the situation.
this would be an issue anyway if the application stopped and restarted because the static token would be invalid even if it was renewed inside the app, and even more if this was a wrapping token, since unwrapping is done only once.
that said, the token I create is renewable, and can do indeed:

$ vault token create -ttl=1m
Key                  Value
---                  -----
token                hvs.72k71EIdorOWmPR6uUKa846d
token_accessor       LnpnFaYzOfQiDeb51c31bwLm
token_duration       1m
token_renewable      true
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Sevel@DEVVX700077 MINGW64 /e/tmp/test/vault-wrapping-token
$ vault token renew hvs.72k71EIdorOWmPR6uUKa846d
Key                  Value
---                  -----
token                hvs.72k71EIdorOWmPR6uUKa846d
token_accessor       LnpnFaYzOfQiDeb51c31bwLm
token_duration       1m
token_renewable      true
token_policies       ["root"]
identity_policies    []
policies             ["root"]

one difference I see between the token providers for the other auth methods (approle, k8s, ...) is that the clientToken is the only one that does support caching. in VaultClient:

        public Builder clientToken(VaultStaticClientTokenAuthOptions options) {
            return tokenProvider(new VaultStaticClientTokenProvider(options));
        }

        public Builder appRole(VaultAppRoleAuthOptions options) {
            return tokenProvider(new VaultAppRoleTokenProvider(options).caching(options.cachingRenewGracePeriod));
        }

        public Builder kubernetes(VaultKubernetesAuthOptions options) {
            return tokenProvider(new VaultKubernetesTokenProvider(options).caching(options.cachingRenewGracePeriod));
        }

        public Builder userPass(VaultUserPassAuthOptions options) {
            return tokenProvider(new VaultUserPassTokenProvider(options).caching(options.cachingRenewGracePeriod));
        }

so I suspect this is the reason why renewing does not work.
@kdubb would you agree?

[mafa73]

No idea if it ever worked, haven't used Vault in Quarkus before.

That is my conclusion also, that the caching is not used and because of that the renewal/renew-self isn't used.

I managed to get it to work by overriding some classes:

package io.quarkus.vault.runtime.client

import io.github.oshai.kotlinlogging.KotlinLogging
import io.quarkus.vault.client.VaultClient
import io.quarkus.vault.client.auth.*
import io.quarkus.vault.runtime.VaultConfigHolder
import io.quarkus.vault.runtime.config.VaultAuthenticationConfig
import io.quarkus.vault.runtime.config.VaultRuntimeConfig
import io.vertx.core.Vertx
import jakarta.enterprise.inject.Produces
import jakarta.inject.Singleton
import org.eclipse.microprofile.config.inject.ConfigProperty
import java.time.InstantSource
import java.util.concurrent.CompletionStage


private val logger = KotlinLogging.logger {}

/**
 * Overrides Quarkus's VaultClientProducer to add caching to get a renewal in place
 * Added quarkus.arc.exclude-types=io.quarkus.vault.runtime.client.VaultClientProducer in application-properties
 * to not use VaultClientProducer to create the VaultClient
 *
 * Not needed when using kubernetes (it already has caching)
 */
@Singleton
class VaultClientProducerWithCachingSupport : VaultClientProducer() {

    @Produces
    @Singleton
    @Private
    override fun privateVaultClient(
        vaultConfigHolder: VaultConfigHolder,
        @ConfigProperty(name = "quarkus.tls.trust-all", defaultValue = "false") globalTrustAll: Boolean
    ): VaultClient {
        logger.debug { "privateVaultClient called using $globalTrustAll caching client for $vaultConfigHolder" }
        return super.privateVaultClient(vaultConfigHolder, globalTrustAll)
    }

    @Produces
    @Singleton
    override fun sharedVaultClient(
        vertx: Vertx, vaultConfigHolder: VaultConfigHolder,
        @ConfigProperty(name = "quarkus.tls.trust-all", defaultValue = "false") globalTrustAll: Boolean
    ): VaultClient {
        logger.debug { "sharedVaultClient called using $globalTrustAll caching client for $vaultConfigHolder" }
        return super.sharedVaultClient(vertx, vaultConfigHolder, globalTrustAll)
    }

    override fun configureAuthentication(builder: VaultClient.Builder, config: VaultRuntimeConfig) {

        logger.debug { "configureAuthentication called using builder $builder and config $config" }
        val authConfig: VaultAuthenticationConfig = config.authentication()

        if (authConfig.isDirectClientToken && authConfig.clientTokenWrappingToken().isPresent) {
            logger.info { "creating VaultClient with renewal using grace period: ${config.renewGracePeriod()}" }
            val authOptions = VaultStaticClientTokenAuthOptions.builder()
                .unwrappingToken(authConfig.clientTokenWrappingToken().orElseThrow())
                .caching(config.renewGracePeriod())
                .build()
            builder.tokenProvider(
                VaultCachingTokenProvider(
                    VaultRenewableClientTokenProvider(authOptions),
                    config.renewGracePeriod()
                )
            )
        } else super.configureAuthentication(builder, config)
    }
}

// Makes the auth request's VaultToken renewable, that is Vault will be called with renew-self request
// instead of trying to log in using already used wrapped token (that will fail)
private class VaultRenewableClientTokenProvider(val authOptions: VaultStaticClientTokenAuthOptions) :
    VaultStaticClientTokenProvider(authOptions) {

    override fun apply(authRequest: VaultAuthRequest): CompletionStage<VaultToken> {
        return authOptions.tokenProvider.apply(authRequest)
            .thenApply { clientToken: String ->
                //Lease duration isn't based on the real validTo, but this is easier
                VaultToken.renewable(
                    clientToken,
                    authOptions.cachingRenewGracePeriod.plusMinutes(30),
                    null,
                    InstantSource.system()
                )
            }
    }
}

This works but yeah it ain't pretty

[vsevel]

may be that is not a concern, but with the renewed token, and even more with the wrapping token, if the application was to stop, and get restarted, this could not work. I assume you are not running in a kubernetes cluster (or an infra with probes, and automatic restarts). or the control plane would have to go through generating a new token, and wrapping it.
with these limitations in mind, there might be an easy change in the vault extension where we would allow caching also for the VaultStaticClientTokenProvider

[mafa73]

It works because we get a new wrapped token when the service is restarted

[mafa73]

We currently have chef that manages this

[vsevel]

I was able to get it to work in a hack, that we can't commit as is.
in VaultClient activate caching with:

        public Builder clientToken(VaultStaticClientTokenAuthOptions options) {
            return tokenProvider(new VaultStaticClientTokenProvider(options).caching(options.cachingRenewGracePeriod));
        }

and in VaultStaticClientTokenProvider, comment caching(Duration) , and in apply(VaultAuthRequest), replace neverExpires(...) with:

    @Override
    public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
        return tokenProvider.apply(authRequest).thenApply(token -> VaultToken.from(token, true, Duration.ofSeconds(60), null, authRequest.getInstantSource()));
    }

this would hardcode the token as 1) renewable 2) with leaseDuration=60secs

obviously, this last statement is what causes an issue.

in normal login, we would make the appropriate call (e.g. /auth/login), and as a result get a token with valid metadata, such as the renewable and leaseDuration fields. and that tells us if we can renew or not.
whereas, when using the direct token, we just have the token. so we have no idea if the token is renewable, and how much is the lease.
we would have to call vault to get this information before we start using it. so for instance: vault token lookup <token>.

[vsevel]

so in short, renewal is not currently supported for direct client token, and there would be a little bit of work if we wanted to support it.
would that be an option to use one of the other auth methods, such as approle?
you can still using a wrapping token to hide the secretId.
see quarkus.vault.authentication.app-role.secret-id-wrapping-token

[mafa73]

We're moving to Kubernetes soon/few months and then we will use approle, so I think I leave it as it is for now.

Thanks for looking into this for me