Secrets Management & Key Security

Most credential leaks are not sophisticated. They are a key in a repository, an environment variable in a manifest, or a password that has not changed since the person who set it left.

Inventory first#

The "never" column is the finding. Most organisations discover during this exercise that credentials issued years ago are still live, still shared, and known to people who have left. The inventory itself is often more valuable than the tooling that follows.

Vault#

spring:
  cloud:
    vault:
      uri: https://vault.internal:8200
      authentication: KUBERNETES
      kubernetes:
        role: order-service
        # The pod's service account token — no static credential to distribute.
        service-account-token-file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kv:
        enabled: true
        backend: secret
        default-context: order-service
      database:
        enabled: true
        role: order-service-db
        backend: database
  config:
    import: 'vault://'

Kubernetes authentication removes the bootstrap problem. The pod authenticates with a token the platform already provides, so there is no static Vault credential to distribute or rotate — which is otherwise the awkward part of any secret manager.

Dynamic credentials#

vault write database/roles/order-service-db \
    db_name=orders \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
                         GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

Note what the grant does not include: no DROP, no CREATE, no schema ownership. A dynamic credential is also an opportunity to apply least privilege, since you are writing the grant explicitly rather than reusing an account that accumulated permissions over years.

Spring Cloud Vault renews the lease automatically and refreshes the connection pool when credentials rotate. Test that path deliberately — a pool that does not pick up new credentials fails an hour after deployment, which is a confusing incident to diagnose.

AppRole outside Kubernetes#

// role-id is configuration; secret-id is short-lived and delivered separately,
// so neither alone is sufficient to authenticate.
@Bean
ClientAuthentication clientAuthentication(VaultProperties props) {
    return new AppRoleAuthentication(
            AppRoleAuthenticationOptions.builder()
                    .roleId(RoleId.provided(props.roleId()))
                    .secretId(SecretId.provided(props.secretId()))
                    .build(),
            restOperations);
}

The two-part split is the point: a role id in configuration is not a credential on its own, and the secret id is delivered at deploy time with a short TTL and a use limit. Compromising the configuration repository does not yield Vault access.

External Secrets Operator#

For teams not ready to add a Vault client to every application, the operator syncs secrets into native Kubernetes objects:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: order-service-secrets
spec:
  refreshInterval: 1h
  secretStoreRef: { name: vault-backend, kind: ClusterSecretStore }
  target:
    name: order-service-secrets
    creationPolicy: Owner
  data:
    - secretKey: STRIPE_API_KEY
      remoteRef: { key: order-service, property: stripe_api_key }
    - secretKey: SMTP_PASSWORD
      remoteRef: { key: order-service, property: smtp_password }

Applications read ordinary environment variables and know nothing about Vault, while the source of truth gains rotation and audit. It is the pragmatic migration path — adopt the operator first, move individual services to direct integration and dynamic credentials as they need it.

Rotation with overlap#

@Service
public class TokenVerifier {
 
    /** During rotation both keys verify; only the current one signs. */
    public Claims verify(String token) {
        for (SigningKey key : keyRegistry.activeKeys()) {   // current first, then previous
            try {
                return Jwts.parser().verifyWith(key.publicKey()).build()
                           .parseSignedClaims(token).getPayload();
            } catch (JwtException ignored) {
                // Try the next key.
            }
        }
        throw new InvalidTokenException();
    }
}

The overlap window is what makes rotation routine rather than an event. Publish the new key alongside the old, switch signing, wait for everything signed by the old key to expire, then retire it — no coordinated deploy, no window where valid credentials are rejected.

Rotation that requires a simultaneous change everywhere gets deferred until a breach forces it, which is the worst possible time to attempt it for the first time.

What must never happen#

Secrets in source control, including history — rotate anything found there rather than only removing it. Secrets in container images, recoverable from docker history. Secrets in logs, where they propagate into every downstream index and backup. Secrets in error messages returned to clients. And secrets in environment variables printed by a debug endpoint — /actuator/env masks common patterns, but only patterns it recognises.

Add a scanner for each of these paths rather than relying on review, because each has been a real breach for someone.

What to take away#

Take the inventory first — it usually finds credentials nobody remembered. Move to a secret manager, using the platform identity to authenticate rather than a bootstrap credential. Prefer dynamic credentials with short leases so a leak expires on its own. And build rotation with an overlap window before you need it.