Docker in CI/CD & Production

A container image is what actually runs in production, so the build that produces it deserves the same rigour as the code inside it — reproducible, cached, scanned and attributable.

BuildKit#

# syntax=docker/dockerfile:1.7
FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /build
 
COPY pom.xml .
# The cache mount survives between builds and between images. Without it,
# every CI run downloads the entire dependency tree again.
RUN --mount=type=cache,target=/root/.m2,sharing=locked \
    mvn -B dependency:go-offline
 
COPY src ./src
RUN --mount=type=cache,target=/root/.m2,sharing=locked \
    --mount=type=secret,id=maven_settings,target=/root/.m2/settings.xml \
    mvn -B clean package -DskipTests

DOCKER_BUILDKIT=1 docker build \
  --secret id=maven_settings,src=$HOME/.m2/settings.xml \
  -t acme/order-service:$(git rev-parse --short HEAD) .

The secret mount is the important detail. A credential passed as ARG or copied into the image is recoverable from the image history by anyone who can pull it — docker history shows the build arguments. A secret mount exists only during that RUN instruction and leaves no trace in any layer.

sharing=locked on the cache mount prevents two concurrent builds corrupting the Maven repository, which is a genuine problem on a busy CI runner.

Tagging#

The reason this matters is rollback. When production is broken and you need the previous version, "the image that was tagged latest yesterday" is not a thing you can retrieve. The SHA is.

For registries that support it, enable tag immutability so a tag cannot be overwritten at all — which removes the possibility of the image under a given tag changing between the test that passed and the deploy that followed.

Scanning#

- name: Scan image
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: ghcr.io/acme/order-service:${{ github.sha }}
    format: table
    severity: 'HIGH,CRITICAL'
    exit-code: '1'
    ignore-unfixed: true
    vuln-type: 'os,library'
 
- name: Scan Dockerfile for misconfiguration
  uses: aquasecurity/trivy-action@master
  with:
    scan-type: config
    scan-ref: Dockerfile
    severity: 'HIGH,CRITICAL'

Trivy checks both the OS packages in the base image and the Java dependencies in the jar. The config scan catches Dockerfile problems the vulnerability scan cannot — running as root, no USER instruction, a mutable base tag.

The most effective way to reduce findings is not triage but base-image choice. A distroless or Alpine JRE has a fraction of the packages of a full Debian image, and most reported CVEs are in packages your application never invokes.

SBOM and signing#

# Generate a software bill of materials
syft ghcr.io/acme/order-service:$SHA -o cyclonedx-json > sbom.json
 
# Attach it to the image in the registry
cosign attach sbom --sbom sbom.json ghcr.io/acme/order-service:$SHA
 
# Keyless signing: the OIDC identity of the CI job becomes the signer
cosign sign --yes ghcr.io/acme/order-service:$SHA
 
# Verify, scoped to the workflow that is permitted to build this image
cosign verify ghcr.io/acme/order-service:$SHA \
  --certificate-identity-regexp 'https://github.com/acme/order-service/.github/workflows/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Keyless signing removes the key-management problem entirely. There is no private key to store or rotate — the CI job's OIDC token is exchanged for a short-lived certificate, the signature is recorded in a public transparency log, and verification checks that the signer was the workflow you expect.

An SBOM becomes valuable the day a widespread vulnerability is announced. "Which of our 200 services ship this library, and at what version" is a query against stored SBOMs rather than a week of investigation.

Enforcing at admission#

Signing only helps if something checks. A Kubernetes admission policy closes the loop:

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata: { name: require-signed-images }
spec:
  images:
    - glob: 'ghcr.io/acme/**'
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subjectRegExp: 'https://github.com/acme/.*'

With this in place, an image pushed by a compromised registry credential will not schedule, because it carries no signature from your workflow. Roll it out in warn mode first and check what fails — there is almost always a base image or a third-party tool nobody remembered.

Registry hygiene#

Images accumulate. A build per commit across fifty services fills a registry quickly, and storage is not the only cost — a registry with a hundred thousand tags is slow to query and hard to audit.

Set a retention policy: keep all tagged releases, keep SHA-tagged images for ninety days, delete untagged manifests after a week. Most registries support this natively; where they do not, a scheduled cleanup job is worth the hour it takes to write.

Use a pull-through cache for public base images. It removes a dependency on Docker Hub availability and rate limits during a deploy, which is a failure mode that has taken down more pipelines than any vulnerability.

What to take away#

Use BuildKit cache mounts for dependencies and secret mounts for credentials — never build args. Tag with the git SHA and promote that one artefact. Scan with ignore-unfixed, sign keylessly, and verify at admission so signing actually enforces something. Then set a retention policy before the registry becomes unmanageable.