Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Containerization with Docker & Kubernetes

Containerize and deploy a Spring Boot AI application: a production Dockerfile with layered JARs, Kubernetes deployment with secrets for API keys, health probes and resource limits.

Intermediate4 min readUpdated
On this page

Deploying a Spring Boot AI application is mostly deploying a Spring Boot application — the AI part runs on someone else's hardware. The differences that matter are secret handling for API keys, health probes that account for slow model dependencies, and resource sizing for an I/O-bound workload.

Key Takeaways

  • Use a layered JAR Dockerfile so dependency layers cache and rebuilds are fast.
  • API keys go in Kubernetes Secrets mounted as env vars — never in the image.
  • Health probes must not fail just because the model provider is slow.
  • AI services are I/O-bound: size for memory and concurrency, not CPU.

A production Dockerfile

The naive Dockerfile copies the fat JAR in one layer, so any code change re-downloads every dependency. Spring Boot's layered JARs fix that.

Dockerfile
# --- Build stage ---
FROM eclipse-temurin:21-jdk-alpine AS build
WORKDIR /app
COPY .mvn/ .mvn/
COPY mvnw pom.xml ./
# Download dependencies first; this layer caches unless pom.xml changes.
RUN ./mvnw dependency:go-offline -B
COPY src ./src
RUN ./mvnw clean package -DskipTests -B
 
# Extract the layered JAR so Docker can cache dependency layers separately.
RUN java -Djarmode=layertools -jar target/*.jar extract --destination extracted
 
# --- Runtime stage ---
FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
 
# Run as non-root. A container running as root is an unnecessary risk.
RUN addgroup -S app && adduser -S app -G app
USER app
 
# Copy layers in order of change frequency: dependencies rarely, code often.
COPY --from=build /app/extracted/dependencies/ ./
COPY --from=build /app/extracted/spring-boot-loader/ ./
COPY --from=build /app/extracted/snapshot-dependencies/ ./
COPY --from=build /app/extracted/application/ ./
 
EXPOSE 8080
# Virtual threads shine here; enable them in application.yml.
ENTRYPOINT ["java", "org.springframework.boot.loader.launch.JarLauncher"]

The API key: never in the image

# NEVER do this — the key is now in the image history forever, readable by
# anyone who can pull it.
# ENV OPENAI_API_KEY=sk-proj-abc123

The key is injected at runtime from a Kubernetes Secret. Create it out of band:

kubectl create secret generic ai-secrets \
  --from-literal=openai-api-key="$OPENAI_API_KEY"

Kubernetes deployment

deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-service
spec:
  replicas: 2
  selector:
    matchLabels: { app: ai-service }
  template:
    metadata:
      labels: { app: ai-service }
    spec:
      containers:
        - name: ai-service
          image: registry.example.com/ai-service:1.0.0
          ports:
            - containerPort: 8080
          env:
            - name: SPRING_AI_OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ai-secrets
                  key: openai-api-key
            - name: JAVA_TOOL_OPTIONS
              # Let the JVM see the container's real memory limit.
              value: "-XX:MaxRAMPercentage=75.0"
          resources:
            requests:
              cpu: "250m"      # I/O-bound: modest CPU
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "768Mi"  # above heap + overhead
          # Liveness: is the process alive? Keep it cheap and local.
          livenessProbe:
            httpGet: { path: /actuator/health/liveness, port: 8080 }
            initialDelaySeconds: 20
            periodSeconds: 10
          # Readiness: can it serve traffic? Also local — do NOT make it depend
          # on the model provider being up.
          readinessProbe:
            httpGet: { path: /actuator/health/readiness, port: 8080 }
            initialDelaySeconds: 15
            periodSeconds: 5

Health probes in Spring Boot

Expose the liveness and readiness groups:

application.yml
management:
  endpoint:
    health:
      probes:
        enabled: true
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true
spring:
  threads:
    virtual:
      enabled: true   # cheap concurrency for the I/O-bound model calls

Scaling on the right metric

CPU-based autoscaling is wrong for an I/O-bound AI service — it barely uses CPU while waiting on the model, so it never scales up even under heavy concurrent load. Scale on concurrency or a custom metric instead:

hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_server_active_requests   # concurrency, via Micrometer
        target:
          type: AverageValue
          averageValue: "20"

Running a local model in the cluster (optional)

If you run Ollama or vLLM yourself rather than calling a hosted API, that is where GPU node pools and much larger resource requests come in. Most applications do not need this — see Spring AI with Ollama for when local models make sense.

Deployment checklist

  • Non-root container user
  • Multi-stage build, JRE runtime, layered JAR
  • API keys from Secrets, ideally external-managed
  • MaxRAMPercentage so the JVM respects container limits
  • Liveness/readiness probes that do not depend on the model provider
  • Resource requests/limits sized for I/O-bound work
  • Autoscaling on concurrency, not CPU

Next

Frequently Asked Questions

How should I store the model API key in Kubernetes?
As a Kubernetes Secret, mounted into the pod as an environment variable — never baked into the image and never committed to Git. For production, back the Secret with an external manager such as AWS Secrets Manager or Vault via the External Secrets Operator, so the key is rotated centrally and never stored in plain etcd.
What resource limits should an AI service have?
AI services are I/O-bound — they spend most of their time waiting on the model API — so they need less CPU than a compute-heavy service but benefit from enough memory for connection pools and any local embedding models. Start with modest CPU requests, set memory limits above your JVM heap plus overhead, and scale on request concurrency rather than CPU.
Do I need a GPU to deploy an AI application in Kubernetes?
Not if you call a hosted model API — that runs on the provider's hardware. You only need GPUs if you run models yourself, for example a local Ollama or vLLM deployment. Most Spring AI and LangChain4j applications that call OpenAI, Anthropic or a cloud provider need no special hardware at all.

Related tutorials