LLMOps & MLOps for Generative AI

Once an LLM feature is in production, keeping it good is an operational discipline: versioning prompts, testing changes, rolling them out safely, and tracking quality and cost over time. This tutorial covers LLMOps — the practices that turn a working prototype into a maintainable production system.

Prompts are code#

The foundational LLMOps practice: a prompt is a behavioural specification, and it belongs under the same discipline as code.

// Prompts live in version control — as resource files or typed constants —
// not as strings edited in a database or a config UI in production.
public enum PromptVersion {
    SUPPORT_V3("prompts/support-v3.txt"),
    SUPPORT_V4("prompts/support-v4.txt");   // the candidate, tested before rollout
 
    private final String path;
    // ... load from classpath
}

Evaluation in CI/CD#

A prompt or model change must pass your evaluation suite before it ships:

// Run on every change to a prompt or model config. Fail the build if the pass
// rate regresses beyond a threshold.
@Test
void promptChangeDoesNotRegress() {
    double baseline = evaluate(PromptVersion.SUPPORT_V3, evalSuite).passRate();
    double candidate = evaluate(PromptVersion.SUPPORT_V4, evalSuite).passRate();
 
    assertThat(candidate)
            .as("candidate prompt must not regress quality")
            .isGreaterThanOrEqualTo(baseline - REGRESSION_TOLERANCE);
}

The deterministic parts of this (retrieval hit rate, structured-output validity) run on every commit; the model-calling parts run pre-release, guarded to skip without an API key. See testing AI applications.

Model and prompt lineage#

Record what served when, so any behaviour is reproducible and traceable:

record Deployment(
        String modelProvider, String modelVersion,   // pinned, not "latest"
        String promptVersion,
        Instant deployedAt,
        EvalReport evalAtDeploy) {}

Even with hosted models, pin exact versions and record them. When a regression appears, "which model and prompt version was live?" is the first question — and an unanswerable one if you did not track it. See foundation models deep dive.

Safe rollout: canary and A/B#

Never swap a model or prompt for all traffic at once. Roll out gradually:

public String handle(Request request) {
    // Route a small fraction to the candidate; monitor both cohorts.
    boolean canary = hash(request.userId()) % 100 < canaryPercent;
    PromptVersion version = canary ? candidateVersion : stableVersion;
 
    String output = process(request, version);
    metrics.record(version, request, output);   // compare cohorts
    return output;
}

A/B testing is the same mechanism aimed at comparison: split traffic between two variants and measure which performs better on real usage — quality, cost, user satisfaction — before committing.

Monitoring in production#

LLMOps needs the observability from Spring AI observability, watched continuously:

• Quality — sampled outputs judged or human-reviewed, watching for drift.
• Cost — tokens and money per feature, with alerts on anomalies.
• Latency — P95, split into retrieval and generation.
• Error rate — provider errors, validation failures, guardrail blocks.

A model provider can change a model's behaviour under a version alias, or your input distribution can shift; continuous monitoring is how you notice before users do.

Handling provider changes#

Providers deprecate models and ship new versions. Your LLMOps process should make this routine:

1. A new model version appears.
2. Run your evaluation suite against it.
3. If it passes, canary it; if it regresses, stay on the pinned version and plan.
4. When a deprecation forces a move, you have already evaluated the alternatives.

Because your code depends on the ChatModel abstraction, the switch itself is a config change — the work is the evaluation, not the code.

The LLMOps maturity ladder#

1. Prompts in version control — the baseline everyone should have.
2. An evaluation suite run before changes ship.
3. Cost and quality monitoring in production.
4. Gradual rollout with canaries and rollback.
5. Automated regression gates in CI/CD.

Most teams should aim for the first three quickly; four and five come as the system matures and the stakes rise.

Next#

• GenAI on AWS, Azure & GCP
• AI observability & LLM tracing
• AI in CI/CD pipelines