Microservices Architecture Deep Dive

You do not need to rebuild your architecture to add AI. But where an AI capability sits in a microservices topology, and how you protect the rest of the system from its failure modes, is worth getting right. This is a pragmatic tour of the patterns that intersect with AI work.

Where the AI service sits#

Isolating AI into its own service gives you three things: independent scaling (model calls are I/O-bound and scale differently from CPU-bound services), a blast radius limited to AI features when the provider has an outage, and a natural boundary for cost tracking and rate limiting.

Circuit breakers: the non-negotiable one#

A model call is slow, and providers have bad days. Without protection, a provider slowdown fills your thread pool with waiting requests until everything fails. Resilience4j gives you a circuit breaker in a few lines.

@Service
public class ResilientChatService {
 
    private final ChatClient chatClient;
 
    public ResilientChatService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }
 
    // Opens the circuit after a failure-rate threshold, then fails fast for a
    // cooldown before letting a trial request through. When open, callers get
    // the fallback immediately instead of piling up on a dead provider.
    @CircuitBreaker(name = "modelProvider", fallbackMethod = "fallback")
    @TimeLimiter(name = "modelProvider")
    public CompletableFuture<String> ask(String question) {
        return CompletableFuture.supplyAsync(() ->
                chatClient.prompt().user(question).call().content());
    }
 
    private CompletableFuture<String> fallback(String question, Throwable t) {
        log.warn("model provider unavailable, serving fallback", t);
        return CompletableFuture.completedFuture(
                "Our assistant is temporarily unavailable. Please try again shortly.");
    }
}

resilience4j:
  circuitbreaker:
    instances:
      modelProvider:
        failure-rate-threshold: 50
        wait-duration-in-open-state: 30s
        sliding-window-size: 20
  timelimiter:
    instances:
      modelProvider:
        timeout-duration: 30s

API gateway: the place for AI-specific controls#

The gateway is where cross-cutting AI concerns belong: per-user rate limiting (model calls are expensive), authentication before an expensive call is made, and request routing.

spring:
  cloud:
    gateway:
      routes:
        - id: ai-service
          uri: lb://ai-service
          predicates:
            - Path=/api/ai/**
          filters:
            # Rate-limit expensive AI endpoints per user, not per IP.
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 5
                redis-rate-limiter.burstCapacity: 10

The saga pattern and agent workflows#

A saga breaks a distributed transaction into local steps, each with a compensating action that undoes it if a later step fails. This is the same shape as a multi-step agent workflow.

// An agent books travel: flight, then hotel, then car. If the hotel fails,
// the flight booking must be compensated (cancelled). Every forward action
// carries its undo.
public sealed interface SagaStep permits BookFlight, BookHotel, BookCar {}
 
record BookFlight(String flightId) implements SagaStep {
    // compensate: cancelFlight(flightId)
}

The lesson for agent design is the mindset, not the framework: before you let an agent take an action, know how to undo it. An agent that can book but not cancel, or send but not retract, is an agent whose mistakes are permanent. This is why the DevAgentic project puts every write action behind an approval gate and prefers reversible operations.

CQRS and event sourcing, briefly#

CQRS (separate read and write models) helps AI systems where the read side is a vector store optimised for semantic search and the write side is your transactional database. You write to Postgres; a projection embeds and indexes into the vector store.

Event sourcing (storing state as a log of events) pairs well with AI observability: the event log is a natural place to record every agent action and model call for audit and replay — which is exactly what the trajectory logging in agent systems needs.

Service-to-service resilience checklist#

For any service that calls a model:

• Circuit breaker — fail fast when the provider is unhealthy.
• Timeout — bound every call; a model call with no timeout is a latent outage.
• Bulkhead — cap concurrent model calls so they cannot exhaust shared resources.
• Retry with backoff — but only for transient errors (429, 5xx), never for a bad request.
• Fallback — a degraded response beats an error page.
• Budget — track token cost per service and alert on anomalies.

Next#

• Containerization with Docker & Kubernetes — deploying these services
• Event-driven AI architectures — the async worker pattern in depth
• Multi-tenant AI architectures — isolation at scale