Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Event-Driven AI Architectures

Build event-driven AI systems with Kafka and Spring Boot: async AI processing pipelines, decoupling model calls from request threads, dead-letter handling and back-pressure for LLM workloads.

Advanced4 min readUpdated
On this page

Not all AI work needs to happen inline while a user waits. Enriching records, processing uploads, running agents on incoming events — these belong in an event-driven pipeline, where model calls are decoupled from request threads and the queue absorbs bursts. This tutorial builds one with Kafka and Spring Boot.

Key Takeaways

  • Event-driven AI decouples slow model calls from request threads and from each other.
  • The queue is back-pressure — bursts accumulate and drain at a sustainable rate.
  • Dead-letter persistently failing messages; retry transient failures with backoff.
  • Use it when nobody is waiting on a connection — background enrichment, uploads, scheduled agents.

Why decouple AI work

An inline model call couples your request thread to a slow, rate-limited, sometimes-failing external service. Under load, that coupling is fragile. Producing an event and processing it asynchronously breaks the coupling:

An event-driven AI pipeline: stages are decoupled topics, bursts queue, failures dead-letter.

Producing events

Enqueue AI work
@Service
public class DocumentIntakeService {
 
    private final KafkaTemplate<String, DocumentEvent> kafka;
 
    public String submit(UploadedDocument doc) {
        String jobId = UUID.randomUUID().toString();
        // Return immediately; the model call happens downstream, decoupled.
        kafka.send("documents", jobId, new DocumentEvent(jobId, doc.content(), doc.tenantId()));
        return jobId;
    }
}

Consuming and processing

AI processing consumer
@Component
public class DocumentProcessor {
 
    private final ChatClient chatClient;
    private final KafkaTemplate<String, ExtractedEvent> kafka;
 
    @KafkaListener(
            topics = "documents",
            // Concurrency bounds how many model calls run at once — your rate-
            // limit and cost control lever. The queue absorbs the rest.
            concurrency = "4")
    public void process(DocumentEvent event) {
        try {
            Extracted extracted = chatClient.prompt()
                    .user(u -> u.text("Extract structured data from:\n{doc}")
                            .param("doc", event.content()))
                    .call()
                    .entity(Extracted.class);
 
            kafka.send("extracted", event.jobId(), new ExtractedEvent(event.jobId(), extracted));
 
        } catch (TransientAiException e) {
            throw e;   // let the retry mechanism handle it (see below)
        } catch (Exception e) {
            // Permanent failure — do not retry forever; dead-letter it.
            deadLetter(event, e);
        }
    }
}

Failure handling: retry vs dead-letter

Model calls fail in distinct ways, and you treat them differently:

Retry configuration
@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<String, Object> template) {
    // Route messages that exhaust retries to a dead-letter topic instead of
    // blocking the partition or losing them.
    var recoverer = new DeadLetterPublishingRecoverer(template);
    // Exponential backoff for transient failures (rate limits, timeouts).
    var backoff = new ExponentialBackOff(1000L, 2.0);
    backoff.setMaxElapsedTime(60_000L);
    return new DefaultErrorHandler(recoverer, backoff);
}

The queue as back-pressure

The most valuable property of the event-driven approach for AI: the queue is a natural back-pressure mechanism. When a burst of work arrives, it accumulates in the topic and drains at the rate your consumers (bounded by their concurrency) can process — which you tune to stay within the model provider's rate limits and your cost budget.

// Consumer concurrency is your throttle. Four concurrent model calls means at
// most four in flight regardless of how fast events arrive — the queue holds
// the rest. This protects both your budget and the provider's rate limit.
@KafkaListener(topics = "documents", concurrency = "4")

This is why an event-driven pipeline handles a traffic spike gracefully where inline calls would trip rate limits and errors. See productionizing agentic systems for budget enforcement.

Async agent pipelines

Agents that run on incoming events — a support ticket arrives, an alert fires — fit naturally:

@KafkaListener(topics = "alerts")
public void onAlert(AlertEvent alert) {
    // The investigation agent runs asynchronously, decoupled from whatever
    // produced the alert. Its findings publish to a downstream topic.
    Investigation result = investigationAgent.investigate(alert);
    kafka.send("investigations", alert.id(), new InvestigationEvent(result));
}

The DevAgentic project uses this pattern to trigger its investigation agent from alerts.

Ordering, idempotency and exactly-once

Two practical concerns:

  • Idempotency — a message may be delivered more than once. Make processing idempotent (keyed on the job ID) so a redelivery does not double the work or the cost. This mirrors the idempotent ingestion in the RAG pipeline.
  • Ordering — Kafka orders within a partition. If order matters (conversation turns), key by conversation ID so related events land on the same partition.

When to use event-driven AI

Use event-drivenUse synchronous
Background enrichmentA user waits for the answer
Processing uploadsLive chat
Scheduled agent runsReal-time extraction
Bursty, high-volume workLow-latency single requests
Multi-stage pipelinesSimple single calls

Next

Frequently Asked Questions

Why use Kafka for AI processing?
It decouples slow, expensive model calls from request threads and from each other. Producing an event and processing it asynchronously means a spike in AI work queues up rather than overwhelming the system, retries are natural, and stages scale independently. For any AI processing that does not need a synchronous response, event-driven architecture is more robust than inline calls.
How do I handle failures in an AI processing pipeline?
Retry transient failures with backoff, and route messages that keep failing to a dead-letter topic for inspection rather than blocking the pipeline or losing them. Because model calls fail in distinct ways — rate limits, timeouts, content issues — distinguish retryable failures from permanent ones, retrying the former and dead-lettering the latter.
How do I control cost and rate limits in an async AI pipeline?
Control consumer concurrency so you do not exceed the model provider's rate limit, and let the queue absorb bursts rather than hammering the provider. The queue itself is a back-pressure mechanism: work accumulates when the AI stage is saturated and drains at a sustainable rate, which protects both your budget and the provider's limits.
When should AI be event-driven versus synchronous?
Synchronous when a user waits for the result — a chat response, a live extraction. Event-driven when the work can happen in the background — enriching records, processing uploads, generating reports, running agents on a schedule. The rule is: if nobody is waiting on the connection, decouple it into an event pipeline for robustness and scale.

Related tutorials