Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Building AI-Powered REST APIs

Design robust AI REST APIs in Spring Boot: streaming with Server-Sent Events, async processing for long tasks, timeouts, back-pressure and the API patterns that make AI features reliable.

Intermediate4 min readUpdated
On this page

An AI feature is only as good as the API around it. The patterns that matter — streaming for responsiveness, async for long tasks, timeouts and back-pressure for robustness — are what separate an AI endpoint that feels fast and stays up from one that hangs and falls over. This tutorial covers them for Spring Boot.

Key Takeaways

  • Stream interactive responses with Server-Sent Events — the user sees output immediately.
  • Async with a job ID for long tasks — do not hold a connection open for minutes.
  • Timeouts on every model call — a hung call with no timeout is a latent outage.
  • Handle client disconnection so slow or vanished clients do not tie up resources.

Streaming with Server-Sent Events

For any response a user waits on, stream it. SSE is the right transport — one-way, server-to-client, over plain HTTP.

StreamingController.java
@RestController
public class StreamingController {
 
    private final ChatClient chatClient;
 
    public StreamingController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }
 
    @GetMapping(value = "/api/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> stream(@RequestParam String q) {
        return chatClient.prompt()
                .user(q)
                .stream()
                .content()
                // A stalled provider must not hold the connection forever.
                .timeout(Duration.ofSeconds(60))
                // A graceful message beats a broken stream.
                .onErrorResume(e -> Flux.just("Sorry — something went wrong."));
    }
}
Client
const source = new EventSource(`/api/chat/stream?q=${encodeURIComponent(q)}`);
source.onmessage = (e) => { output.textContent += e.data; };
source.onerror = () => source.close();   // stop retrying on completion/error

See reactive programming with Project Reactor for the Flux mechanics.

Synchronous non-streaming, when you must validate

If you need to inspect the whole response before returning it — schema validation, moderation — you cannot stream. Use a blocking call with a timeout:

@PostMapping("/api/extract")
public ExtractResponse extract(@RequestBody ExtractRequest request) {
    // Validate the whole output before returning it. Streaming forfeits that
    // chance, so structured, validated responses use call(), not stream().
    Invoice invoice = chatClient.prompt()
            .user(u -> u.text("Extract invoice fields from:\n{doc}").param("doc", request.document()))
            .call()
            .entity(Invoice.class);
 
    validate(invoice);   // business rules before the response leaves
    return new ExtractResponse(invoice);
}

Async processing for long tasks

Some AI work — processing a large document, running a multi-step agent — takes too long to hold a connection open. Accept the job, return an ID, process in the background, deliver later.

Async job pattern
@PostMapping("/api/analyse")
public ResponseEntity<JobResponse> submit(@RequestBody AnalyseRequest request) {
    String jobId = jobService.submit(request);   // enqueue, return immediately
    return ResponseEntity.accepted()              // 202 Accepted
            .body(new JobResponse(jobId, "PROCESSING"));
}
 
@GetMapping("/api/analyse/{jobId}")
public JobResult poll(@PathVariable String jobId) {
    return jobService.status(jobId);   // client polls, or you push via webhook
}

The background worker processes the job — on Java 21, holding many concurrent long jobs is cheap on virtual threads. For heavy pipelines, back the queue with a message broker — see event-driven AI architectures.

Timeouts everywhere

Handling client disconnection

A user closes the tab mid-stream. Without handling, you keep generating (and paying) for a client that is gone:

return chatClient.prompt().user(q).stream().content()
        .timeout(Duration.ofSeconds(60))
        // When the client disconnects, stop work — do not keep spending tokens.
        .doOnCancel(() -> log.info("client disconnected, stopping generation"));

Rate limiting and cost protection

AI endpoints are expensive; protect them per user:

@RateLimiter(name = "aiEndpoint")   // Resilience4j / bucket4j
@PostMapping("/api/chat")
public ChatResponse chat(@RequestBody ChatRequest request) { /* ... */ }

And bound input size before it becomes an expensive call — see securing AI applications.

API design checklist

  • Interactive responses stream via SSE with a timeout and error path
  • Responses needing validation use blocking call(), validated before return
  • Long tasks are async with a job ID, not held connections
  • Every model call has a timeout; a circuit breaker guards the provider
  • Client disconnection stops generation
  • Endpoints are rate-limited and input-bounded per user
  • Token cost is tracked per endpoint

Next

Frequently Asked Questions

How do I stream an AI response from a Spring Boot REST API?
Return a Flux of String chunks from a controller method that produces text/event-stream, using chatClient.prompt().stream().content(). Spring maps the Flux to Server-Sent Events, and the browser consumes them with EventSource. SSE is the standard choice because only the server pushes, which is simpler than WebSockets.
Should AI endpoints be synchronous or asynchronous?
Stream synchronously for interactive requests a user waits on, so they see output immediately. Use asynchronous processing — accept the request, return a job ID, process in the background, deliver by webhook or polling — for long-running tasks like processing a large document, where holding the connection open is impractical.
Why must AI endpoints have timeouts?
A model call can hang, and without a timeout it holds a request thread indefinitely, eventually exhausting the pool and taking down the service. Every AI endpoint needs a timeout on the model call and, for streaming, on the stream itself, plus an error path that returns a graceful message rather than hanging.
How do I handle back-pressure when streaming to slow clients?
Reactor handles it: when a client consumes tokens slower than the model produces them, the reactive stream signals demand upstream so you do not buffer unbounded data. Set a timeout on the stream and handle client disconnection so a slow or vanished client does not tie up resources. This is a core reason streaming AI uses reactive types.

Related tutorials