Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Chatbot & Conversational AI Architecture

Design production chatbots in Java: intent classification, dialog state management, slot filling, multi-turn context, tool integration and handoff to humans — beyond a single ChatClient call.

Intermediate4 min readUpdated
On this page

A single ChatClient call is a chatbot the way a single SQL query is a database application — the core, surrounded by everything that makes it real. A production chatbot needs memory, intent understanding, tools, guardrails and human escalation. This tutorial covers the architecture.

Key Takeaways

  • A production chatbot is a ChatClient plus memory, intent, tools, guardrails and escalation.
  • Dialog state tracks a multi-turn flow so information can arrive across messages.
  • Tools let it take real actions; guardrails keep it in scope.
  • Always provide a graceful human handoff — never trap a user.

The architecture

A production chatbot: memory, intent, a tool-using agent, validation and a human escalation path.

Memory and multi-turn context

A conversation is stateful; the model is not. Bounded chat memory carries context across turns:

Conversational service
@Service
public class ChatbotService {
 
    private final ChatClient chatClient;
 
    public ChatbotService(ChatClient.Builder builder, ChatbotTools tools) {
        this.chatClient = builder
                .defaultSystem(SYSTEM_PROMPT)
                .defaultTools(tools)                 // real actions
                .defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())
                .build();
    }
 
    public Reply chat(String conversationId, String message) {
        String response = chatClient.prompt()
                .user(message)
                // Conversation ID from the authenticated session, keeping each
                // user's context separate and private.
                .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
                .call()
                .content();
        return new Reply(response);
    }
}

Dialog state and slot filling

For task-oriented flows — booking, ordering, troubleshooting — track what has been established. Modern LLMs handle this more fluidly than old rule-based systems, but the concept remains: know which information you still need.

Slot tracking
record BookingState(
        Optional<LocalDate> date,
        Optional<LocalTime> time,
        Optional<Integer> partySize) {
 
    boolean isComplete() {
        return date.isPresent() && time.isPresent() && partySize.isPresent();
    }
 
    List<String> missingSlots() { /* return the empty ones */ }
}
public Reply handleBooking(String conversationId, String message, BookingState state) {
    BookingState updated = slotExtractor.extract(message, state);   // fill from this message
    if (!updated.isComplete()) {
        // Ask only for what is still missing — not for what the user already gave.
        return askFor(updated.missingSlots());
    }
    return confirmAndBook(updated);
}

Scope guardrails

A chatbot that answers anything confidently is a liability. Keep it in its lane:

private static final String SYSTEM_PROMPT = """
        You are a booking assistant for Acme Restaurants.
        Help only with reservations, menus, hours and locations.
        If asked about anything else, politely say it is outside what you can
        help with, and offer to connect the user to a person.
        Never invent availability, prices or policies.
        """;

Combine the prompt with output validation — see guardrails & safety systems.

Tools for real actions

A chatbot that can only talk is half a product. Tools let it check availability, make bookings, look up orders — with the same discipline: read tools freely, write tools authorized in code and confirmed.

@Tool("Check table availability for a date, time and party size.")
String checkAvailability(LocalDate date, LocalTime time, int partySize) { /* read */ }
 
@Tool("Book a table. Only call after the user confirms the details.")
String book(LocalDate date, LocalTime time, int partySize, String confirmation) {
    if (!"CONFIRMED".equals(confirmation)) return "CONFIRMATION_REQUIRED";
    // authorize, validate, book
}

Graceful human handoff

The most important design principle: never trap a user in a bot that cannot help them.

Escalation
public Reply handle(String conversationId, String message) {
    if (frustrationDetector.isFrustrated(message) || userRequestsHuman(message)) {
        // Hand off WITH the conversation so the human continues, not restarts.
        return escalate(conversationId, "user requested or frustrated");
    }
    Reply reply = process(conversationId, message);
    if (reply.confidence() < ESCALATION_THRESHOLD) {
        return escalate(conversationId, "low confidence");
    }
    return reply;
}

Streaming for responsiveness

Chat is interactive, so stream the response — the user sees it forming rather than waiting for a complete reply. The exception is when you must validate the whole response before showing it, where you block, validate, then reply.

Putting it together

A production chatbot is: bounded memory + intent understanding + dialog state for tasks + tools for actions + scope guardrails + output validation + streaming + graceful escalation. Each piece is covered elsewhere in this curriculum; the architecture is assembling them around the core model call.

Next

Frequently Asked Questions

How do I build a production chatbot beyond a single model call?
A production chatbot combines bounded conversation memory, intent understanding, tool integration for real actions, scope guardrails, and a path to hand off to a human. A single ChatClient call is the core, but the architecture around it — state, tools, guardrails, escalation — is what makes it reliable and useful rather than a demo.
What is dialog state management?
Tracking where a conversation is and what has been established — which slots are filled, what the user is trying to do, what the bot is waiting for. It lets a chatbot handle multi-turn flows like booking, where information arrives across several messages, without losing track or asking for the same thing twice.
What is slot filling?
Gathering the specific pieces of information a task needs across a conversation — for a booking, the date, time and party size. The bot tracks which slots are filled and asks for the missing ones, rather than expecting everything in one message. Modern LLM chatbots do this more fluidly than older rule-based systems, but the concept of tracking required information still applies.
When should a chatbot hand off to a human?
When it is out of scope, when the user is frustrated or explicitly asks, when confidence is low on a consequential matter, or when a task requires authority the bot does not have. A good chatbot escalates gracefully with the conversation context, so the human continues rather than starting over. Never trap a user in a bot that cannot help them.

Related tutorials