Prompt Engineering Masterclass

Basic prompt engineering — role, examples, delimiters, output contracts — solves most problems. This masterclass covers the advanced techniques for the cases that remain: chaining, meta-prompting, self-consistency and systematic optimization. It builds on prompt engineering for Java developers.

Prompt chaining#

A single prompt asked to extract, analyse and format all at once often does none well. Chain focused prompts instead:

// Each step is a focused prompt; its output is validated before the next.
Entities entities = extractPrompt.run(document);      // step 1: extract
validate(entities);
Analysis analysis = analysePrompt.run(entities);      // step 2: analyse
Report report = formatPrompt.run(analysis);           // step 3: format

Each step is easier to get right, easier to test, and its output can be checked before it propagates. This is task decomposition at the prompt level, and it is often the difference between a flaky prompt and a reliable pipeline. Build chains as LangChain4j chains or plain Java.

Meta-prompting#

Use the model to improve your prompts. It is good at generating examples, critiquing instructions and proposing variations:

// Ask the model to critique a prompt against a goal, then apply its suggestions.
String critique = model.chat("""
        Here is a prompt intended to classify support tickets by urgency.
        Identify weaknesses that would cause misclassification, and suggest
        specific improvements.
 
        Prompt:
        %s
        """.formatted(currentPrompt));

Self-consistency#

For reasoning tasks, a single run can make a random mistake. Self-consistency runs the prompt several times and takes the majority answer:

public Decision decideRobustly(String input) {
    // Sample several times at a moderate temperature so runs vary, then take
    // the majority. Averages out individual reasoning errors.
    Map<Decision, Long> votes = IntStream.range(0, 5)
            .mapToObj(i -> decider.decide(input))   // temperature ~0.7
            .collect(Collectors.groupingBy(d -> d, Collectors.counting()));
 
    return votes.entrySet().stream()
            .max(Map.Entry.comparingByValue())
            .map(Map.Entry::getKey)
            .orElseThrow();
}

It is five times the cost, so reserve it for decisions where accuracy justifies it — a human-in-the-loop confidence signal, a high-value classification. For routine work, a single low-temperature call is right.

Structured reasoning#

Guide the model's reasoning with an explicit structure rather than hoping it reasons well:

String prompt = """
        Analyse this contract clause using this structure:
        1. PLAIN MEANING: what the clause says in plain language.
        2. OBLIGATIONS: who must do what.
        3. RISKS: what could go wrong for our side.
        4. RECOMMENDATION: accept, negotiate, or reject, with one reason.
 
        Clause:
        %s
        """.formatted(clause);

The structure both improves the reasoning and makes the output parseable and reviewable. It pairs naturally with structured output when the machine consumes the result.

Systematic optimization#

Prompt engineering becomes engineering when you measure. The loop:

// 1. A fixed evaluation set — inputs with expected outcomes.
// 2. Change ONE element of the prompt (add an example, tighten a constraint).
// 3. Measure against the set.
// 4. Keep the change if it helped; revert if not.
for (PromptVariant variant : variants) {
    double score = evaluate(variant, evaluationSet);
    log.info("variant={} score={}", variant.name(), score);
}

Automated prompt optimization#

Tools exist that search prompt variations against an evaluation set automatically — generating candidates, testing them, and iterating toward a higher score. They are useful for squeezing out the last few points on a well-defined task, but they need the same foundation you would build by hand: a representative evaluation set and a clear metric. Automation optimizes; it does not decide what "good" means.

When to stop#

Prompt engineering has diminishing returns. If a prompt is still unreliable after focused optimization, the problem may be elsewhere: the task needs retrieval to ground it, structured output to constrain it, or genuinely fine-tuning. Knowing when to stop tuning a prompt and change the approach is itself a skill.

Next#

• Tokenization & context windows
• LLM evaluation & benchmarks
• Guardrails & safety systems