AI in CI/CD Pipelines

CI/CD is a natural home for AI because the tasks — reviewing diffs, generating tests, drafting docs — are well-scoped and the feedback is fast. But AI CI tools succeed or fail on one discipline: precision. This tutorial covers the useful integrations and the guardrails that keep them from becoming noise.

The code review bot#

The most valuable CI integration, built in full in the DevAgentic project. Its success is entirely about what it chooses not to say.

private static final String SYSTEM = """
        Review this Java diff for correctness and security bugs only.
        Report only issues you can point at a specific line for, with high
        confidence.
 
        Do NOT report:
        - Style or formatting (a formatter handles that)
        - Preferences without a concrete defect
        - Speculation about code not in the diff
 
        An empty review is a good review.
        """;

Test generation#

Good for the tedious breadth humans skip, but the output is a draft:

// In a CI job on new/changed classes:
String tests = testGenerator.generate(className, sourceCode);
 
// Compile and run them automatically — never propose tests that fail to build.
var result = sandbox.compileAndRun(sourceCode, tests);
if (!result.compiles()) {
    tests = testGenerator.fix(tests, result.errors());
}
// Post as a suggested addition for a human to review, not an automatic commit.
github.proposeTests(pullRequest, tests);

Documentation and release notes#

Lower-risk, genuinely time-saving:

// Summarise a set of merged PRs into draft release notes for human editing.
String draftNotes = summariser.summarise(mergedPullRequests, """
        Write concise release notes grouped by Features, Fixes and Breaking
        Changes. Base them only on the PR titles and descriptions provided.
        """);

Because a human edits the output before it ships, the stakes are low and the time saved is real — one of the safest AI CI integrations to start with.

PR and issue triage#

Route and label incoming work:

@KafkaListener(topics = "github-events")   // or a webhook
public void triage(PullRequestOpened event) {
    Triage triage = triager.classify(event.title(), event.body(), event.diff());
    // Suggestions, applied automatically only for low-stakes actions like
    // labelling — never for closing or merging.
    github.addLabels(event.number(), triage.labels());
    github.suggestReviewers(event.number(), triage.suggestedReviewers());
}

Guardrails for CI bots#

• Scoped access — the repository and task at hand, not blanket write access.
• No autonomous merge or deploy — the bot comments, suggests and drafts; a human decides. See human-in-the-loop systems.
• Bounded cost — a per-run budget so a large diff or a loop cannot run up a bill.
• Handle the untrusted diff — a PR can contain adversarial content; the same prompt-injection caution applies.

Measuring whether it helps#

Track the signals that tell you the bot is earning its place:

• Action rate — what fraction of the bot's suggestions get acted on. The headline metric for a review bot.
• False-positive rate — sampled, to catch drift toward noise.
• Time saved — anecdotal but real for docs and triage.
• Cost per run — so the value clearly exceeds the spend.

If the action rate falls, tighten precision before adding features. A focused bot that catches the occasional real bug beats a chatty one nobody reads.

Next#

• AI observability & LLM tracing
• AI for data engineering
• AI-assisted code generation