feat: add Runner, RunnerResult, Judge, and Evaluator

[mattrmc1]

Summary

Adds the AIEVALS types — Runner, RunnerResult, Judge, and Evaluator — and wires Evaluator.noop() into all config types. Callers can now implement a Runner to wrap any model provider, construct a Judge to evaluate AI outputs against a judge prompt with structured {score, reasoning} output, and coordinate multiple judges through an Evaluator.

New types

public interface Runner {
  RunnerResult run(String input, Map<String, Object> outputType) throws Exception;
  default RunnerResult run(String input) throws Exception;
}

Wraps a model provider SDK. outputType carries a JSON-Schema-like map when structured output is needed. Single-arg overload delegates with outputType = null.

RunnerResult.builder(String content, AIMetrics metrics)
    .raw(Object raw)
    .parsed(Map<String, Object> parsed)
    .build();

Immutable result of a Runner invocation. parsed is defensively copied and returned as unmodifiable.

public Judge(AIJudgeConfig config, Runner runner, LDLogger logger);

JudgeResult evaluate(String input, String output);
JudgeResult evaluate(String input, String output, double samplingRate);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response, double samplingRate);

Evaluates AI output by invoking a runner with a formatted evaluation prompt and parsing the structured response. Sampling gate runs first — below the rate, returns sampled=false immediately. Creates a fresh tracker per evaluation via config.createTracker(). Parses score (Number, [0.0, 1.0]) and reasoning (String, optional). Runner exceptions are caught and returned as JudgeResult(success=false) — judge failures are results, not exceptions. Does not call trackJudgeResult.

public static Evaluator noop();
public Evaluator(Map<String, Judge> judges, JudgeConfiguration judgeConfiguration, LDLogger logger);

CompletableFuture<List<JudgeResult>> evaluate(String input, String output);

Coordinates sequential execution of judges. Missing judges skipped with a warning. Evaluator.noop() returns a singleton whose evaluate immediately returns an empty list. For v1.0, all configs receive Evaluator.noop().

Config type changes

AIConfig base class gains an Evaluator field and getEvaluator() accessor. AICompletionConfig and AIAgentConfig constructors accept an Evaluator. AIJudgeConfig always wires Evaluator.noop() internally — judges do not evaluate themselves.

Test plan

• ./gradlew :lib:sdk:server-ai:test passes
• JudgeTest — successful evaluation, score boundary validation (0 and 1), reasoning optional, runner exception handling (caught not rethrown), null/missing parsed output, score out of range, sampling rates (0 always skips, 1 always runs), message formatting, getter accessors
• EvaluatorTest — noop returns empty list, noop singleton identity, single/multiple judge execution, missing judge skipped, evaluator does not call trackJudgeResult, returned future is already complete
• RunnerResultTest — builder field assignment, immutability, defensive copy of parsed map

---

Note

Medium Risk
Adds new public SDK types and extends AIConfig construction, but v1.0 retrieval still uses Evaluator.noop(); main risk is future judge runs invoking customer Runner implementations and extra model calls when fully wired.

Overview
Introduces AI evaluation plumbing in server-ai: a Runner abstraction for provider calls, immutable RunnerResult, a Judge that scores model output via structured {score, reasoning} responses (with sampling, validation, and metrics via trackMetricsOf), and an Evaluator that runs configured judges sequentially and returns a completed CompletableFuture.

Config wiring: AIConfig now carries an Evaluator exposed through getEvaluator(). Completion and agent configs accept an evaluator at construction; LDAIClientImpl currently passes Evaluator.noop() everywhere so retrieval behavior stays a no-op until later integration. Judge configs always use Evaluator.noop() internally.

Unit tests cover judge success/failure paths, evaluator noop and multi-judge behavior, and RunnerResult immutability.

^{Reviewed by Cursor Bugbot for commit f42de0b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit faa4981. Configure here.}