All work

2026 · Python · FastAPI · SQLAlchemy

LLM Judge Evaluation

LLM-as-judge that treats position, verbosity, and self-enhancement bias as first-class problems.

Source
Results
MT-Bench · G-Eval · Prometheus
Methods
3 systematic
Bias checks
REST · Streamlit · SDK
Interfaces
4 ready-made
Rubric templates

Problem

Naive LLM judges are unreliable: they favor the first answer, reward verbosity, and prefer their own model family. This framework measures and mitigates those biases instead of ignoring them.

Approach

  • 01Four modes: pointwise scoring, pairwise A/B, reference-based, and concurrent batch.
  • 02Bias mitigation: position swapping, verbosity detection, self-enhancement warnings.
  • 03YAML-driven weighted multi-dimensional rubrics; dual-provider routing across OpenAI + Anthropic.
  • 04SQLAlchemy storage persists token/USD/latency/confidence for real cost analytics.

Architecture

  1. Prompt → judge_engine orchestration
  2. Bias pass · swap / verbosity / self-enh
  3. Rubric manager · YAML weighted
  4. SQLite analytics store

Challenges

  • Making bias measurable — position-swap agreement as a signal, not a vibe.
  • Accurate per-call cost accounting across two providers.

Key features

  • Research-backed methods (MT-Bench, G-Eval, Prometheus)
  • Position / verbosity / self-enhancement bias mitigation
  • Dual-provider routing (OpenAI + Anthropic)
  • YAML weighted rubrics
  • Cost + latency tracking persisted for analytics
Back to all work