Model evaluation · 2026 · Solo — evaluation design, statistical methods, API boundaries, interaction design, testing, deployment
AI Code Judge
A public evaluation instrument that turns paired task outcomes into bounded, reproducible model-comparison evidence without hiding uncertainty.
The problem
Code-model comparisons are easy to overstate. A single pass rate discards repeated-sampling behavior, independent confidence intervals ignore shared task difficulty, and a leaderboard number rarely exposes the seed, sample size, or statistical boundary behind the claim. AI Code Judge combines an executable weighted rubric with an evaluation workbench for paired Boolean task outcomes, making the estimator, uncertainty, reproducibility controls, and limitations visible before a reviewer accepts a winner.
Architecture
Key decisions
Preserve task pairing
Both systems are resampled on the same task indices. The paired percentile bootstrap retains shared task difficulty instead of treating two correlated result vectors as independent samples.
Use the established repeated-sampling estimator
The workbench reports unbiased pass@k with the standard without-replacement combinatorial estimator. It distinguishes the probability of at least one correct generation from the single-sample pass rate.
Make reproducibility and cost explicit
Confidence, bootstrap count, sample size, and seed travel with every result. Request limits cap the public API at 500 paired tasks and 20,000 bootstrap resamples so deterministic analysis cannot become unbounded compute.
State what the interval cannot prove
The percentile interval characterizes uncertainty on the supplied task set. It does not prove generalization to another benchmark, deployment population, prompt, or model version, and bootstrap frequency is not presented as a Bayesian posterior.
Separate statistical analysis from code execution
The public endpoint accepts Boolean outcomes only and does not execute uploaded code. Candidate execution remains a local evaluator concern, preventing the Vercel surface from becoming a remote code-execution service.