How the Code Golf Benchmark Works
Models solve small programming holes. Scores reward correct, tiny source files and penalize failures without throwing away partial progress.
The Basic Loop
A course defines the holes
Each course is an ordered set of tasks. The Python, Racket, and Rust courses each cover the same 18 holes, with language-specific submission contracts and runners.
The prompt shows examples
The generated prompt includes the task description and public tests. Private tests stay out of the prompt and are used during published scoring.
The model returns raw source
The final answer is saved as the submitted file. The benchmark does not format, normalize, or rewrite the source before counting or running it.
The runner executes the submission
Python submissions define a function named f. Racket submissions define a procedure named f. The harness calls that entrypoint for every selected test case and compares the returned value to the expected output.
The scorer counts bytes and failures
The byte count is the raw UTF-8 length of the submitted file. Passing all tests means the score is exactly that byte count. Failed or partial submissions keep their byte count but receive a correctness penalty.
Scoring
Lower is better, matching normal code golf. Each hole has a penalty budget, usually 1000 bytes. The scorer computes correctness from test pass rate, then adds a squared penalty for the missing correctness.
incorrectness = 1 - correctness
score = raw_bytes + penalty_bytes * incorrectness^2
A fully correct 31-byte submission scores 31. A short partial solution can receive credit, but only if its small byte count outweighs the penalty for failed tests. Missing submissions count as zero correctness for that hole.
Course and Leaderboard Scores
A course score is the sum of its hole scores. The leaderboard keeps the best result for each model on each course, preferring lower score, then more holes and tests passed, then fewer raw bytes.
The combined view ranks models that have results for every available course by total score. Course tabs rank models within one language/course. Selecting a model opens course-level results, and selecting a course shows hole-by-hole detail.