Your evaluation set is probably contaminated
The benchmark score went up because the answer was already in the training data.
Public benchmarks leak into scraped corpora within weeks. A model that scores well may simply have seen the questions.
The reliable fix is a private holdout that is never published. That makes comparison across labs harder, which is the point.
Contamination is not always fatal. It becomes fatal when a team uses the score to decide what to ship.
#models#evaluation#inference
More in this section
🔒aiSubscriber
Why small models are winning on-device
Four-bit quantization plus better distillation is making 3B models genuinely useful in production.
ai
Synthetic data is hitting a wall nobody wants to name
Model collapse is real, but the more immediate problem is boring: attribution and licensing.
🔒aiSubscriber
The open-weights licence split is becoming permanent
Two families of licence, two incompatible expectations, and a lot of downstream confusion.