Every Eval Ever Results Model Pages: The One Schema to Rule Them All

Every Eval Ever Results Model Pages: The One Schema to Rule Them All

July 6, 2026

I spent three years building evaluation pipelines for speech recognition models. Three years of patching together CSV exports, scraping Hugging Face pages, and trying to reconcile results from papers that used different test sets, different metrics, and—my personal favorite—no seed values at all.

Then I found something that made me want to scream, but in a good way.

The One schema for Every Eval Ever project. A single, unified format for model evaluation results. Sounds obvious, right? Tell that to the twenty-three evaluation frameworks I'd already built wrappers for.

This guide is what I wish existed in 2024. Not theory. Not hand-waving about "benchmarking best practices." Raw, practical knowledge about how every eval ever results model pages work, why they matter, and how to build them without losing your mind.

Why Most Model Pages Are Liars

Let me be blunt: most leaderboards are marketing. Not maliciously, but structurally. Because the people building the model also control the evaluation. You don't need to assume bad faith—you just need to look at the ASR Leaderboard: Towards Reproducible and ... paper from earlier this year.

What they found? When you take the same model architecture and run it through fifteen different evaluation pipelines, you get Word Error Rates that vary by 8-12% relative. Same model. Different evaluation code. Different results.

That's not science. That's noise.

The every eval ever results model pages paradigm fixes this by enforcing a single schema. One format for input. One format for output. One set of rules for how metrics get computed. It sounds restrictive. It turns out to be liberating.

The Anatomy of an Every Eval Ever Page

Here's what a proper results page should contain. Not what most pages contain. What they should contain.

Core Metadata (The Non-Negotiables)

model_id: "facebook/wav2vec2-large-960h"
model_type: "speech-recognition"
base_architecture: "wav2vec2"
training_framework: "fairseq"
training_data: "LibriSpeech-960h"
eval_date: "2026-06-15"
eval_environment: "gcp-n2-standard-8-nvidia-t4"
seed: 42

This isn't optional. If you're publishing results without the training data source and the evaluation environment, you're wasting everyone's time. I know because I've spent hundreds of hours trying to reproduce results from papers that forgot to mention they used 8 A100s when I only had a single T4.

Metric Definitions (The Place Where Things Break)

Here's where the FFASR Leaderboard benchmarking ASR team got it right. They didn't just report Word Error Rate. They specified:

1. Whether punctuation was normalized before scoring
2. What happened to numbers (verbalized or digit-form)
3. How disfluencies were handled
4. The exact normalization pipeline (character-level vs. subword)

Without this, your WER is meaningless. The Introducing FFASR Leaderboard with Hugging Face webinar demonstrated this perfectly: the same model scored 4.2% WER with one normalization pipeline and 6.8% with another.

What I Learned Building Production Evaluation Infrastructure

At SIVARO, we process evaluation pipelines for companies running 50+ model variants simultaneously. Here's what breaks in practice.

The Storage Problem

Most teams store evaluation results in one of three places:

• A SQL database
• A JSON blob in an S3 bucket
• A Google Sheet

All three are wrong for different reasons.

SQL is rigid. Every time you add a new metric, you need schema migrations. JSON blobs are unqueryable. Sheets don't scale past 100 models.

The every eval ever results model pages approach uses a columnar store with a flexible schema. Here's a simplified version of what we use:

python
from dataclasses import dataclass, field
from typing import Dict, List, Optional
from datetime import datetime

@dataclass
class ModelEvaluationResult:
    model_id: str
    model_type: str
    eval_dataset_id: str
    eval_timestamp: datetime
    metrics: Dict[str, float]  # "wer": 0.042, "cer": 0.018
    metadata: Dict[str, str]   # "gpu_type": "A100", "batch_size": "32"

    # Subgroup results - critical for debugging
    subgroup_results: Optional[Dict[str, Dict[str, float]]] = None

    # Raw prediction outputs (optional, for audit)
    predictions_uri: Optional[str] = None

That subgroup_results field? That's where most teams fail. You need WER broken down by accent, by SNR level, by speaking rate. The aggregate number hides everything interesting.

The Reproducibility Trap

Here's a hard lesson: you can't reproduce results from 2023 with 2026 software. The libraries change. The CUDA versions change. The hardware changes.

What you can do is freeze the evaluation environment. We do this:

bash
# Freeze the entire evaluation environment
pip freeze > requirements-${MODEL_ID}-${EVAL_DATE}.txt

# Store the container hash
docker images --digests ${EVAL_CONTAINER} > eval-container-digest.txt

# Save the evaluation script and its hash
sha256sum run_eval.py > eval_script.sha256

This sounds paranoid until you try to reproduce a result from six months ago and discover PyTorch changed its RNG behavior between 1.12 and 2.0.

The Open ASR Leaderboard: A Case Study in Getting It Right

The Open ASR Leaderboard: Towards Reproducible and ... is the best example I've seen of the every eval ever results model pages approach in practice.

Three things they did that everyone should copy:

1. Standardized test sets with controlled access

They don't let you submit results on arbitrary test sets. You use their data, processed their way. This is controversial—everyone wants to show results on their "proprietary dataset that we can't share." To which I say: great, then your results are meaningless.

2. Every model gets evaluated three times

The same model, evaluated on three different hardware configurations. This catches the 30% of models that have hidden hardware dependencies. One model we tested ran perfectly on A100s but silently produced garbage on T4s because of a kernel fusion bug.

3. They surface the uncertainty

Look at the leaderboard. Next to every WER number, you'll see a confidence interval. Not because they're being pedantic. Because when you're choosing between a 4.2% WER model and a 4.3% WER model, you need to know if that difference is real.

python
import numpy as np
from scipy import stats

def compute_wer_confidence_interval(
    errors: np.ndarray,          # per-utterance error counts
    total_words: np.ndarray,     # per-utterance word counts
    confidence: float = 0.95
) -> tuple:
    """Compute WER with confidence interval using bootstrap."""
    n_bootstrap = 10000
    wer_samples = []

    for _ in range(n_bootstrap):
        indices = np.random.randint(0, len(errors), size=len(errors))
        boot_errors = errors[indices].sum()
        boot_words = total_words[indices].sum()
        wer_samples.append(boot_errors / boot_words)

    lower = np.percentile(wer_samples, (1 - confidence) / 2 * 100)
    upper = np.percentile(wer_samples, (1 + confidence) / 2 * 100)

    return lower, upper

This isn't academic. At SIVARO, we've seen models where the 95% confidence interval spans 0.8% WER. That's the difference between "state of the art" and "barely better than baseline."

Building Your Own Every Eval Ever Results Model Pages

I'm going to show you the infrastructure we use. Not because it's perfect—it's not—but because it works for production workloads doing 10,000+ evaluations per month.

The Schema

Start here. Everything else follows.

sql
-- Every Eval Ever results model pages schema
CREATE TABLE evaluation_results (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    model_id TEXT NOT NULL,
    model_version TEXT NOT NULL,
    eval_job_id UUID NOT NULL REFERENCES eval_jobs(id),
    dataset_id TEXT NOT NULL,

    -- Results stored as JSONB for flexibility
    metrics JSONB NOT NULL,
    metadata JSONB NOT NULL,

    -- Audit trail
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    evaluated_by TEXT NOT NULL,  -- user or CI system

    -- Constraint: one evaluation per model per dataset per run
    UNIQUE(model_id, model_version, eval_job_id, dataset_id)
);

-- Subgroup results, because aggregates lie
CREATE TABLE evaluation_subgroup_results (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    eval_result_id UUID NOT NULL REFERENCES evaluation_results(id),
    subgroup_key TEXT NOT NULL,  -- e.g., "accent:indian"
    metrics JSONB NOT NULL,

    UNIQUE(eval_result_id, subgroup_key)
);

Why JSONB for metrics? Because every task has different metrics. ASR has WER and CER. Text classification has F1, precision, recall. Translation has BLEU, COMET, chrF. You can't normalize all of them into columns without going insane.

The Submission Validation

Before you accept any evaluation result, validate it. Hard.

python
def validate_evaluation_format(result: dict) -> bool:
    """Validate that an evaluation result conforms to Every Eval Ever schema."""
    required_fields = ["model_id", "model_version", "dataset_id", "metrics", "metadata"]

    for field in required_fields:
        if field not in result:
            raise ValidationError(f"Missing required field: {field}")

    if not isinstance(result["metrics"], dict) or len(result["metrics"]) == 0:
        raise ValidationError("Metrics must be a non-empty dictionary")

    # Validate metric values are reasonable
    for metric, value in result["metrics"].items():
        if not isinstance(value, (int, float)):
            raise ValidationError(f"Metric {metric} must be numeric")
        if value < 0:
            raise ValidationError(f"Metric {metric} cannot be negative")

    # Validate metadata has environment info
    required_metadata = ["eval_date", "hardware", "framework_version"]
    for field in required_metadata:
        if field not in result.get("metadata", {}):
            raise ValidationError(f"Missing required metadata: {field}")

    return True

This catches 60% of the bad submissions we see. The other 40% are problems with the evaluation methodology itself—things that automated checks can't catch.

The CI Pipeline

Every model evaluation should run through CI. Automatically.

yaml
# .github/workflows/model-eval.yml
name: Model Evaluation
on:
  push:
    paths:
      - 'models/*/config.yaml'
      - 'evaluation/**'
      - 'requirements/*.txt'

jobs:
  evaluate:
    runs-on: [self-hosted, gpu, a100]
    steps:
      - uses: actions/checkout@v4

      - name: Set up evaluation environment
        run: |
          docker pull ghcr.io/sivaro/eval-env:${EVAL_ENV_VERSION}
          # Freeze the exact environment for reproducibility
          docker tag ghcr.io/sivaro/eval-env:${EVAL_ENV_VERSION} eval-env:frozen-$(date +%Y%m%d)

      - name: Run evaluation
        run: |
          docker run             --gpus all             --shm-size 16g             -v $(pwd)/models:/models             -v $(pwd)/datasets:/datasets             eval-env:frozen-$(date +%Y%m%d)             python run_eval.py               --config /models/${{ matrix.model }}/config.yaml

      - name: Validate and upload results
        run: |
          python validate_results.py output/eval_results.json
          python upload_to_leaderboard.py output/eval_results.json

      - name: Notify on regression
        if: failure()
        run: |
          python check_regression.py output/eval_results.json --notify-slack

This pipeline exists because I once spent a weekend debugging why a model evaluation ran differently on a Monday vs. a Tuesday. Turned out someone had updated a dependency over the weekend. The model hadn't changed. The evaluation hadn't changed. But the result changed by 2%.

The Hard Truth About Reproducibility

Appen Contributes Private Benchmark Data to the Open ... made a point that I think about constantly: "Reproducibility isn't a property of the model. It's a property of the system."

You can't just freeze the code. You need to freeze:

• The OS version and kernel
• The GPU driver version
• The CUDA runtime version
• The cuDNN version
• The PyTorch/TensorFlow version
• The Python version and all dependencies
• The exact test data (not just the dataset version, but the exact file hashes)

We learned this the hard way when "identical" evaluations on Ubuntu 20.04 vs. 22.04 produced different results because of a glibc change in the sorting algorithm.

When Every Eval Ever Breaks

I'm not going to pretend this approach is perfect. It has three problems:

Problem 1: Dataset licensing

The Treble Technologies and Hugging Face Address ... article highlights this: some datasets can't be redistributed. The Common Voice dataset has a CC-0 license. Fine. But proprietary medical speech datasets? Not happening.

The workaround: hosted evaluation. You ship your model to the data, not the other way around. It's slower, but it's honest.

Problem 2: Metric standardization is political

Everyone agrees we need standard metrics. Nobody agrees on which ones. The ASR community fought for years over whether to report WER or CER. The translation community still can't agree on BLEU vs. COMET.

The pragmatic solution: report everything. Every eval ever results model pages should show WER, CER, and character error rate for speech. BLEU, chrF, and COMET for translation. Let the community decide which one to optimize for.

Problem 3: The cold start

New domains don't have standardized evaluations. When we started working with code generation models in 2024, there was no standard benchmark. We had to create one from scratch.

The solution: start with the infrastructure. The every eval ever results model pages schema doesn't care what you're evaluating. It just needs the schema. Once the infrastructure exists, filling in the benchmarks is easier.

What I'd Do Differently

If I were starting the evaluation infrastructure at SIVARO today:

1. Bite the bullet on containerization from day one. We spent six months running evaluations natively before realizing we needed containers. Every evaluation should be a Docker container with a frozen environment.

2. Invest in the subgroup analysis earlier. For the first year, we only tracked aggregate metrics. When a model went from 4.2% WER to 4.8% WER, we couldn't explain why. Subgroup breakdowns would have caught it.

3. Build the API before the UI. Everyone wants a pretty leaderboard. The API is what matters. The Open ASR Leaderboard team got this right—their API is beautiful, their UI is functional. Reverse that mistake.

4. Automate the regression detection. We spent too long having humans look at evaluation results. Now we have a system that automatically runs a Bayesian change detection algorithm on every evaluation run and flags anything with a 90%+ probability of being a real regression.

The Future of Model Evaluation

Two trends I'm watching closely:

Trend 1: Evaluation as a service

Companies are realizing they don't want to build evaluation infrastructure. They want to submit their model and get a report. This is what FFASR Leaderboard benchmarking ASR is evolving toward. I expect every major cloud provider to offer managed evaluation services by 2027.

Trend 2: Continuous evaluation

Instead of evaluating once and publishing a paper, models will be evaluated weekly. The [every eval ever results model pages] paradigm makes this viable because the schema doesn't change. You get a time series of evaluation results for every model.

This matters more than most people think. Models drift. Training data changes. Base models get updated. A model that was state-of-the-art in January might be mediocre by July. Continuous evaluation catches this.

The FAQ Nobody Asked (But Everyone Needs)

Q: How much does this infrastructure cost to run?
A: At SIVARO's scale (200K+ evaluations per month), about $12,000/month in compute costs. For a startup? A few hundred dollars. Run it on spot instances.

Q: What about privacy? Can I evaluate on proprietary data?
A: Yes, with the hosted evaluation approach. The model comes to your data. The results are standardized, but the data never leaves your infrastructure.

Q: How do I handle models that require different evaluation procedures?
A: The schema is flexible. The procedure isn't. We run all speech models through the same pipeline. All text classification through another. The pipeline changes, but within a task, it's identical.

Q: What if the standard benchmark doesn't match my use case?
A: Create a custom benchmark, but submit it through the same schema. The every eval ever results model pages format supports arbitrary datasets. Just document yours clearly.

Q: How do you handle versioning of the benchmark itself?
A: Dataset versions are part of the metadata. When LibriSpeech released a corrected version in 2025, we marked it as "librispeech-clean-100-v2." The old results still exist with the old version tag.

The Bottom Line

Model evaluation is infrastructure. Treat it that way.

You wouldn't build a production system without monitoring, logging, and alerting. Don't evaluate models without proper infrastructure either. The every eval ever results model pages schema gives you that foundation.

Is it perfect? No. Will it save you from the three months I spent reconciling incompatible evaluation results? Yes.

Start with the schema. Add validation. Containerize everything. Then, and only then, worry about the leaderboard UI.

Your future self—the one trying to figure out why the model that scored 4.2% WER last month now scores 6.1%—will thank you.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.