---
license: other
license_name: wca-export-terms
license_link: https://www.worldcubeassociation.org/export/results
language:
- en
library_name: sklearn
tasks:
- other
tags:
- benchmark
- sports-analytics
- speedcubing
- wca
- tabular
- time-series
- causal-inference
- machine-learning
pretty_name: WCA-Bench Baselines
---

# WCA-Bench example results

This directory contains baseline results produced on the **official WCA Results Export
(format v2.0.2, export date 2026-09-21)** over a sampled test window.

Everything here is regenerated by the commands in [Reproducing](#reproducing) below.

## Dataset scale (full build)

| Metric | Value |
| --- | --- |
| results | 6,909,454 |
| persons | 298,551 |
| competitions | 18,708 |
| train / validation / test | 3,211,294 / 1,978,851 / 1,719,290 |
| Global attempt-level DNF rate | 3.18% |
| Frozen person–event statistics | 612,224 |
| result_attempts | 31,847,257 |
| scrambles | 3,186,380 |

## Evaluation setup

`scripts/run_all_baselines.py --mode small` samples a handful of competitions inside the test
window and takes a representative subsample of the training split, so that the full baseline
suite completes on a single machine. The numbers below are therefore **indicative, not final** —
the public leaderboard must be re-run on the full dataset under the rolling-window protocol.

| Setting | Value |
| --- | --- |
| Mode | `small` |
| Seed | 42 (single run); 42, 43, 44 (multi-seed) |
| Device | **CUDA** — NVIDIA GeForce RTX 4060 Laptop GPU (8 GB), CUDA 13.0, torch 2.13.0+cu130 |
| Baselines | 21, across 6 method families |

The device is recorded per baseline in every report: pure tabular baselines report
`device: "cpu"`, while the LSTM, the GNN and the boosting baselines report
`device: "cuda:NVIDIA GeForce RTX 4060 Laptop GPU"` together with `gpu_hours`.

## Files

| File | Content |
| --- | --- |
| `leaderboard.md` / `leaderboard.csv` / `leaderboard.json` | Aggregated leaderboard, one section per task |
| `multi_seed.md` | Primary metric as mean ± standard deviation over seeds 42 / 43 / 44 |
| `<task>__<model>.json` | One full report per baseline (21 files) |
| `submission_template/` | Skeleton of a valid leaderboard submission |

Each report uses the same top-level schema:

```json
{
  "task": "...", "model": "...",
  "overall": {...},
  "stratified": {"by_event": {...}, "by_skill_level": {...}, "by_time_slice": {...}, "by_continent": {...}},
  "hard_subset": {...},
  "significance": {"reference": "...", "metric": "...", "paired_unit": "...", "n_pairs": 0,
                   "mean_diff": 0.0, "ci95": [0.0, 0.0], "p_value": 1.0,
                   "effect_size": {"name": "cohens_d", "value": 0.0}, "test": "paired_t", "seed": 42},
  "cost": {"mode": "small", "baseline_kind": "...", "device": "cuda:...",
           "wall_clock_sec": 0.0, "cpu_hours": 0.0, "gpu_hours": 0.0, "gpu_model": "..."},
  "extras": {...}, "elapsed_sec": 0.0, "seed": 42, "data_summary": {...}
}
```

## Leaderboard summary

Single run (seed 42). See [`leaderboard.md`](leaderboard.md) for the complete tables.

| Task | Best baseline | Primary metric | n |
| --- | --- | --- | --- |
| T1 Result prediction | `xgboost_log` | MAE(log) ↓ 0.1639 | 2196 |
| T2 Placement prediction | `kde_simulation` / `plackett_luce` / `psych_sheet` | Kendall τ ↑ 0.7673 | 2260 |
| T3 DNF prediction | `xgboost_dnf` | AUC-PR ↑ 0.3731 | 2260 |
| T4 Human limit estimation | `gp_evt` | leave-one-out stability ↓ 0.9836 | — |
| T5 Skill transfer | `spearman_correlation` | 413 identifiable event pairs | 80000 |

Multi-seed view (see [`multi_seed.md`](multi_seed.md) for the full table):

| Task | Best baseline (mean ± std) | Metric |
| --- | --- | --- |
| T1 | `xgboost_log` 0.1055 ± 0.0417 | MAE(log) ↓ |
| T2 | `psych_sheet` / `plackett_luce` / `kde_simulation` 0.8132 ± 0.0342 | Kendall τ ↑ |
| T3 | `beta_binomial` 0.2938 ± 0.0570 | AUC-PR ↑ |
| T4 | `gp_evt` 0.9836 ± 0.0000 | LOO stability ↓ |
| T5 | `spearman_correlation` 413 | identifiable pairs |

Two findings worth noting:

1. **Domain-aware baselines beat generic deep and graph models.** On T1 the LSTM
   (MAE(log) 0.9436 ± 0.0754) is far behind gradient-boosted trees (0.1055 ± 0.0417); on T2 the
   GNN (τ 0.7440 ± 0.1258) trails the rule-aware Psych Sheet baseline (0.8132 ± 0.0342).
2. **Rank stability depends on the task.** On T1–T2 the multi-seed ordering matches the single
   run. On T3 the single run favours `xgboost_dnf` (0.3731) while the multi-seed mean favours
   `beta_binomial` (0.2938 ± 0.0570 vs 0.2779 ± 0.0724) — the sampling variance of the evaluation
   window is large enough to reorder close contenders, which is exactly why multi-seed reporting
   is mandatory.

## Reproducing

```bash
python -m venv .venv
.venv\Scripts\activate            # Windows
# source .venv/bin/activate       # macOS / Linux
pip install -e ".[dev,fast,boost]"
pip install -e ".[deep]"          # optional: torch for the LSTM / GNN baselines

python scripts/download_data.py                  # fetch the official TSV export
python scripts/build_dataset.py --source raw     # decode, featurize, split, freeze statistics
python scripts/run_all_baselines.py --mode small --device cuda
python scripts/build_leaderboard.py --report-dir outputs/reports --out-dir examples
python scripts/run_multi_seed.py --seeds 42 43 44 --mode small --device cuda \
    --out-dir outputs/multi_seed --examples-dir examples
```

Use `--device cpu` (the default) for a CPU-only, fully reproducible run; `--device auto` selects
CUDA when it is available.

Offline / CI path, using synthetic data:

```bash
python scripts/generate_synthetic.py --small
python scripts/build_dataset.py --source synthetic
python scripts/run_all_baselines.py --mode small
python scripts/build_leaderboard.py
```