Quantum computing benchmarks

Please create an issue in the metriq-data GitHub repository.

Yes, in fact everyone is encouraged to. Please review the metriq-data dataset scope and upload policy, then use the upload command from the metriq-gym CLI.

The baseline published by metriq-data is used as a consistent reference point for the derived “score” scale, so 100 means “on par with the baseline.” The selection is maintained alongside the scoring rules, with benchmark coverage as a key consideration. It does not alter raw benchmark metrics; it only provides an anchor for normalization and visual comparison. Within an individual metric, changing the baseline rescales values without changing their ordering; composite scores can shift because their components are normalized independently.

Metriq draws inspiration from established “classical” benchmarking efforts like Geekbench for CPU/GPU, and MLCommons and Epoch AI for AI/ML models, adapted to the needs and constraints of quantum hardware and workflows.

The original vision of Metriq started in 2022 as a website to crowd-source and share benchmarking results ingested from the literature in a wiki-like website. However, we found that this approach did not fully address the challenges of fragmentation and reproducibility in quantum benchmarking. As a result, we evolved Metriq into a more integrated platform that combines benchmark execution, data collection, and public presentation into a unified workflow. In the future, we may explore ways to incorporate literature results in a way that complements the core focus on executable benchmarks and reproducibility, but for now we are prioritizing building out the integrated workflow and encouraging direct contributions of benchmark results through the metriq-gym CLI.