Quantum computing benchmarks

Please create an issue in the metriq-data GitHub repository.

Yes, in fact everyone is encouraged to. Please use the upload command from the metriq-gym CLI.

The device ibm_torino from IBM Quantum Cloud is used as a consistent reference point for the derived “score” scale (so 100 means “on par with the baseline”); it was chosen because it is a widely-run, stable platform that provides good coverage across benchmarks. Note this choice does not bias the data: raw benchmark metrics are unchanged, and the baseline is only an anchor for normalization/visual reference (and can be changed in the configuration of the dataset). Picking a different baseline would primarily rescale the scores, but the relative comparisons between platforms would remain the same.

Metriq draws inspiration from established “classical” benchmarking efforts like Geekbench for CPU/GPU, and MLCommons and Epoch AI for AI/ML models, adapted to the needs and constraints of quantum hardware and workflows.

The original vision of Metriq started in 2022 as a website to crowd-source and share benchmarking results ingested from the literature in a wiki-like website. However, we found that this approach did not fully address the challenges of fragmentation and reproducibility in quantum benchmarking. As a result, we evolved Metriq into a more integrated platform that combines benchmark execution, data collection, and public presentation into a unified workflow. In the future, we may explore ways to incorporate literature results in a way that complements the core focus on executable benchmarks and reproducibility, but for now we are prioritizing building out the integrated workflow and encouraging direct contributions of benchmark results through the metriq-gym CLI.