SupermarQ: a hardware-agnostic quantum benchmark
The SupermarQ benchmark suite [1] was initially proposed in 2022 and defines a framework and a set of instances for benchmarking quantum computers using application-level figures of merit. The company Infleqtion is still updating this framework, which is part of the Superstarq quantum software platform.
Motivation
The motivation for this benchmark suite is to compare the performance of different quantum computer architectures in a hardware-agnostic way. This benchmark draws inspiration from the classical benchmark suite PARSEC [2], which was designed to evaluate multithreaded applications. SupermarQ benchmark suite also introduces a list of criteria used to quantify how the benchmark set stresses the quantum process unit.
Architecture
The SupermarQ benchmark suite is structured around a set of quantum subroutines and applications that collectively ensure a representative coverage of quantum workloads. Each benchmark is described by a feature vector capturing different aspects of how a program stresses the quantum processor. The following features are used to evaluate the coverage of the benchmark suite:
- Communication (CO): Captures how different qubits of the system need to interact with each other when executing the quantum circuit.
- Critical depth (CD): Reflects how the program is serialized.
- Entanglement ratio (E): Measures how much the program relies on multi-qubit gates compared to single-qubit gates.
- Parallelism (P): Indicates the amount of parallelism available in the quantum program (quantum operations occurring simultaneously).
- Liveness (L): Describes how actively qubits are processed by quantum operations versus sitting idle during execution.
- Measurement (M): Tracks how often intermediate measurements are used, which can introduce additional noise. This feature is primarily used for error-correction benchmarks.
These features, normalized to \([0, 1]\), produce a 6-dimensional space used to quantify the coverage of a benchmark set. This coverage can then be represented as a radar chart:
Each benchmark is reported with a benchmark score that depends on the type of application/quantum routine being benchmarked, along with the instance’s coverage. The authors then assess the Pearson correlation coefficient between the score and each feature used to define the coverage.
Benchmark instances
The framework includes both quantum subroutine and application-oriented benchmarks.
Quantum sub-routines
The benchmark suite involves the test of a series of quantum computing sub-routines:
- GHZ state generation: assesses the system’s ability to produce multipartite entangled states, with the Hellinger fidelity serving as the benchmark.
- Mermin-Bell benchmark: builds on GHZ states to validate the device’s quantum behavior, using the expectation value of the Mermin operator to compute the benchmark score.
- Error correction: evaluates error-correction routines such as bit flip and phase flip repetition codes, using the Hellinger fidelity as the benchmark score.
Application benchmark
The benchmark suite also involves tests on a series of algorithms used for applications:
- QAOA: builds on Ising problems such as the Sherrington-Kirkpatrick and Max-Cut problems, where the expectation value of the experiment is compared to the expectation value of an ideal noiseless emulation to set the benchmark score.
- VQE: encodes the 1D transverse-field Ising model to find its ground-state energy. This benchmark uses the same score as for the QAOA.
- Hamiltonian simulation: encodes the 1D transverse-field Ising model using Trotterization and compares the expectation value of the experiment and the expectation value of an ideal noiseless emulation to set the benchmark score.
Extensions
The SupermarQ benchmark suite has been extended to include other system-level benchmarks such as Cross-entropy benchmarking (XEB), Interleaved Randomized Benchmarking (IRB), Symmetric Stabilizer Benchmarking [3] and SU(2) benchmarking.
Devices being benchmarked
The framework permits connections to different types of quantum computers:
- IonQ trapped-ion quantum systems via Amazon Braket.
- IBM superconducting quantum systems via IBM Qiskit.
- AQT trapped-ion quantum systems.
Simulators:
Implementation
The SupermarQ framework source code is open source and was last updated on 30/07/2025.
The instances used in the initial publication of the benchmark suite are available on Zenodo.
References
- [1]T. Tomesh et al., “Supermarq: A scalable quantum benchmark suite,” in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), IEEE, 2022, pp. 587–603.
- [2]C. Bienia, S. Kumar, J. P. Singh, and K. Li, “The PARSEC benchmark suite: characterization and architectural implications,” in Proceedings of the 17th international conference on Parallel architectures and compilation techniques, in PACT ’08. ACM, Oct. 2008, pp. 72–81. doi: 10.1145/1454115.1454128. Available at: http://dx.doi.org/10.1145/1454115.1454128
- [3]R. B.-S. Tsai, X. Sun, A. L. Shaw, R. Finkelstein, and M. Endres, “Benchmarking and fidelity response theory of high-fidelity Rydberg entangling gates,” PRX Quantum, vol. 6, no. 1, p. 010331, 2025.