QASMBench benchmark suite

The QASMBench benchmark suite [1] was introduced in 2020. This benchmark initiative is led by academics from Pacific Northwest National Laboratory (PNNL), USA. The suite targets low-level benchmarks and is built around OpenQASM as a common assembly-level representation for circuits.

Motivation

The motivation for this benchmark suite is to ease quantum software-hardware co-design by providing a set of methods for the verification and validation of quantum computers. QASMBench is intended to provide a broad benchmark set spanning many domains, while also adding more informative circuit-level figures of merit than width and depth alone.

Architecture

The QASMBench benchmark suite implements a set of quantum algorithms and subroutines in the OpenQASM assembly language to assess the performance of a quantum computer. The benchmark instances are organized in three categories:

Small-scale instances: consist of instances with 2-5 qubits, allowing intensive benchmarking of quantum computers using tomography.
Medium-scale instances: consists of instances from 6-15 qubits.
Large-scale instances: contain benchmark instances with more than 15 qubits for testing large-scale quantum computers.

The QASMBench benchmark suite also introduces 6 figures of merit to better analyze and characterize the performance of quantum computers over quantum algorithms and subroutines. These figures of merit are:

Circuit width: defines the number of qubits used for the quantum circuit.
Circuit depth: length of the critical path (number of sequential gate layers) after decomposition into OpenQASM gates.
Gate density: ratio of executed gates to the total available circuit spacetime (width × depth), indicating how densely operations are packed.
Retention lifespan: defines the longest qubit lifespan during the circuit. For each qubit, the lifespan is the time spent between its first and last uses (gate operation).
Measurement density: ratio of the number of measurements to the circuit spacetime (width x depth), indicating how often final and intermediate measurements are used in the quantum circuit.
Entanglement variance: defines variance in the number of two-qubit gates applied per qubit, where high variance means that groups of qubits are more connected than others.

The 6 different metrics are computed for each instance. The final benchmark score is based on the Hellinger fidelity evaluated from the ideal emulation of the quantum circuit and the actual experimental run on the quantum computer.

Benchmark instances

The following table integrates table 3., 4. and 5. from [1] with description of small-scale instances:

Benchmark	Description	Category	Algorithms	#Qubits	#Gates	CX
adder	Quantum ripple-carry adder	Small-scale	Quantum Arithmetic	4	23	10
basis_change	Transform the single-particle baseis of an linearly connected electronic structure	Small-scale	Quantum Simulation	3	53	10
basis_trotter	Implement Trotter steps for molecule LiH at equilibrium geometry	Small-scale	Quantum Simulation	4	1626	582
bell_state	Bell State	Small-scale	Logical Operation	2	3	1
cat_state	Cat State	Small-scale	Logical Operation	4	4	3
deutsch	Deutsch algorithm with 2 qubits for f (x ) = x	Small-scale	Hidden Subgroup	2	5	1
dnn	Quantum Deep Neural Network	Small-scale	Quantum Machine Learning	2	268	84
fredkin_n3	Fredkin gate benchmark	Small-scale	Logical Operation	3	19	9
qec_dist3	Error correction with distance 3 and 5 qubits	Small-scale	Error Correction	5	114	49
grover	Grover’s algorithm	Small-scale	Search and Optimization	2	16	2
hs4	Hidden subgroup problem	Small-scale	Hidden Subgroup	4	28	4
inverseqft	Performs an exact inversion of quantum Fourier tranform	Small-scale	Hidden Subgroup	4	8	0
iSWAP	An entangling swapping gate	Small-scale	Logical Operation	2	9	2
linearsolver	Solver for a linear equation of one qubit	Small-scale	Linear Equation	3	19	4
lpn	Learning parity with noise	Small-scale	Machine Learning	5	11	2
pea	Phase estimation algorithm	Small-scale	Hidden Subgroup	5	98	42
qaoa	Quantum approximate optimization algorithm	Small-scale	Search and Optimization	3	15	6
qec_sm	Repetition code syndrome measurement	Small-scale	Error Correction	5	5	4
qec_en	Quantum repetition code encoder	Small-scale	Error Correction	5	25	10
qft	Quantum Fourier transform	Small-scale	Hidden Subgroup	4	36	12
qrng	Quantum Random Number Generator	Small-scale	Quantum Arithmetic	4	4	0
quantumwalks	Quantum walks on graphs with up to 4 nodes	Small-scale	Quantum Walk	2	11	3
shor	Shor’s algorithm	Small-scale	Hidden Subgroup	5	64	30
toffoli	Toffoli gate	Small-scale	Logical Operation	3	18	6
teleportation	Quantum Teleportation	Small-scale	Quantum Communication	3	8	2
jellium	Variational ansatz for a Jellium Hamiltonian with a linear-swap network	Small-scale	Quantum Simulation	4	54	16
vqe_uccsd	Variational Quantum Eigensolver with UCCSD ansatz	Small-scale	Search and Optimization	4	220	88
wstate	W-state preparation and assessment	Small-scale	Logical Operation	3	30	9
dnn	Quantum Deep Neural Network	Medium-scale	Quantum Machine Learning	8	1200	384
adder	Quantum ripple-carry adder	Medium-scale	Quantum Arithmetic	10	142	65
bb84	A quantum key distribution circuit	Medium-scale	Quantum Communication	8	27	0
bv	Bernstein-Vazirani Algorithm	Medium-scale	Hidden Subgroup	14	41	13
ising	Ising model simulation via QC	Medium-scale	Quantum Simulation	10	480	90
multipler	Quantum multipler	Medium-scale	Quantum Arithmetic	15	574	246
multiply	Performing 3×5 in a quantum circuit	Medium-scale	Quantum Arithmetic	13	98	40
qaoa	Quantum approximate optimization algorithm	Medium-scale	Search and Optimization	6	270	54
qf21	Using quantum phase estimation to factor the number 21	Medium-scale	Hidden Subgroup	15	311	115
qft	Quantum Fourier transform	Medium-scale	Hidden Subgroup	15	540	210
qpe	Quantum phase estimation algorithm	Medium-scale	Hidden Subgroup	9	123	43
sat	Boolean satisfiability problem via QC	Medium-scale	Searching and Optimization	11	679	252
seca	Shor’s error correction algorithm for teleportation	Medium-scale	Error Correction	11	216	84
simons	Simon’s algorithm	Medium-scale	Hidden Subgroup	6	44	14
vqe_uccsd	Variational quantum eigensolver with UCCSD	Medium-scale	Linear Equation	8	10808	5488
adder	Quantum ripple-carry adder	Large-scale	Quantum Arithmetic	127	3991	910
bigadder	Quantum ripple-carry adder	Large-scale	Quantum Arithmetic	18	284	130
bv	Bernstein-Vazirani algorithm	Large-scale	Hidden Subgroup	19	56	18
cat_state	Cat State	Large-scale	Logical Operation	100	100	99
cc	Counterfeit coin finding problem via QC	Large-scale	Hidden Subgroup	18	34	17
ghz_state	GHZ State preparation and assessment	Large-scale	Logical Operation	23	23	22
ising	Ising model simulation via QC	Large-scale	Quantum Simulation	26	280	50
ising	Ising model simulation via QC	Large-scale	Quantum Simulation	60	654	118
multipler	Quantum multipler	Large-scale	Quantum Arithmetic	25	3723	750
qft	Quantum Fourier tranform	Large-scale	Hidden Subgroup	20	970	380
qft	Quantum Fourier tranform	Large-scale	Hidden Subgroup	85	17935	7140
square_root	Computing the square root of a number via amplitude amplification	Large-scale	Quantum Arithmetic	18	4640	898
swap_test	SWAP Test algorithm implementation	Large-scale	Logical Operation	25	446	96
wstate	W-state preparation and assessment	Large-scale	Logical Operation	27	446	96

Devices being benchmarked

The framework permits connections to different types of quantum computers:

Rigetti superconducting quantum systems via Qiskit provider
IBM superconducting quantum systems via IBM Qiskit.
IonQ trapped-ion quantum systems via Microsoft Azur.
Quantinuum trapped-ion quantum systems via Microsoft Azur.

Limitation

The QASMBench benchmark suite focuses on low-level benchmarks and hence on small quantum systems. Because it relies on the computation of the Hellinger fidelity, it is not suitable for generalization to large instances. Other figures of merit than the Hellinger fidelity should be used for large-scale quantum circuits.

The figures of merit defined in the benchmark suite are computed according to the OpenQASM standard gate set. It might not be representative of real circuits run on real physical quantum computers.

Implementation

The QASMBench source code is open source and was last updated on 20/01/2025.

References

[1]A. Li, S. Stein, S. Krishnamoorthy, and J. Ang, “Qasmbench: A low-level quantum benchmark suite for nisq evaluation and simulation,” ACM Transactions on Quantum Computing, vol. 4, no. 2, pp. 1–26, 2023.