Benchmarking

Under the leadership of Fraunhofer FOKUS and with the participation of Fraunhofer ITWM, IAIS, and CML, the DIN specification [VH1] DIN SPEC 91480, “Benchmarking Quantum Computers with Determined KPIs,” has been developed. The document specifies requirements for criteria used to evaluate various quantum computers, quantum algorithms, and their implementations, as well as the implementation itself, based on a standardized set of key performance indicators (KPIs) and associated benchmark specifications. The implementation of these specifications in a benchmark suite is intended to make the developed standard practically applicable. Algorithmic building blocks, application benchmark suites, and low-level benchmarks for characterizing qubits have been evaluated. The work will be incorporated into CENELEC’s standardization activities.

The Fraunhofer IOSB-AST developed a web platform for creating benchmarks of problems, algorithms, and quantum computers. The platform can store benchmarks, generate reports, and take snapshots of quantum computers to document the quality and state of the systems at the time of execution. Furthermore, it enables the easy programming and specification of benchmarks via a browser interface that supports the exchange of algorithms and problem instances.

The Fraunhofer ITWM integrates benchmarking—of both quantum algorithms and quantum hardware—into its existing projects. As part of this effort, quantum computers from AQT, D-Wave, IBM, IQM, IonQ, OQC, Quantinuum, and QuEra, as well as the quantum simulator from QPerfect, were evaluated in terms of their specifications and performance. Furthermore, the algorithms developed in the projects are compared with existing classical and quantum-based methods. As part of the RYMAX project, Fraunhofer ITWM developed a benchmark library for the automated modeling, storage, solution, visualization, and interpretation of real-world optimization problems using classical methods, quantum-based hybrid algorithms, quantum annealing, and specially developed decomposition strategies. Interfaces for real-world data have been implemented, and data from industry partners has been provided for several use cases. The goal is to integrate the RYMAX quantum computer, currently under construction, into the project. This library is supplemented by benchmarking software from the EnerQuant project, which provides instances for the power plant unit commitment problem, modeling approaches for various solvers (Gurobi MIP, Gurobi QUBO, D-Wave MIP, D-Wave QUBO), and parameter optimization for QUBOs.

As part of the Quant²AI project, Fraunhofer ITWM is developing benchmarking software for the DLR to evaluate end-to-end quantum AI pipelines. This project draws primarily on the institute’s expertise in benchmarking HPC and quantum HPC architectures. The goal is to develop software that runs on an HPC system and compares QML pipelines with one another as well as with classical ML models. To this end, a dedicated evaluation module with model-, use-case-, and hardware-specific metrics is being provided.