On March 19th, Hyperledger, a global collaboration hosted by The Linux Foundation, announced Caliper has been accepted by the Technical Steering Committee as a Hyperledger project. Hyperledger Caliper is a benchmark tool that allows blockchain projects to consistently track performance characteristics across different blockchain implementations. The initial release of Hyperledger Caliper focuses on transactions per second (TPS), latency and resource utilization.
Hyperledger Caliper plans to provide TPS, latency and resource utilization results so other Hyperledger projects can use this information as they build out their frameworks. This will assist other projects in choosing the right blockchain implementation for their specific use cases.
Low TPS is a common criticism of blockchain implementations. However, by not having a universal tool to benchmark performance, this doesn't improve that situation. Hyperledger aims to address how blockchains are benchmarked and ensure common baselines can be established:
Clearly, every blockchain framework or platform proposed nowadays has its own unique strengths. In the light of such variety, we consider the absence of a common benchmarking tool as harmful.
Hyperledger also cites three core issues and why a general purpose tool is necessary:
- Lack of source code - There are some performance reports for different projects, but since they do not provide the source code, it is hard to validate the evaluation and impossible to perform the same evaluation on different projects.
- There is no common definition of performance indicators - The Hyperledger Performance & Scalability Working Group (PWSG) is the ideal place to define these.
- There is no common accepted benchmark use cases for benchmarking - Users are always curious about use cases, with proper use cases, they help the users to understand the blockchain itself as well as the performance indicators. The use cases for benchmarking are also open to discussion and welcome for contribution.
Hyperledger Caliper will not publish benchmarking results for projects that include this tool within their implementation. Instead, Hyperledger Caliper is focused on providing the tools to these projects so they can execute their own tests using common tools, metrics and terminology defined by Hyperledger PWSG.
From a technical perspective, Hyperledger Caliper leverages a pluggable adaptor solution to work with different blockchain implementations:
The key component in Hyperledger Caliper is the adaptation layer, which is introduced to integrate multiple blockchain solutions into the Caliper framework. An adaptor is implemented for each blockchain system under test (SUT), the adaptor is responsible for translation of Caliper NBIs into corresponding blockchain protocol. Caliper NBI is a set of common blockchain interfaces, which contains operations to interact with backend blockchain system, for example, to install smart contract, invoke contract, query state from the ledger, etc.
This can be seen in visual form in the figure below:
The Caliper project initially began in May, 2017 and has been actively designed and developed with Huawei, a global information and communications technology company. The project has a GitHub repository and is welcoming contributions. Currently, Hyperledger Fabric 1.0 and Sawtooth support Caliper, while support for Ethereum and other blockchains are in the planning stages.