Exploring HyperLedger: Experience in Being a Framework Early Adopter

Some time ago our team explored deploying a HyperLedger-based blockchain within a production environment. This article presents a story of our attempts to integrate it, the problems we encountered, and tricks that helped us solve them. Several important updates have been introduced into the HyperLedger framework, and so some of the challenges were overcome, while others still wait for a solution. 

In the first part of the article we explain why we decided to use a blockchain to solve a  business problem and why we chose the HyperLedger framework over Ethereum. The second part of the article is dedicated to HyperLedger-based blockchain architecture and technical aspects of the framework implementation. 

Why use blockchain?

We initially believed that blockchain was unnecessary for our business. After all, most businesses resolve their trust issues by referring to centralized facilities or arbitrage centers. As a result, it took us a long time to decide whether in our case a blockchain solution was needed or not. 

Our platform is a kind of web resource where companies can reduce their development time and maintenance costs by working directly with open source teams. We identified that for some customers it might be difficult to establish working relationships with open source maintainers and key collaborators. The standard practice of resorting to services of in-house developers or freelancers for fine-tuning open source code seemed sub-optimal, because of increased project time and price.  

With our platform we aimed to resolve this inefficiency by providing an ‘entry point’ and simple interface for customers to request and co-finance developing new features in OSS. For this system to be sustainable we needed to introduce a tool that would incentivise developers to fulfill customers’ requests. After some deliberation, we come up with an idea of ‘digital ownership’.

The idea was quite simple: a person who registers a project on our platform receives digital ‘shares’, which they may transfer to his fellow contributors at their own discretion. As the name suggests, our shares enable holders to receive a share of a project income proportional to the share of ownership. On the top of that constraint, we added a rule that any ‘outsider’ developer could create  a requested piece of functionality, and if this solution is accepted by clients, receive a portion of the project’s shares. 

We wanted developers to treat our project shares as a valuable long-term asset, which inherently implies that developers believe it won’t disappear. We basically had two options: we could either postpone introduction of this functionality until we gain the community’s trust, or we could build a trustless system. The latter path would require building a platform in such a way that would leave the assets untouched even if the parent organization exited the business.   

We also planned to integrate the platform with numerous partner organizations, which would outsource development tasks to our platform and automatically receive a fee when they are completed.  In an ideal scenario, we would just provide an access point for organizations into our network through some simple registration process, such as API integration. Our goal was to make the process as easy as possible to avoid all the legal complications and paperwork. After some doubts, we decided that blockchain would help us to realize this vision.

Ethereum vs HyperLedger

Ethereum was our first choice, even though we didn’t have significant experience with the platform. We hadn’t studied the documentation in detail; we just got right to spiking the integration. At first glance, it seemed like an easy choice. Several factors led us to give it a try:

• It’s rather mature;
• It’s stable;
• it’s easy to integrate;
• it’s easy to develop;
• It has a large community;
• It’s been rapidly developing;
• It’s been used in numerous projects;
• It gives an opportunity of private deploying in our private network.

One the other hand, there were a few factors that eventually convinced us that Ethereum was not the right choice for us: 

• Non-determinacy of the consensus algorithm.
• Proof-of-work (POW) is unpredictable. 
• Non-existence of roles.
• Uncontrollable access to the network.  
• Transaction fee and high CPU workload even in a sleep mode (minor).

Some of these issues have potential mitigations in development in the Ethereum world. For example, the GHOST protocol modification may well help, but even in this case if owners of the major pools suddenly decide that their branch is longer than yours, while your tanker with tuna is in on its way to the destination point… well, the cancellation of the payment transaction may surprise you, if the transaction is being conducted over a public (or shared-use) Ethereum network.

In an extreme case, we may even have a partner whose node capacity allows them to outhash the whole network, so there would be no point in using blockchain. 

It was also very important to us to understand whether a network member is a client or a partner. We have to know this for sure. The Ethereum network does not support this feature, so we would need to build this on top. We certainly could integrate our VPN into the blockchain. But if we provide access to partners, there, naturally, should be a way to open up such access. At the same time, we would like to exercise control over who has access to our network and what they can get from it.

A key takeaway is that it’s important to remember that within a business-use case corporate node capacities may significantly preponderate over private ones. This is why we opted for a private blockchain, using Hyperledger framework. 

HyperLedger also allowed us to avoid the minor inconveniences that we observed regarding the costs of transactions and CPU usage that we observed with Ethereum.

At this time the HyperLedger Fabric was one of the most advanced and mature frameworks in the family. It also has a few features that make it to stand out. The permissioned architecture ensures that if someone accessed your blockchain, you know whether they have a certificate issued by a Certificate Authority (CA). We also liked its deterministic PBFT algorithm, with which you can be 100% sure that a transaction is completed once your received such notification. Test launching on docker-containers is also very simple.

We tried to figure out whether we need Byzantine Fault-Tolerance. Do we really trust our partners, and do they really trust us? Can we afford to expose ourselves to Byzantine Generals’ Problem, knowing that at any moment any node could start sending incorrect data to the network? We eventually decided that we should have such protection, and it was fairly easy with HyperLedger. 

Still in doubt, we conducted some tests to compare HyperLedger Fabric and Ethereum in a private network.  We coded a trivial contract that generates a long array and then sorts it. You can see the results on the graph below. We have added two lines for 1 million and 10 million elements to the image just to show that Hyperledger is here too. In fact, the difference is such big that lines are actually invisible. 

Y axis: Milliseconds.  

Y axis: Megabytes 

Now let’s consider the time required for reaching a consensus. We took a simple empty transaction and put it in a cluster of 8 machines. The machines had to reach an agreement and return confirmation: we waited for six confirmations in the Ethereum private network and a confirmation from each node in the HyperLedger network.  The speed was still better in the HyperLedger cluster. 

Y axis:  seconds

We should note that we conducted the tests on version 0.6 of the HyperLedger Fabric framework; as of today the latest version is 1.2.0, which has a separate node responsible for maintaining  transaction order. Back then, the network became frozen if you increased the number of nodes to 16 and the speed to 500 transactions per second. At such pace the network was not able to reach a consensus before receiving a new transaction request. 

HyperLedger architecture 

Before we move forward, let’s consider the basic architecture of the Hyperledger blockchain. 

Peer - the main node, which stores information about all transactions (in version 1.0 it is divided into Endorser, a peer that confirms transactions, and Committer, a peer that records transactions to the register.)  

App - the client initiating a transaction can be replaced with its own application on Hyperledger SDK

CA - provides users with certificates that allow them to make transactions and read data from the registry

Orderer - arranges transactions in block and transfer blocks to nodes for recording in the ledger

HyperLedger can separate nodes by roles. In particular, there is a peer that stores the register. In the 1.2 version there are several subtypes of peers, but generally peers are responsible for storing registers and validating incoming transactions. They store all smart-contracts and chain-codes, and approve incoming transactions and save them to the register. 

The application we built is on the frontend. It can send information about transactions to the blockchain, and it can login to blockchain with a member certificate. It is also responsible for consensus. 

The CA issues certificates. By default, HyperLedger can distinguish nodes by organizational attribute; each organization has its own root certificate. With a membership certificate you can assign rights on completing smart-contracts, rights on changing network configuration, and rights on adding new peers – basically whatever you may want. In the newest versions of the framework, you can also add any attributes you like to certificates, so you can be even more flexible in providing different sets of rights to system participants. 

An ordering service, or the “orderer”, is a set of nodes responsible for a transaction order in a block. The orderer collects transactions into a block and sends this block to peers, so they can commit it to a register. It does not store smart contracts, though stores ledger data  in a binary file, which is used to bootstrap new peer. Losing this file means losing all blockchain data. The orderer also performs some validation: it checks hashes and signatures.

For example, our system consists of the following elements: 

• A web application, 
• A peer, 
• An OpenGift organization, 
• A root CA of the organization 
• An intermediate CA, which was designed with an idea of scaling up the system; 
• A cluster of orderers on Apache Kafka to which all partner peers refer 

At the present moment, our blockchain is deployed on four real peers, and we have four orderers in Kafka. We ultimately need five, as it is recommended to use an odd number of nodes for the ordering service in this mode. We have approximately 100 client applications, 1 Root CA and 1 Intermediate CA. In the first several months of our work we have conducted over 1000 transactions, but our system allows us to process the same quantity in 1 second. 

Partners have their own peers so that they can store a register and validate transactions, and customers can refer to any peer they like to interact with the blockchain.  

Client applications log in to the blockchain by providing a certificate, which can be issued by a Certification Authority intermediate server trusted by blockchain, for example, “organization one”. CA Intermediate servers are authorized by a CA Root server, which is kept aside of blockchain network.   Then the client application can interact with peers within the framework of available policies, in compliance with restrictions and permissions. Once any peer confirms a transaction submitted by the application, and if it uses any consensus algorithm, it sends the transaction to the orderer. The orderer commits these transactions to peers. After that, the application can wait for any number of confirmations from the peers to make sure that the transaction was recorded in the ledger. 

What is it like to implement HyperLedger Fabric in production?

Perhaps the first thing you notice is an absence of any simple admin panel. It's very difficult to maintain it all in production mode without Kubernetes or Swarm, so we had to write a lot of supporting scripts. Hopefully,  with the Cello project this will change for the better. 

We faced several technical challenges while trying to implement this architecture. First, the orderer service can operate in two modes: solo mode and Apache Kafka mode. If you use solo mode, you can’t switch to the scalable mode without re-creating the entire network. 

Second, If you use the orderer services on Kafka, you cannot scale it to other organizations. If other organizations already have their own orderer services, you will need to reach an agreement on who will be in charge of arranging transactions in blocks. This means that only one organization can be responsible for the order of transactions in a block, which leads to some vulnerability. However, in general, if transactions are valid, their order in a block is not of a particular importance. If someone changes the order of transactions and they become invalid, they will simply be marked as invalid in the block, and your request will return “fail”.

CAs (certification authorities) are easily scalable. Each organization has a root CA, and it can issue any number of certificates to intermediate CAs. This is great because the CAs are responsible for adding users to the network. However, the certificate revocation mode is not well configured. First, in order to request several parties to sign a revocation certificate, you need to write an additional chain code. Second, even when you add information about a revoked certificate in a blockchain, the certificate ex-holder can still connect to peers. You have to generate the certificates manually and add them to folders of peers and orders. Controlling that type of process may be challenging in a decentralized structure.

You also need to keep in mind that until the orderer has created a new block, all queries to the register will return the previous state of the network, i.e. the register has transactional (versioned) semantics. This means that if you have a business process that consists of multiple read queries and a write query right after them that takes into account the result of the read queries, you had better make them asynchronous. Because in this case, your expectation of reading the registry will not be consistent with its real state. In general, you need to wait for the orderer to form a block and send it to the ledger; only after that can you send read queries, assuming that the state has already been changed. 

Since the blocks are not created according to POW protocol, you can set any block creating frequency for the ordering service. In solo mode, you will not be able to create more than one block per second, and in Apache Kafka mode, you can configure this parameter quite flexibly. Keep in mind though, if you decrease waiting time for creating new block, your network will increase in size quite quickly. Disk space will also be consumed very quickly, and so you always need to find a balance between a speed of transactions confirmation and your capacity. 

The consensus mechanism is realized at the transaction level, so you can specify requirements that transactions will need to comply with to be valid in smart contract. For instance, when you introduce a new smart contract in the chain code, you set a procedure of its confirmation, how many participants have to sign the transaction for it to remain valid. 

Smart contracts can be written in several languages, Golang and Java being the main ones. A typical smart contract has the simplest structure. Only two simple methods are required to be used in smart contract: one  of the methods is called when a new chain code is set up or upgraded (init) , and the other one when it is called(invoke). Different policies are configured to initialize a new smart contract and to call it. One group of users can be responsible for updating of a smart contract; another group can be responsible for its implementation. Here we consider the simplest function call, which takes a function and parameters of this function as an input argument and depending on the name of the function calls the needed method. 

func (t *SimpleChaincode) add(stub shim.ChaincodeStubInterface, args []string) pb.Response {
    	var cs clientState;
	clienState.Name = args[0]
	clientState.Balance = 0
	strState, er := json.Marshal(clientState)
    
	err = stub.PutState(pName, []byte(strState))
    
	if err ~= nil {
    	return shim.Error("Failed to add Client state")
	}
    
	return shim.Success([]byte(“OK”))
}

Data storage in HyperLedger may be considered as a key-value map, referred to as KV-storage.  Working with KV-storage is quite low-level.  With PutState() method you can write in KV-storage, and with GetState() you read from it. But the most interesting thing is that you can work in a smart contract with the attributes of certificates. In this example you can see how the hash of the public key of an authorized user is used as an identifier for his wallet. In the 395th line we get a hash and use it as a key for KV-storage.

func (t *SimpleChaincode) add(stub shim.ChaincodeStubInterface, args []string) pb.Response {
            pk, err := cid.GetX509CertificatePublicKey(stub)
    
	var cs clientState;
	clienState.Name = args[0]
	clientState.Balance = 0
	strState, er := json.Marshal(clientState)
    
	err = stub.PutState(pName, []byte(strState))
    
	if err ~= nil {
    	return shim.Error("Failed to add Client state")
	}
    
	return shim.Success([]byte(pk))
}

func (t *SimpleChaincode) query(stub shim.ChaincodeStubInterface, args []string) pb.Response {
	pk, err := cid.GetX509CertificatePublicKey(stub)
    
	strState, err :- stub.GetState(pk)
	if strState == nil {
    	return shim.Error("Client not found")
	}
    
	var cs clientState
	err = json.Unmarshal(Avalbytes, &cs)
    
	return shim.Success([]byte(cs.Balance))
}

Although, we are still using the 0.6 version of the framework, the newer versions contain some major improvements, which we have to mention:

• In the older versions, you needed to recreate all blockchain to include a new organization in a genesis block. Now it’s quite simple and you also can change policies of working with blockchain for each organization. 
• Starting with 1.2. version the system can have its peers compute the requested information dynamically and present it to the SDK in a consumable manner. 
• External applications can receive and process information about events from a chain. This feature may be helpful in a number of cases, for example - for notifying a controlling organization about suspicious activity.  

HyperLedger experience in a nutshell

From the technical perspective, the system is still developing (steadily but firmly.)  There are some technical issues, but hopefully that the community will find solutions for them. Still and all, we believe HyperLedger is one of the best options for companies looking to implement blockchain in real-world business. 

On the business side, thanks to the framework we successfully realized the intended digital ownership functionality, which helps us to incentivize development teams to work on open source projects. The network is easy to expand and can exist without a parent organization. If we disappear, the community agrees upon setting a new ordering service, updates the channel and continue working.

Based on a feedback we’ve received, this capability facilitates adoption of the platform, since our users don’t need to trust us and rely on our ability to do business. We are actively looking for partners to hand over the nodes and plan to undertake first technical integrations for our blockchain in early 2019. 

About the Authors

Yegor Maslov is the CEO of OpenGift Inc., platform for open source software monetization, Head of The Hive project, system empowering code reusability in organizations. Yegor has over 15 years of software development experience in web and mobile fields combined with an extensive background in technical entrepreneurship.

Konstantin Erokhin is a DevOps engineer with over 10 years of professional experience. He worked in such companies as Kaspersky, Sberbank Technologies, Moscow Stock Exchange.