Solutions for Testing Blockchain: Private Blockchains, Permutations, and Shifting Left

Blockchain is an emerging software architecture that has the potential to be a big disruptor in the industry. With change however, comes the added risk of quality issues. As developers and test engineers, we need to be prepared for those changes to better adapt to the new technology and allow for the continued development of software and products through it, without compromising on quality.

In this article, I will talk about some of the key challenges faced by teams needing to test blockchain technology, while also providing solutions to meeting those challenges.

Challenges in Testing Blockchain Systems

Given the relative newness of the technology, there is still very little in the way of formal best practices or testing tools around blockchain technology, though this is something that is rapidly changing with a lot of new up and coming frameworks in this area. When I started making sense of blockchain systems and how to approach testing them, part of the initial challenge was the limited scope of examples I can learn from and established frameworks in place. 

Many companies had great solutions in place to help them with their testing efforts, but most were created by developers and so while these different testing engines were effective at mocking and replicating aspects of blockchain functionality, the scope of understanding their end-to-end functionality was not established. Many more specialized testers have since gotten involved and many of the new frameworks (which are explored further on in this article) have solved some of these end-to-end challenges.

The next key challenge, which is similar to other forms of testing, lies in creating enough user specific data to ensure you can meet all the required use cases. While some of the data rules are often easy to understand in most blockchain systems, the immutability of transaction on the technology simply means that there can be little margin for error and so it's something that testing teams cannot compromise on in terms of understanding their permutations ánd ensuring the coverage of their test data is sufficient.

In the FinTech space where I’ve spent most of my career, this means trying to create enough samples of different transaction types to cover all the relevant permutations required. This includes purchases and withdrawals, varying account types, histories, and balances, understanding different tax regulations, and how the system handled it. It's true that the blockchain component was not responsible for much of this functionality and merely acted as a storage medium, but the majority of data in which these decisions were made was from the individual blocks themselves. 

If you’re trying to do this in a public blockchain setup, it could prove quite expensive and can remove some predictability from your testing effort, as you can’t easily just return the state of your individual blocks to where they're needed. This is why I recommend trying to set up a private test environment, or one that closely resembles the public offering in every way except scale. This too is quite expensive from an infrastructure perspective, but makes your testing and automation efforts far easier and with quality critical in working with blockchain systems, it would be an investment that is well worth it. 

It is possible to also make use of as public testnet, which is essentially a duplication of the public blockchain you are utilizing set up for testing purposes, but then you are still limited by how you can breakdown r alter data to make your testing more predictable, and so a fully controlled private blockchain offering gives your development team greater test coverage. 

And even though blockchain technology is seen as more secure given its encrypted, distributed and immutable nature (as it would require hacking at multiple places to affect change making this more complicated), there is still a lot that can go wrong from a security perspective, especially with the underlying applications that interact with the blockchain model and the individual contracts. The security that blockchain provides only applies to the data at rest and the end points that interact with the blockchain remain key points of vulnerability that should not be overlooked because of this “secure” perception. 

Where blockchain hacks have occurred in the past is when a company's own security measures are compromised and hackers are able to manipulate the software/messages that interact with a blockchain contract rather than a compromise made directly to the blockchain itself.

Much like security, performance can be a potential stumbling block if a blockchain system is not well-architectured or regulated, much like how Bitcoin has become such a time-consuming and energy-sapping technology with each transaction taking an impractically long time to process. This points to not just efficiencies that can be made at an algorithm and processing level, but also scaling where certain restrictions may need to be put in place to prevent smart contracts being distributed in an inefficient manner. 

Or to put it more simply, if you are performing a task a second or two more slowly than you should be, you might not think much of it, but if you now have to scale that takes out to 1000 different machines many times a day, you will end up losing a significant amount of time in processing. Even if many of those 1000 transactions can happen in parallel, it would still likely take over a minute longer (on top of the encryption/decryption time) to process across all the machines if they are distributed around the globe. Now imagine waiting that minute for your transaction to complete. And with most blockchain architectures, that number is in the tens of thousands. My example might be a little exaggerated, but is a reminder of how important it is to ensure algorithms are always optimised and why it might be useful to limit the number of processors (or minors) to prevent inefficiencies from escalating.

While we don't have answers to the latter, it's something testers should be thinking about as they and their teams develop blockchain systems that can meet the needs of the future. To combat this we shouldn’t leave performance testing too late in the development cycle, but rather incorporate performance testing into the development pipelines to ensure our code is suitably optimized. While this is true for all software architectures, poor optimization is more acutely felt in distributed systems.

It’s also important for teams to find the right balance of user and data load for their specific system and incorporate ADCs (Application Delivery Controllers) and Load Balancers into their design to manage this load and help scale it out where needed, without overwhelming the blockchain servers with excessive transactions. 

Additionally, the cost of tokens at a public level and scale that can run across thousands of servers are elements of the technology that can be difficult to reproduce in testing, while the immutability of the technology means that errors cannot be tolerated and calls for high standards of software quality. Rather, teams need to rely on using containers that can run samples of the blockchain or an established private network to continuously update and restore to get predictable results. This cost-saving approach does however mean that the scale of your core testing systems is never at the scale of the bigger production system. 

Making use of a public testnet may solve some of these scale concerns should you still need to test your blockchain at scale, where you can still get high functional coverage against your private blockchain network and a better sense of load and performance with the public testnet. 

Even if you are getting a good sense of how transactions perform on the public testnet in a more real-world situation, there is still a variability in the data between test and production and as such, it's important to still monitor the production operation closely. Key things like monitoring and synthetics that check the operations of these accounts can help to identify any potential challenges as well. 

Approach to Testing Blockchain Systems

Does blockchain technology require a different testing approach to traditional software architectures? 

The short answer is no, though it does require you to have a strong understanding of what is going on with the software technically to get it right. The testing basics don’t change. Identifying clear requirements, performing solid test analysis and shifting left as much as possible with regards to test automation are key attributes of testing blockchain technology and this will likely only push testers into a more technical direction. 

Despite the increased technical nature of blockchain automation, nothing beats good test analysis, so make sure your testers maintain a strong emphasis on this. Important focus areas for the functional tests can include things like:

• Block Size
• Chain Size
• Data transmission and 
• Adding a block

To achieve the best test analysis, it’s important to have all the required information, so as testers are trying to identify all the required testing permutations, it should no doubt lead to plenty of questions that can help shape the functionality of the system further. 

From a testing coverage perspective, you need to make sure that you can add and interact with one effectively, but also keep in mind the size of it as the number of transactions grows. The larger the block size, the more complicated it takes to process (one of the reasons why Bitcoin is so notoriously slow) and so testing how a system handles excessive transactions and ensuring it can still maintain an effective size is important. It’s not just blocks that grow, but a collection of blocks can also grow (referred to as chains) and so testers need to ensure that in each new transaction, the integrity of the chain is maintained and nothing goes missing. 

It is also critical for testers to thoroughly assess the effectiveness of the APIs that interface with the blockchain, and ensure that the required inputs and outputs are extensively tested in this area; which means that as testers, you need to create data payloads that will cover all the necessary permutations that you can think of for your smart contracts. In the projects I have worked on, we have chosen to utilise JSON files to shape the data where it's easy to assign different data values to each attribute of a contract, an example of which I detail below in my framework example.

My other advice is for teams to think smaller when it comes to blockchain testing. Rather than focusing on bigger end-to-end solutions, spend more effort at a smart contract level and make sure you understand how the model of the contracts works well. Smart contracts are simply programs stored on a blockchain that runs when predetermined conditions are met and are where most of the critical logic around a blockchain are contained, and so by emphasizing your test analysis here, where you can mock the responses and functionality far more easily, you can ensure your testing efforts achieve the greatest coverage.  Yes, end-to-end testing will still be required for all parts of the system, but the blockchain components can operate largely independently and should be tested as such. 

I’ve always been a big proponent of solid unit testing over exhaustive end-to-end testing, but in working with blockchain systems, I have been challenged to expand on this even further to get closer to the code and how it is interacting with the different contracts. When I first started playing around with blockchain systems, I would try and test directly onto a testnet and do some expansive manual functional testing to get familiar with its operation. With little margin for error with data submitted on to the blockchain though, it's wasteful to be identifying issues at this point. So it’s best to focus testing efforts at a unit level where I can cover all the required permutations and reduce dependence on latter forms of testing which are far less efficient. 

This has helped me to think more aggressively about permutations very early on in the development process, and ensure we design our APIs that interact with the blockchain correctly. This makes any additional testing effort far easier and with so little margin for error in working with blockchain, you are forced as a tester to up your game in thinking of everything, including those non-functional aspects like security and performance which can so often get sidelined in any testers toolkit.

Understanding Blockchain Testing Frameworks

While the technology is new, there are still some tools and frameworks available. For example:

• Ethereum Tester: A python-based open-source testing library. Its setup is pretty easy with manageable API support for various testing requirements, like compiling contracts, handling exceptions and identifying events and failed transactions to make test reporting easier.
• Brownie: This framework has the testing functionality of Ethereum embedded in the form of a set of features for test contract deployment. It’s developed around the py.test framework, hence, it is relatively easy to implement.
• Truffle: A commonly referred name for Ethereum developers, which brings in good testing features, such as automated contract testing (using mocha and chai). The framework holds capabilities beyond just testing functionality within the blockchain application, like configuring build pipelines, features built in script generators and runners, as well as supporting both network and package management.
• Embark: A testing framework that focuses on developing decentralized applications (dApps) that run on various systems or nodes. It has integrations with Ethereum blockchain, IPFS, and decentralized communication platforms such as Whisper and Orbit.
• Drizzle: Part of the greater Truffle suite. Drizzle is an assortment of front-end libraries that offers useful components for developing web applications that can seamlessly connect with the Smart Contracts. 
• BitcoinJ: A Java-based framework built for Bitcoin-based apps that enables you to interact with the real BTC network and various testing activities. In order to use it, you don’t have to download the standard BTC Core files from Bitcoin.com. You can even approach a user forum in case you need clarification or are facing hiccups in the testing process. It is an open network available for assistance.
• Ganache – A locally deployed blockchain simulator. Ganache features a graphical user interface that can simulate blockchain networks and live-test Smart Contracts without requiring you to set up real test networks or using a remote network.   

The projects that I have been involved in have found Ethereum Tester, Truffle and Ganache to be the most appropriate for our use cases - but you might find a different approach suits your project. I would recommend playing around with the different testing frameworks to see what works best for your particular use case.

The trick is to approach any smart contract in much the same way you would an API, and focus on expansive unit testing and mocking at a code level to reduce the coverage needed at higher testing levels. You can then also easily adapt certain API testing frameworks to send and receive messages from mocks of the contract or ones running in a virtualized environment to further expand your testing coverage. 

So, for example, if I set up a basic Hello World blockchain example that looks something like this, using Solidity, which is a contract-oriented programming language that is especially useful in shaping smart contracts to work on the Ethereum platform:

pragma solidity 0.8.9;
contract HelloWorld {
   string private message = "Hello World";
   function getMessage() public view returns(string memory) {
       return message;
   }
   function postMessage(string memory newMessage) public {
       message = newMessage;
   }
}

And then set it up with the following config:

module.exports = {

networks: {

   development: {

     host: "127.0.0.1",

     port: 8545,

     network_id: "*"

   }

 },

 mocha: {

 },

 compilers: {

   solc: {

     version: "0.8.9",

     docker: false

   }

 }

}

And deploy it:

var HelloWorld = artifacts.require("HelloWorld");
module.exports = function(deployer) {
  deployer.deploy(HelloWorld);
};

We can then generate test scripts for it that matches the way we would interact with an API:

var abi = [

 {
   "constant": true,
   "inputs": [],
   "name": "getMessage",
   "outputs": [
     {
       "name": "",
       "type": "string"
     }
   ],
   "payable": false,
   "stateMutability": "view",
   "type": "function",
   "signature": "0xce6d41de"
 },
 {
   "constant": false,
   "inputs": [
     {
       "name": "newMessage",
       "type": "string"
     }
   ],
   "name": "postMessage",
   "outputs": [],
   "payable": false,
   "stateMutability": "non-payable",
   "type": "function",
   "signature": "0x368b8772"
 }
];

This would still require formal tests to execute this specific data, but the idea is clear: there is a strong familiarity in the way we would test APIs and blockchain contracts.

The biggest challenge is in being thorough in your analysis of the contract rules and data needs, and ensuring that the software correctly validates those rules and that you have the right level of data to make the testing effective. This can really only be overcome with a solid testing mindset and analysis. While so much of Blockchain technology is new and requires a very technical focus on your testing effort, at its core, it still requires traditional analysis in identifying everything that needs to be tested.

It can often be the simple things like having poor built-in validation on an API that calculates values correctly to far more complex aspects like incorporating certain tax laws and regulations from different countries into your testing. As testers, we need to be aware of these restrictions and ensure our analysis hats are on, always looking for the gaps and where things can go wrong.So as testers, make sure that you understand the business context of the application and data required for the blockchain. Testers will want to identify all the permutations around these areas, specific limits and negative scenarios and ensure they test these. The same considerations which would apply to normal integration testing for interaction between an applications API and the blockchain token or even security and performance testing need to apply here too. 

With blockchain technology requiring a high-quality threshold, focus on low level testing and cover all aspects of functional and nonfunctional testing. It is a technology that truly challenges the skill set of teams, allowing them to be at their best and make the quality of their applications a success.

About the Author

Based out of Cape Town, South Africa, Craig Risi is a man of many talents, but no sense of how to use them. He could be out changing the world, but prefers to make software instead. He possesses a passion for software design, but more importantly software quality and designing systems in a technically diverse and constantly evolving tech world. When not playing with software, he can often be found writing, designing board games or running long distances for no apparent reason.