Key takeaways
|
One of the unsung heroes in the Apache regiment of Java libraries is the load testing tool JMeter. Released in the earliest days of Java 1.1, JMeter has become an indispensable tool for testing load and functionality of multi-tiered applications comprised of web front ends, JVM servers and a wealth of NoSQL and relational databases.
Learn Apache JMeter by Example is a new offering by Sai Matam and Jagdeep Jain, which can be downloaded on Leanpub or ordered in print form from Amazon. Think of it as the missing manual for JMeter. Using clear prose and screenshots this book smooths the learning curve required to harness the full power of this powerful tool. It is targeted at all levels of developers.
The book is organized as follows:
First it makes a case for automated performance testing by graphing the steep drop in conversion rate as a function of application response time, and then makes a case for JMeter as an industrial strength tried and tested solution.
Next up is a performance testing primer covering theory and practical considerations, followed by a sample JMeter test you can build along.
The next few chapters get very specific, covering the creation and recording of your JMeter test plan, allocating thread groups for managing your load (the starting point of your test), creating pre-processors for enriching requests before sending to the server, controllers for defining the test sequence, timers for simulating user click-through behavior, samplers for creating the actual server load simulating a variety of request types, assertions for validating the response, listeners for capturing the server response, post-processors for extracting data from the response, and defining properties and variables for use throughout your test suite.
Applying these basics, the next chapter covers distributed testing, including testing in GUI mode and Non-GUI mode, followed by best practices, troubleshooting, using plugins, and JMeter recipes.
Following that is a comprehensive case study and a tutorial on the performance dashboard.
The whole book is just short of 400 pages but it’s a fast read, replete with large pictures.
From the introduction, we learn most poignantly the motivation for load testing:
In many companies, performance testing is not a priority till it becomes critical. The engineering team is then asked to complete performance testing within an extremely short time.
InfoQ spoke with authors Sai Matam and Jagdeep Jain to learn more about the book.
InfoQ: This is quite a comprehensive guide; what motivated you to write it?
Matam & Jain: We were on a performance engineering team and were tasked to improve the performance of a SaaS web application. We used JMeter for our performance testing. We got stuck in writing JMeter test scripts for some of the use-cases and felt the need for a guide to walk us through the steps. The existing resources(books and online) mentioned a few tips but were not detailed enough. The idea of the book emerged from our frustration as we pieced various bits of information to solve our issues in writing the JMeter tests. We decided to write a book that would have detailed examples without missing any intermediate steps.
InfoQ: When planning load testing, does JMeter factor out the load of the testers when measuring the results? Or is this a separate obligation on the test planner?
Matam & Jain: The response time is calculated from the point the request leaves the JMeter and the time that the response from the server is received. There is very little overhead from JMeter itself, so the results are fairly accurate. The results are accumulated while the test is running. Processing the results, including the generation of graphs, only happens after the test has ended to minimize the impact on the results. Similarly, when JMeter is running in a distributed mode, the results from the slave-nodes are aggregated and processed by the master-node at the end. The load from a user is simulated by a thread. Obviously, the number of threads that can be run efficiently is limited by the system it is running on. The engineer executing test should monitor the CPU and memory usage on the system running JMeter. It would be a good idea to keep the load below 60% to ensure that there is no overhead on testing.
InfoQ: There is not much on using in continuous integration environments like Jenkins or Bamboo. Was there a reason for that?
Matam & Jain: In the book, we mention that it is very important to incorporate performance testing into your continuous integration cycle. The case study also features this. However, we felt that a detailed discussion on this would add to the scope (and page-length) of our book. We wanted to focus on the essentials.
InfoQ: Besides performance testing, JMeter provides important functional testing capabilities as well. Did you choose not to cover these?
Matam & Jain: JMeter was not built for functional testing. However, it can be used to do functional testing, but it would not be very optimal. Functional testing needs a combination of unit tests and tests covering end-to-end use-cases. There are special tools and frameworks for this. With JMeter, you would be limited to testing only the functionality that can be tested over the net. Besides, if you want to use JMeter for functional testing, your test script needs to be structured differently. It needs to have a comprehensive list of 'assertions' to verify the response. In the book, we assume that functional testing is being done using appropriate tools. We mention that the performance testing makes sense only after the functional testing has passed.
InfoQ: Why did you develop the sample application? Was it just for this book?
Matam & Jain: There were two reasons:
- Usually, when learning JMeter, the sample test scripts are developed to hit various public websites (like Yahoo or Google). It is not correct to put load on them.
- Sometimes, when learning JMeter, we need a special use-case like a URL that generates a redirect, a SOAP endpoint or a REST endpoint that accepts a JSON payload. The sample application, 'dt', meets all these needs. The reader can follow the examples in the book without having to modify any of the test scripts.
The sample web application is an imaginary online ecommerce shop selling digital gadgets. We called it 'Digital Toys' or 'dt' for short. It is available on our github repository. It is extremely simple to deploy - just unzip and run it on command-line. All the examples illustrated in the book were developed against this.
InfoQ: For whom is the book intended? How can we utilize this book effectively?
Matam & Jain: This book is intended for beginners and engineers who have some experience with JMeter. Beginners can read the first four chapters to learn the basics of recording, creating and running the tests. The experienced engineers can refer to the specific examples based on their needs. The chapters on components clarify the configuration and usage whereas the cookbook illustrates specific uses. A performance architect can benefit from the 'primer' and the 'case study,' using which he can plan the performance strategy for his company.
InfoQ: Are there any performance tests that JMeter is not suitable?
Matam & Jain: JMeter has a plugin mechanism using which testing new protocols can be supported. However, you need to invest your time to develop the plugin. JMeter may not be the tool of choice if a certain protocol is not supported out-of-the-box and if there are no suitable open-source plugins available.
JMeter is not the best choice to test lower level protocols where the expected throughput is near the wire speed. For example, it is not suitable to test Ethernet and other lower protocols where the expected throughput is in the order of 100 Mbps or 1 Gbps. There are specialized appliances for such purposes.
InfoQ: What are some of the weaknesses of JMeter?
Matam & Jain: JMeter, before version 3.0, did not have good reports. But this has been alleviated to some extent with the introduction of the dashboard. Another aspect is the error reporting. The cryptic error messages or, sometimes, the lack of error messages makes it difficult to develop test scripts or to notice errors during the real test run. However, these are minor issues, and the JMeter team is working on these aspects even as we speak. By properly structuring the JMeter tests, making use of reports and dashboard, companies can reap the benefits of performance testing and avoid spending money on commercial tools.
About the Book Authors
Sai Matam has 20+ years of diverse experience in software development. He has significant experience in performance testing and tuning, having worked on tuning Java applications as well as web applications with many millions of page visits. Sai develops custom web solutions and offers performance tuning services through his company Enablor Inc.
Jagdeep Jain has more than a decade of experience in Software Quality Assurance and Testing. With a degree in Computer Science & Engineering, Jagdeep is a firm believer and advocate of test automation. He has used Apache JMeter quite extensively.
