Automated Error Reporting: The Gateway to Better Quality

Ignorance might be bliss, but it goes straight to the bottom line when it comes to software bugs.

According to software quality experts, those who can ferret out bugs and improve the quality of their software will be rewarded with greater customer trust, higher renewal rates, lower development and maintenance costs, streamlined delivery schedules, and fewer opportunities for the competition to poach their customers.

Quality counts -big time

If you think quality doesn’t count all that much, consider these points made by Capers Jones in an article published in the June 2011 issue of the American Society for Quality’s Software Quality Professional journal:

• High-quality product schedules are 15% shorter than those for lower-quality products.
• Total cost of ownership for high-quality applications from the first release through the next five years will be about 30% lower than for identical projects with poor quality.
• Annual maintenance costs are lower by 40% for high-quality applications.
• The larger the application, the more valuable quality becomes.

According to Jones, testing is only part of the answer, as it is less than 35% effective in finding bugs.

It’s not surprising, considering the impact of quality on their businesses, that software development teams are looking for new methods to drive improvements. One of the most promising of these is Automated Error Reporting, which taps into an organization’s user base to identify common software errors.

(Click on the image to enlarge it)

The trick: simple, complete, customizable

There have been many attempts to co-op users into bug testing -including email and forum reporting, in-house authored software, Windows Error Reporting (WER), and myriad off-the-shelf solutions.

Most of these fail to deliver the full potential of Automated Error Reporting for one or more key reasons: they put too much burden on users; they don’t gather precise and accurate information; they are missing some key attributes, such as the ability to report the values of local variables; and/or they don’t sync to commonly used bug-tracking systems.

The trick is to make the software simple, complete and customizable, so the reporting process becomes almost effortless from the customer side, but comprehensive from the developer’s side. This isn’t easy to accomplish; each report submitted should include the full stack trace and all the contextual information needed to find and fix the bug. The Automated Error Reporting system should also work in concert with other important tools such as obfuscation and bug-tracking software.

"At the end of the day [it’s about] making the reporting of issues incredibly simple for our clients," says Andrew Neville, senior software engineer at Neville & Rowe, who uses Automated Error Reporting in Red Gate’s SmartAssembly to track down bugs in the company’s ImpactEdge business intelligence and analytics software.

"We receive all the information we need in order to rectify the issue simply through the mechanism of the user pressing the send button. By building Automated Error Reporting into our solutions we are offering our clients an extra level of service and a more direct line of communication and dialog with us."

Know what you didn’t know before

The value of Automated Error Reporting is upending assumptions and enabling development teams to receive detailed information about exceptions they didn’t know existed. It provides value in five key areas of software development:

1. It offers an efficient way to get customer feedback, reducing or even eliminating the to-and-fro between your customers and your developers.
2. It gives you insight into which bugs are the most severe or frequent, allowing you to prioritize bug fixes based on facts, not guesswork.
3. It enables you to deliver faster fixes for your customers and meet -or even surpass -their expectations.
4. It can lower support costs by identifying and fixing the issues your end-users are actually experiencing.
5. It enables you to gather lots of early user feedback before you release an application. If you’re using Agile development methodologies, Automated Error Reporting helps you go through rapid iteration based on customer feedback, allowing you to build -and ship -a robust application as quickly as possible.

"Knowing the frequency of problems (especially immediately after a release) is extremely helpful in prioritizing and triaging bugs that are reported internally," says Ed Blankenship, a Microsoft MVP and IT consultant. "Additionally, having the context of where those errors occurred, including debugging information, really gives you that leap forward to start troubleshooting and diagnosing the issue."

How it should work

An optimal Automated Error Reporting system makes it easy for customers to tell your development team when things are going wrong with your applications. Ideally, when an exception occurs, the customer should receive a simple exception message, enabling reporting at the click of a button. It should eliminate the need for time-consuming email exchanges to try and work out why an exception occurred. All of the necessary, detailed information should be collected automatically, with minimal disturbance to the customer.

(Click on the image to enlarge it)

A typical error report should comprise a full stack trace and details about the exception context, including values of all the local variables.

Another nice feature is to allow the customization of the exception dialog to provide additional information that can be packaged with the exception report. Customizations might include a log file or a screenshot taken at the time of the error, or asking end users for contact information so you can notify them when a fix is released.

Privacy and annoyance issues

Three issues are often cited as obstacles to successful error reporting implementation:

• Error reporting software that sends back users’ private information.
• Annoying users who receive too many prompts to report the same error.
• Annoying developers who receive a bunch of identical error reports.

The first issue is one of care and trust. There is no guarantee that even the best Automated Error Reporting software won’t transmit personal information back to the developer in the course of doing its job. But, well-designed Automated Error Reporting software should contain attributes the developer can apply to regions of code containing sensitive information. This will prevent the error reporting system from collecting information in those areas. If the developer fails to attribute to the sensitive area, it falls back to users trusting the developer to protect their information, just as they do with routine cloud or Internet commerce transactions.

Avoiding the second issue -user annoyance -requires customizable Automated Error Reporting within a well-designed application.

Alex Davies, a Red Gate developer, says he will often write a custom error-reporting template showing dialogs at a sensible frequency. This could take the form of a checkbox that reads, "Don’t ask me again. Always send a report."

Davies adds if an error reporting service is in place and developers fix the most frequent bugs, the application will increase in quality and users should face fewer and fewer exception dialogs.

Thirdly, on the developer’s end it’s important to have a user interface grouping together similar error reports, so they can be dealt with as a whole. Even better, says Davies, is to have the ability to sync error reports to a bug tracker that groups the same bugs in one place.

Making life easier

The bottom-line benefits of Automated Error Reporting should be fairly obvious and measurable over time. But, there’s one less quantifiable, and perhaps most important, benefit: the way it can deliver satisfaction for both developers and customers. One comment from an IT decision-maker I spoke with says it all:

"At the click of a button, the user can send all this information back to the software development team. No more repeated telephone calls, endless back and forth emails and incomplete error reporting. Not only will it make a developer happier and your life easier, but it will make the customer happier and less frustrated."

About the Author

Laila Lotfi works in the .NET tools division of Red Gate Software and is a contributor to the Simple-Talk website. Red Gate creates software tools used by more than 500,000 .NET development and SQL Server professionals worldwide. The company works to uplift the markets it serves through free web community sites, technical publications and conference sponsorships that reach millions annually.