How Medical Companies are Innovating through Agile Practices

Introduction: Agile in a Regulated World

We’ve noticed adoption of Agile methods steadily growing in medical product companies over the past ten years. Practices vary with the types of development work, from cloud-based continuous flow for data-intensive services (clinical trial data management) to a sprint-based flow for physical devices with embedded software. The rise of home-use medical products has spurred use of Agile continuous delivery for mobile hardware platforms.  The question is no longer whether, but how Agile can work in medical product development - for our mix of technical, market, and regulatory constraints?

Doubts persist, based on misinformation about Agile practices, regulations, or merely on a deep suspicion that work completed quickly must be taking unsafe shortcuts. One belief is that regulators (both FDA and in the EU) prefer stage-gate, waterfall-like product development. This has been refuted by regulators and by industry people multiple times. The FDA’s own Design Control Guidance, immediately after depicting development in a waterfall-like diagram, notes that “In practice, feedback paths would be required between each phase of the process and previous phases, representing the iterative nature of product development.”

This article is based on case studies from our new book Agile Methods for Safety-Critical Systems: Case Studies of Medical Product Companies.  Specific companies and products are examined, and the actions their Agile change leaders took. 

We chose our cases based on several factors, chief of them being that we knew enough about the situation to say that it’s a valid use of Agile. By valid, we mean a sincere effort by both management and technical contributors to move the company through genuine change. While no company or group should merely copy another’s approach, these case studies offer insights into how readers might implement Agile principles in their own company.

How Agile Fits into the Medical Regulatory World

Regulatory bodies have four key concerns for medical product companies:

• Each company is to choose its own development process
• Every company must have a Quality Management System
• Documentation shall provide evidence of following a quality process
• Hazard analysis and risk management must be incorporated, not added afterward

Although the regulatory rules allow companies to choose their own development process, the companies must provide evidence that they are in fact following their stated process.  When Agile was a new concept, the traditional approach had the advantage of familiarity; auditors and QA / RA were accustomed to the stage-gate idea, and development processes typically involved only minor variations on that theme.  Businesses did not want to explain a very different approach like Agile, especially if they had had a rocky relationship with regulators.

In 2012 a group from industry, standards organizations, and regulatory bodies collaborated to provide just such an explanation bridge: AAMI TIR45 (an advisory document available for purchase from the Association for the Advancement of Medical Instrumentation: Guidance on the use of AGILE practices in the development of medical device software). Written in plain language, it describes key Agile practices and how they each compare with traditional practices to cover safety and quality concerns.

After stating their development process and how they’ll show their adherence to it, medical product companies must then demonstrate that their development work occurs within the framework of a Quality Management System (QMS), and that they have adequately addressed potential hazards to the patient/user, caregiver, or bystander.

A QMS needs to support product development and maintenance – to outline the activities and outputs which ensure quality of the company’s products and services. Overall goals for quality are provided in 21 CFR part 820 and in ISO 13485 – from providing adequate resources and proper work environment, to controlling and documenting design, to managing measurement processes, to following procedures to correct or prevent product flaws. Closely tied to the QMS are processes for estimating and minimizing product safety hazards.

Key to all these processes is documentation – the classic FDA quote is, “If it isn’t documented, it wasn’t done.” Too often, engineering groups wait to generate their documentation in an extra step, after design and development are well in hand. Agile turns that around, by generating documentation as a natural output of the development process. For example, tools like Jira,  Microsoft Teams and others track Agile stories, tasks, and other data that are needed for requirements traceability as called for by FDA and EU regulators.

Wide Variety of Agile Practices by Medical Product Companies

Within regulatory constraints, there is plenty of room for successful use of Agile and Lean principles, despite the lingering doubts of some in quality assurance or regulatory affairs. Agile teams in other industries have demonstrated that they can develop without any compromise to  quality. Additional documentation is necessary in regulated work, but most of it can be automated and generated incrementally, which is a well-established Agile practice.

Medical product companies are choosing multiple practices, from both Agile and Lean. Change leaders within the companies are combining those ideas with their own deep knowledge of their organization’s patterns and people.  They’re finding creative ways to achieve business goals previously out of reach with traditional “big design up front” practices.

Examples provide a great way to learn, but it’s not always clear which elements to replicate and which are only important to another company’s circumstance.

Our goal here is to show how the same core principles in Agile and Lean played out in very different day-to-day actions at the companies we profiled, and how they drove significant business goals for each company.

Agile Principle: Courage

Courage is a core value for Scrum and Extreme Programming: the courage to speak up when things aren’t right, or courage to try new practices, to experiment and learn as a team. Consider two situations where medical device companies needed to “reboot” their own home-grown Agile processes, and one example about an Agile sales process.

At Sanofi

The Cambridge, Massachusetts-based software development group within Sanofi (a large pharmaceutical company) has been following an Agile approach, customized to its needs and experience, since at least 2016. Their product, for diabetes management, is classed as “software as a medical device” (SaMD). The path has not always been smooth, and in 2018 they decided they needed an Agile reboot.

Prior to 2018, the Cambridge team employed a somewhat undisciplined approach, which Sanofi QA representative Ben Blakemore characterized as: “We have all these tasks—let’s form a twenty-five-person Scrum team. We’ll have standups, talk about what we’re doing, and do a retro every two weeks.” Though they used sprints, user stories were frequently not broken down into pieces small enough to complete in a single sprint. Often, development would do a lot of work and push several documents to Quality for approval—but then Quality would find problems (design input, tests, whatever), and back the project would go to the development team.

In 2018, the team and their software development director decided to reboot their internal Agile software process. Two things stand out here: how they regrouped, and what medicine they decided was needed to get their Agile process healthy again. They regrouped by holding a week-long series of meetings to discuss what they wanted to change. This involved a great deal of what Ben Blakemore termed “storming and norming”—sometimes heated discussion of the ways of working and how these would affect productivity.

The chosen medicine: they clarified their operational processes by creating a Ways of Working document (in essence, saying how Agile is there to support business goals), and also built a detailed checklist aimed at sharpening the definitions of Ready and Done for stories and tasks. Their checklist considered questions for tasks, risks, requirements, specifications, and tests. Their definition of ready involved elements from multiple areas of expertise, for example product owner (requirements 90% complete), risk management (UX specification 95% complete, with human factors review/inputs), quality (edge cases that need testing identified; integration environment ready), and test lead (identification of unplanned backend dependency). These clarifications helped immensely in assuring smooth project flow.

At Inpeco

Inpeco SA was in a similar state in 2017.  Headquartered in southern Switzerland and with development teams near Milan, Italy, they dominated the market for clinical laboratory automation systems. Their setups were handling hundreds to thousands of blood sample tubes per day in clinical laboratories across the world.

Their laboratory automation software is customized for each installation, running on third-party instruments to do analysis steps in a sequence defined by each laboratory.   

Figure 1. Physical view of a typical Inpeco automated-laboratory system

Each instrument and mechanical component involves a separate software development effort.  Under their old stage gate process, these efforts were carried out in isolation.

In their first attempt, Inpeco tried to institute Agile in all development groups at once—but not all teams were ready to make the change, and the implementation failed.

Still, enough people believed that an Agile approach would help them, so the transition effort started again. They collected success stories from other companies, trained in AAMI TIR 45, and studied how they might adopt an Agile approach in their own unique situation.  Since Agile appeared to be most applicable to software, they started with the software group - but elected to follow a gradual introduction, having gone too fast in the earlier campaign.

Both the QA lead and the software development lead were committed to Agile and worked closely to clarify the ways of working and introduce tooling to reduce manual testing.  In their ways of working, they needed to address risk management (patient and operator safety in their end product). Rather than being an observer in the Agile transformation, Quality Assurance became one of the drivers and worked closely with development to shape the process. Through the risk management effort, the Inpeco teams generated mitigation methods as inputs for software development. In time, they formed a team to work on each specific feature or automation module—a team that included a developer, a tester, and someone from Quality/Compliance.

Like Sanofi, they found the courage to - as a group - admit that their earlier attempt had failed.  They also clarified what the new process should do. The “medicine” that worked for them was to go a bit slower on introducing new practices, and to get creative by re-purposing the customer demo software into a development simulator. That contrasts with what the Sanofi team needed: tighter control over their Agile process which had been too lax.  This makes sense because Inpeco’s main mistake had been to charge ahead too fast on all fronts.

It’s interesting that both teams got themselves back on track by much discussion and consensus-building in the Agile team and those immediately around it.  This makes use of transparency and communication, which are also key for Agile teams.

At Bluefruit Software

Courage also appears in other contexts besides process reboot.  Bluefruit Software builds embedded software for clients, and has always used Agile practices exclusively. Bluefruit’s founder and employees - all engineers –  were excellent at responding to technical requests, but their sales process was time consuming and not very productive. Clients were often found by cold calling or connections through engineers.

With the hire of a new dedicated business development director, the sales process at Bluefruit began to get a new level of focus. Caitlin Gould had been consulting with them on business development and growth before becoming an employee. She recalled, “The biggest thing that hit me after I started at Bluefruit is we have a huge amount of training, drilling into us the Agile ideas around feedback loops.” She was referring to the training that the company gives all new employees so they will have a common foundation of Agile knowledge.  As she looked at the existing sales process (figure 2), it became clear to her that it didn’t line up with Agile concepts at all. 

In brief, their process had the engineers doing a lot of speculative work in order to create a detailed proposal up front of how they’d address the client’s needs.  As she looked into failed sales attempts, it was obvious that these big leaps of assumptions were a key factor. 

Figure 2 The company’s sales process before making changes

Gould introduced several new ways to break up the sales process to introduce more feedback loops. She experimented, in a number of different ways, qualifying clients, adding more value, or discoveing new client needs.  For example, they found that medical device clients placed a high premium on the quality that Bluefruit built in from scratch through practices like TDD.  One change they made was to explain TDD’s benefits early to potential clients; some didn’t realize that they would not need to do testing after Bluefruit delivered the contracted software.  They could just take over running the automated test suite. 

The upshot of the experiments and investigations she guided was that the company more than tripled their closes per year while generating fewer proposals per year, spending far less technical lead time on each proposal, and increasing their profit margin.  Figure 3 shows their new sales process.

Figure 3. Overview of Bluefruit’s Agile sales process. Each box after the first is an exit ramp to weed out unsuitable leads.

It took courage for Caitlin, a new employee who was also new to the Agile process, to take a critical look at the sales process and ask some very powerful questions like:

• Why are we making so many assumptions instead of talking more to clients?
• Why are we doing big design up front on proposals when we would never do a real project that way?

It also took courage for her to tell her colleagues that their sales process was a long way from being Agile, given that she was new to Agile and they had many years of experience using it. 

Of course it helps that the company provided Agile training for new employees and had the kind of culture that’s more willing than many to hear a different message. 

Courage take-aways

When things are not what they should be, when our plan isn’t working out, the first thing needed is the courage to admit it to ourselves, and the next thing needed is the courage to talk about it with colleagues – not for blaming, but for creating a positive path forward.  The reboot examples we saw were both like that.  Courage gave the spark, but the transparency and communication with colleagues was the fuel that put them back on a path to success.  In the sales example, the company was very Agile in their engineering practices, but as a growing startup there was some tendency to do other kinds of work in the traditional way.  The company’s own Agile training helped them by making the contrast between theory and practice even more stark for Caitlin, as she arrived new to Agile and somewhat new to Bluefruit (she’d consulted to them before). It would not be hard to imagine that a new employee seeing such a big gap would just look to exit quietly and see if another offer they turned down might still be there!  The company’s Agile culture made it less of a “heavy lift” to speak up and ask those questions.  Like the reboot examples, she simply asked important questions and invited all to help find the answers.  Transparency and communication paved the way to a better sales process.

Lean Principle: Autonomation

Autonomation means automation with a human touch. In other words, keep people in the loop so they can monitor the automatic machinery.  For today’s Agile product teams, this involves the use of software test suites and tooling that automates regression testing and also rapidly tests new software in multiple scenarios.

At OpenClinica

The OpenClinica product (for remote capture and management of clinical trial data) is highly configurable in the customer’s hands: a development environment for the data management customers, since every clinical study is different. Data managers need to be able to create their data definitions, their visit structure, the forms logic/events flow, their skip logic, and their edit checks. The incredible amount of configurability results in a huge number of permutations. Previously, OpenClinica ran as much ad-hoc testing as they could, and their whole development process would grind to a halt so they could begin extensive manual testing (even hiring part-time testers!) to make sure everything worked in context. Then, if an issue surfaced, there was always a question of how much testing to repeat. The process was incredibly labor-intensive and time-consuming.

When OpenClinica decided to invest in automated testing, they adopted the BDD approach and Cucumber technology, which allowed them to map requirements to testable scenarios. Requirements are the features and scenarios in Cucumber, based on the user stories written in JIRA (with acceptance criteria). Each requirement in Cucumber is tagged to link it to a user story in JIRA, but the Cucumber output provides the complete requirement set and traceability. Developers express each requirement in Gherkin “Given/When/Then” format (where some Givens include a large number of And clauses) and document them in the Cucumber tool. All product requirements take the form of executable tests; the Cucumber tool can then generate reports and a complete traceability matrix.

These tests do not have to wait until a release is being done. Instead they run nightly - keeping people in the loop - and discrepancies found can be addressed promptly.  This change to using BDD required a big investment, but all agree that it was worthwhile.

At Inpeco

Inpeco, the Swiss-Italian company mentioned earlier, found a novel way of bringing in software automation with a human touch.

Inpeco had hired a software engineer to develop a demonstration tool that could be used to show customers how an automated lab system would work. The engineer built a tool that worked as closely as possible to a physical automation system. When the Inpeco software lead saw it, he and the QA lead both recognized the opportunity to extend the demo tool so that Inpeco’s own software modules would interface with it. The demo tool then became a simulator, permitting the software developers to build and test much faster than before and allowing tests to be run at a scale that simply weren’t possible otherwise.

As customers requested ever-larger automation systems, the developers realized they couldn’t test these systems at all without the simulation capability. Now, the software group has five or six engineers working on the simulator, refining and expanding it along with the product software. The better the simulator, they have realized, the better they can build and test their systems.

At Atomic Object

Atomic Object does full-spectrum product development work with and for their clients.  They co-evolved a medical device product, Quell Relief, with their client NeuroMetrix.  Atomic Object uses Agile product development processes exclusively.  That means they use TDD, set up Continuous Integration for every project, and further DevOps practices as appropriate for the situation.  They view this foundation as critical to their ability to deliver for clients.

While most mobile apps are 100% software, existing only on the phone or tablet, the NeuroMetrix mobile app interacts with a hardware device through a wireless Bluetooth connection. Therefore, a large portion of the app’s behavior is dependent on the signals it receives from the hardware device. That connection to a physical device makes testing far more challenging because Bluetooth cannot be simulated with the iOS simulator. When you run your app in the simulator and get to the screen where it should connect to the real device, it can’t use your computer’s hardware to connect to the Bluetooth device. Atomic Object knew they still needed to test various hardware states, so they wrote tests and exercised the app to access features that were in some cases impossible to reach even with a real device, for example, error states. They built a “shim” right above the hardware layer, which allowed them to create an in-app simulator of a physical device, programmatically controlled through a back door.

Figure 4. How software is tested with or without the Quell Relief product hardware present

Jordan Schaenzle, technical lead for much of the NeuroMetrix project, stated, “I could control our app in a simulator and then send a command over a socket to tell the device to go into low battery state and then see the app reflect that change to be sure that it worked. A company that wasn’t focused on test-driven development (TDD) probably would not have made that investment.”

By taking the hardware device out of the testing loop, he could use the simulator software to make the mobile app behave exactly as it would if it received a real ‘low battery mode’ message from the patient’s hardware device. This and other unusual states like interrupted or invalid sensor data can be created readily, and the mobile app software being tested will execute the very same actions as if the real hardware were in that unusual state.

This capability is valuable for documenting risk management processes and demonstrating them for QA, RA, and business stakeholders.

Autonomation take-aways

Each of these examples fulfills the principle of “automation with a human touch”, but implements it uniquely to the situation. OpenClinica uses the powerful Cucumber tools for their massive data sets. Inpeco happened to have an in-house app meant to show customers how a set of laboratory instruments would operate together and they saw an opportunity to make it into a simulator for testing their lab automation software.  Atomic Object needed to do thorough testing involving the Quell product’s hardware, so they built a simulation layer to decouple software and hardware.

As Schaenzle commented, companies who are not focused on TDD probably would not have  made the investment in the simulation layer. It takes a considerable investment - in time, tooling, and skill building - to put the foundations in place for autonomation, but the payoff in quality is very high. Truly Agile organizations carefully guard their ability to implement autonomation.

Insights

We’ve illustrated only two core principles here, but further cases we’ve explored show others in a variety of contexts. Examples include feedback, respect, communication, transparency, “add nothing but value”, “collaboration based on trust”, to mention a few.  The companies we’ve profiled have applied Agile and Lean principles successfully, by applying the core principles in ways unique to their own circumstances. It would  be tempting to copy one or another of these cases, but understanding the core principles and how they can be applied is the true route to success.

About the Authors

Nancy Van Schooenderwoert is president and principal coach at Lean-Agile Partners. She was among the first to apply Agile methods to embedded systems development. As an engineer, manager, and consultant, she has led Agile change initiatives for software and hardware projects in safety-critical, highly regulated industries, including aerospace, factory automation, medical devices, defense systems, and financial services. She has been coaching Agile teams and managers since 1998.

Brian Shoemaker provides consulting and training in computer system validation, software quality assurance, and electronic records and signatures. He has conducted software validation in a variety of FDA-regulated settings, from the embedded applications driving immunodiagnostics instruments to custom applications for clinical-trial data management. He has also designed and instituted quality systems for software development. Shoemaker is principal consultant for ShoeBar Associates LLC; he earned his Ph.D. in chemistry from the University of Illinois, and has achieved the ASQ Software Quality Engineer certification.