Yes, Hardware Can Be Agile!

“You can’t do 2-week iterations with hardware!” This is the first thing you’ll hear when talk turns to Agile methods in hardware-software product development. A mix of existing robust hardware development ideas, plus a few newly taken from Agile software are being used now by real teams, even to get around - or through - the challenge of doing fast iterations.

Hardware roots of the Agile movement

What we now call test-driven development (TDD) grew from embedded systems research work by Ward Cunningham, and from his collaboration with Kent Beck. In the 1970s Ward was working at a research lab looking for a practical way to test a new kind of hardware device. As he described it to me: “Microprocessors were new in that era and I came to a conclusion that you had to get the machines to tell you what they were doing. And that if you wanted to trust what you got the machines to tell you, that part had to be very simple… That aspect was imprinted on everything I’ve done since.” Kent also had his own experiences of a similar nature.

I met Ward at one of the early Agile conferences, and I was curious to know what led him to the set of Agile practices that became “eXtreme Programming”, as promoted by Kent Beck. Ward told me that in the 1980s he and Kent Beck worked together on Smalltalk at Tektronix, arriving at a pattern of simple practices that was surprisingly productive when combined.

As Ward described it, the company had a bureaucratic process to go through once a new product idea got to a certain point. To avoid having to do all that paperwork, the two of them would take a new idea just so far and then get another group to adopt it. Then they’d repeat that performance – getting good at quickly iterating new concepts, and proving enough viability so that others saw it as worthwhile to pick up the idea and run with it through the internal bureaucracy.

From there, practices and concepts grew – pairing, simplicity, “test first” which evolved into TDD later on. In this way TDD, a foundational Agile practice, got its start in hardware-related work.

Agile ASIC Design borrows TDD from Agile Software

Inspired by the way TDD is taking hold in software development, Neil Johnson – a hardware verification engineer – decided to try it for his work verifying SoC (System on Chip) designs. 

To understand what a verification engineer’s work entails, a look at an analogous work pattern in a different industry can help. A verification engineer works with a circuit designer in SoC development in a pattern similar to how a seamstress works with a fashion designer in that industry. The figure below shows the work steps in producing a new clothing outfit. In the last step, the designer is ready to troubleshoot her design only if the seamstress has done her tasks without introducing any defects. If she has used the wrong fabric, or measured the model incorrectly then the designer will have to deal with problems that are not due to flaws in her design.

Figure 1. What a verification engineer does - by analogy with fashion design.

Figure 2. What a verification engineer does in working with a circuit designer.

In the same way, an electronic engineer designs a circuit and the verification engineer validates the design by writing tests to confirm the functionality on the SoC. If the verification engineer introduces bugs when writing the tests, those bugs obviously threaten the integrity of the test effort.

Having to go through extra cycles just to find and fix bugs introduced in verification adds enormously to hardware development time. So Neil wanted to know how much cleaner his verification work would be if he used TDD for the step “Write the test code.” After some work to put a TDD framework in place and a few weeks to get accustomed to the TDD work flow, he was ready to gather measurements.

Neil benchmarked his results both before and after using TDD, and concluded that when using the SVUnit framework for TDD, his tests had 1 bug where his traditional methods of SoC verification had 15 bugs. This is admittedly based on just one data set, but is still an impressive improvement, especially considering that Neil is an experienced verification engineer.¹

Agile IC Development with Working Hardware Every Sprint

A Digital Design supervisor at Texas Instruments in Germany was interested in using Scrum to speed up delivery of custom ICs – for Flash and FRAM memory². Their normal delivery period was 12 to 36 months, and at least 18 months for this particular IC, but a new customer needed much faster performance. How could Scrum speed up IC development?

At the start of the project, they looked for a Scrum coach to work with them, but could not find any familiar with their domain. Still, they were sure that at least some Scrum or Agile practices should be transferrable to their hardware work.

The team chose to adopt these Agile practices:

• Cross-functional team
• 4-week iterations
• Relative estimation (it is bigger/ smaller than a known feature?)
• Standup meetings 2 or 3 times per week
• Working hardware every iteration
• Retrospectives

The key problem was how, exactly, to deliver working hardware to the customer at “every iteration” when the chip fabrication process is much longer than 4 weeks, and not in their hands. The rest of the Agile practices could readily work with hardware.

One way to work incrementally would be to deliver FPGAs programmed to behave like each of the IP blocks within the whole IC. (Normal internal cycle time for one IP block was 6 – 12 months.) They could deliver each of these as it’s completed, and offered to do so. The external customer didn’t want to bother with this interim step even though it would reduce some risk. They only wanted the final IC.

Without the ability to iterate with their external customer, the Digital Design team found a willing internal customer: They could deliver the FPGAs to the Software Engineering team within TI. Their software is the next step before silicon fabrication.

The Digital Design team began delivering their FPGAs each month to the Software Engineering team. With a working hardware prototype as input rather than just a design, bugs had fewer places to hide. The goal of working hardware every 4-week sprint was achieved although with the FPGA hardware as a prototype for the IP blocks. It had the effect of taking the Digital Design team’s cycle time down from “6 to 12 months” for an IP block, to “every 4 weeks”.

What about the cycle time for delivery to the external customer? Unfortunately, the project was stopped before completion for outside business reasons, so there isn’t data available for how much that cycle time could be reduced.

Agile practices gave other significant benefits besides the virtual shortening of IP block cycle time. The Digital Design team was working with a new platform, which alone adds significant uncertainty and complexity. Using their waterfall method would have meant adding a 12-month specification effort to the front of the work.

Tobias Leisgang, who supervised the Digital Design team, observed “Planning poker and relative estimations helped engineers to uncover communication issues and misunderstandings about scope and get a more realistic feel for the overall effort, but without a multiple week detailed planning phase. Cross-functional teams made sure feedback loops are getting shortened and you don't produce an expensive piece of silicon before the test engineer finds out the analog circuit can't be tested.”³

Agile Electronics with Prototype Circuit Boards

With a bit of advance planning, iteration can work as a main strategy, especially for early prototyping of a product’s circuit boards and other physical components. A cross-disciplined team at Schneider Electric developing a lighting control system were able to put together 3 circuit boards, and alarm lights within one sprint.

The team’s leader, Timo Punkka⁴ points out how their RF-controlled light dimmer system evolved inside a 4-week Sprint:

“The development team consisted of firmware, electronics, PCB layout and plastic designers. The prototype included 3 PCB’s; a power, a control and an UI including the RF transmitter. The first iteration had only the power PCB ready and the other 2 were replaced with common prototyping boards. On second iteration all boards were available and at the end of the Sprint rapid prototypes for plastic parts arrived. In the Sprint review prototypes gave a real feel of how the product would look like if this concept was to be selected. Firmware was developed for a simple on/off functionality. To be able to do this you need trusted partnership with prototype manufacturing. In this case the prototyping shops agreed to deliver the next day after receiving final drawings on earlier-agreed time.”

(Click on the image to enlarge it)

Figure 3. Light dimmer system prototyped within a 4-week sprint.

They validated their idea of using spring-loaded connectors and magnets to connect each module (similar to the Mac power connector). They also tried out building the RF part including antenna in a pcb-size SD card. Both of these experiments involved a lot of uncertainty – they worked but no one could be sure in advance that they would.

For the team, the result of these practices was to put experimentation forward – make it the leading activity, with the other Agile practices supporting:

• TDD guaranteed that they could start experimenting with tests early on and continue to experiment without breaking early design decisions
• Modularity allowed them to isolate areas needing more experimentation
• Simulation enabled experimentation where it is too expensive if done with real targets

Timo observes that experimentation, or emergence, is at the very center of agile thinking. It’s a 180 degree difference from get-it-right-the-first-time thinking. By building a prototype inside of one sprint, it allows hardware, software, and requirements to emerge concurrently.

Agile Car Development

A street-legal car, built by volunteers, revised weekly, that you can buy right now – and it gets 100 miles per gallon of fuel! The Wikispeed project challenges the idea that it has to take years to bring a new car model to market. It was developed in only 3 months, and volunteers continue to build and sell the car. The primary strategy is modularity. One or more of 8 modules is revised each week and is quickly integrated, thanks to stable interfaces. This amounts to a kind of staggered iteration approach that allows for experimentation and learning to happen in parallel, within each module.

Figure 4. Wikispeed car exploded view showing its 8 modules.

Another key strategy is simplicity in the type and number of materials: “A core XM (eXtreme Manufacturing) practice is to only use materials that can be inexpensively iterated (re-made) in less than a week, and then choose the fewest number of different materials / processes practical in order to build deep experience quickly.”⁵

Joe Justice, a Seattle-based software consultant, leads Team Wikispeed, an international group of volunteers building cars using Agile techniques. In 2008, Justice saw the announcement of the Progressive Insurance X Prize—a $10 million prize aimed at seeing if it was even possible to build 100 mpg cars to road-legal safety specifications. He entered it and blogged about his ideas – and readers joined him.

Joe was surprised how many people were willing to work hard, and for free, to help build something like this that would be open sourced so all can benefit. There are active Wikispeed shops in 23 countries at present. The car sells for about $25K, or you can build it from a kit for $10K.

His second key learning from Team Wikispeed is about people’s motivation. He says Daniel Pink’s ideas on Purpose, Mastery, and Autonomy are very much in evidence among the volunteers. They take initiative, learn from each other, and keep on paying it forward.

The Wikispeed car is made from 8 modules (engine, frame, carbon fiber body, etc.) that can be swapped out easily because they are loosely-coupled. Each week, one or more of the 8 modules is revised. There is always a running version of the car, and upgrades can be implemented very rapidly. People who already own the car can buy a new module and swap it in themselves without having any automotive technical background.

In the Progressive Insurance X Prize race, the Wikispeed car tied for tenth place in the mainstream class, outlasting more than one hundred other cars from well-funded companies and universities around the world.⁶

Team Wikispeed’s lessons learned about “eXtreme Manufacturing” (XM) reflect the fact that Joe Justice and many of the volunteers come from an Agile software development background. They bring this orientation to XM. Asked to describe what has been learned about applying Agile methods to manufacturing, and engineering, Joe’s response is:

When I started co-teaching with Jeff Sutherland, I advised engineers to use object oriented architecture, contract-first design, and TDD. My first big learning was Jeff telling me that I needed to develop those ideas for engineering - they’re obvious to software people but they need to be translated for engineers.

For doing Extreme Manufacturing (XM),

Step 1 is the first sprint: split the product or service into less than 10 loosely-coupled modules.

Step 2 is to figure out the object oriented architecture; the inputs and outputs to each of those modules and how they connect, physically and with data, cooling, etc., but intentionally not design or architect the insides of any of the modules. Just their interfaces.

Step 3 is to iterate and incrementally learn by doing as the team builds and re-builds versions of each module. Once they get to working proofs of concept, they add the “illities” to each module, like durability, affordability, etc.

There is a classic problem that happens every time with every team of engineers: They want to do all the architecture up front - all the interfaces and all of the internals of each “box” (subsystem behind the interface). That’s a huge lead time and lots of waste.

Some want to jump in and start building with no architecture. That starts fast but slows to a crawl pretty soon.

The one thing they rarely do is the optimal thing: Architecture up-front of just the interfaces and then stop architecting, and start learning by doing, and build the “box” internals. That’s what seems to be the very fastest way to a shippable product and keeps sustainable speed.

In a complex physical product, some blend of iteration and parallel approaches will be optimal. This is especially true the more there is a “discovery component” in the nature of the work. An example is the way the Wikispeed car’s modules are developed in parallel with each other, with most changes staggered so they do not all fall at the same time. This ensures working hardware and software is delivered every week.

Wrap up

We’ve seen some classic techniques of hardware development:

• Simulation or prototyping (IC development, Electronics)
• Modularity (Electronics, Wikispeed car)
• Simplicity of ingredients (Wikispeed car)

We’ve also seen some direct borrowings from Agile software practices:

• TDD (ASIC verification)
• Short iterations (IC Development, Electronics, Wikispeed car)
• Object oriented design (Wikispeed car)

Hardware has been moving toward more flexibility for a long time. Some hardware can be changed very quickly, making it practical to allow good designs to emerge instead of having to be completely planned in advance. Where that flexibility level does not yet exist, we naturally use a mix of simulation, modularity, and advance planning. Locking down a design far in advance is a very risky strategy, and these techniques mitigate that risk.

Another very powerful mitigation has been touched on here – crowdsourcing (or tapping the collective intelligence of the team) is at work in key Agile practices like retrospectives, pair programming, daily standup meetings, and story writing to name a few. Collective intelligence is a strong effect in any well-functioning Agile team.

Yes, it’s often difficult to work in short iterations and have working hardware at every iteration, but we’ve seen these tactics help real teams do it:

• Classic hardware moves like simulation and modularity offer great flexibility and speed.
• Agile software practices like TDD and short iterations are being imported to hardware development where they accelerate experimentation and learning.
• Sometimes you need to look beyond the requested capability (e.g. completed ASIC in a month) and fulfill the larger need of the customer – or the internal customer. Tapping the collective intelligence of the team will find the creativity to do this well.

With these ‘thought tools’ in your hands, hardware can definitely be Agile.

Sidebar:

Help the Agile Engineering Community to Emerge!

Are you using techniques in your hardware development work that fit Agile principles? Why not share them with the community…

The Agile Engineering program is an initiative within the Agile Alliance to collect short experience reports on techniques in use that support an Agile approach in engineering areas beyond software engineering. These are made freely available to all. Further info here.

About the Author

Nancy Van Schooenderwoert was among the first to apply Agile methods to embedded systems development, as an engineer, manager, and consultant. Beginning in 1998 she has led Agile change initiatives beyond software development in safety-critical, highly regulated industries, and coached clients in the art of Agile technical and management leadership. A contributor to Agile Times and Cutter IT Journal, she also served on IEEE 1648 Agile Customers committee. She speaks at numerous software conferences worldwide, and is past president of Greater Boston’s premier Agile user group, Agile New England. Nancy is founder and President of Lean-Agile Partners, Inc. She can be reached at nancyv at leanagilepartners.com, @vanschoo.

¹ Johnson, Neil; TDD and A New Paradigm for Hardware Verification; Agile 2014 conference, 2014. Available at http://www.AgileSOC.com

² Leisgang, Tobias; How to Play Basketball With a Soccer Team (paper); Agile 2012 conference, 2012

³ Tobias Leisgang, Systems Engineering Manager, Texas Instruments in Munich Germany; interviewed by author.

⁴ Punkka, Timo; Embedded Agile (paper); pub by Embedded Systems Conference, 2010

⁵ http://wikispeed.org/category/wikispeed_shop_manufacturing/

⁶ Denning, Steve; Wikispeed: How a 100 mpg Car Was Developed in 3 Months; Forbes, May, 2012. Available at http://www.forbes.com/sites/stevedenning/2012/05/10/wikispeed-how-a-100-mpg-car-was-developed-in-3-months/