In the Digital Transformation day at the Agile India conference Fred George gave a talk on how the way we solve programming problems needs to change when dealing with what he calls fuzzy problems. He made the argument that when dealing in the "complex" domain of the Cynefin feamework, the speed of response is more important than any other factor, and that the models of software development we have been using don’t support the rapid iterations needed, where software can be deployed to the live environment in cycles of minutes and hours. He said that the ideal development "team" in those environments consists of a single developer working directly with a customer deploying software in very small cycles.
George started his talk, titled IoT and Microservices, by explaining that traditional and agile practices were well suited to the "simple" and "complicated" problem domains, where the problem is understood, and how to address the problem is known. Agile development practices and teams are effective in the problem areas where there is a need to learn about the problem domain but the underlying requirements are already relatively well understood or can be understood.
He then explored the types of problems which do not fit into the simple and complicated domains – these are in the "complex" and "chaotic" domains in the complexity model, where the requirements are very unclear, and how to solve the problem requires rapid learning, feedback and adaptation. He gave an example of the fuzzy decision-making involved in such areas as granting loans for a financial institution, day-trading, google add-words and identifying fraudulent transactions.
In these fuzzy areas, the best way to address the problem is for someone with development skills to sit with the customer, write the code, deploy it and show the results to the customer in very short cycles; in this way both the clarity around the actual problem and how to solve it will emerge from the rapid cycles.
He then used the development of a set of microservices he built to control the IoT ecosystem in his home as an example of how microservices architectures support this way of working, by enabling the development of many small services each with an independent responsibility coordinated via a event/message bus using the "Need Pattern for Asynchronous Communications" (the code is available on his GitHub repository).
He presented the architectural principles he used when designing the system:
- Uncertainty => just-in-time design
- Behaviour-oriented services => services as small as possible with well-defined behaviours
- Publish conclusions => if the service does something interesting, publish it to the bus
- Reliability through idempotency (repeat the same action if needed) and redundancy of all the elements
George walked the audience through the incremental delivery of a number of microservices using examples and scenarios. He concluded the description of the system with some observations:
- Lots of messages appearing on the bus
- No RESTful interfaces between services
- No constraints on multiple instances/versions
- Unit tests are not required because the microservices truly are so small that it is possible to understand their behaviour without needing tests
- Fast-failing system rather than acceptance tests
He then discussed how this development approach for fuzzy problems differs from agile development.
- The extreme programming (XP) values of "Communication, Simplicity, Feedback, Courage and Respect" are critical
- None of the agile practices should be needed because the team is so small (programmer and customer) and tightly coupled -- even pairing is optional
- All the agile roles except developer and customer are not needed
- The role of the customer is not to define the requirements, but to teach the developer the domain and collaborate on understanding both the problem and the solution
He was clear that this approach is not needed (or appropriate) for every development initiative, but for those in the fuzzy domain this is the best way to actually solve the problems.