By using Model Driven Development component tests could be skipped and integration and system testing went a lot smoother, said Bryan Bakker in the presentation Model Driven Development (MDD) and its impact on testing that he gave at the International Conference on Software QA and Testing on Embedded Systems. In his talk he showed how model driven development has been used to redesign software that is used to control an electron microscope. The main results from the MDD approach are a reduction of the amount of testing that is needed to validate the software and increased reliability of the code that was generated from a mathematical model.
The behavior of a part of the electron microscope software was modeled with a modeling tool, and the tool was used to check for certain issues like deadlocks, live-locks, starvation, and race conditions. Since this software is a critical part of the system the reliability needed to be validated with testing.
A model must be precise, complete and correct. To ensure this a model checker was used to validate the model and remove any errors. Then the model was used to generate code. Since the code is generated from the model that has been checked, the code can be guaranteed to be correct.
The model consisted of state machines, sequence diagrams, algorithms. It includes the normal flow and also the exceptions and illegal behavior. It contained both internal interfaces and interfaces to the existing hand written code, using a design by contract approach. Bryan stated that it is important to decide which part of the software will be modeled. He also advised to not try to put everything in a model, such a model will become too complicated.
A question was if the code which has been generated needed to be tested? Since the correctness of the code is guaranteed by the model checker, the assumption was that component tests would no longer be needed for the generated code. Integration tests and system tests would still be needed.
The tests that were done were automated using a GUI for test actions and verifications. System testing was done to test the reliability of the system, those tests were executed for a long time to increase the chance that failures due to for instance race conditions would be happen.
Reliability faults were found in testing but none of them were in the generated code. Some faults turned out to be in the boundary between the generated and hand written code, most of the faults were in the hand written code. Conclusion was that the reliability of the generated code is very high due to using model driven development. With the use of model driven development hardware failures and wear were detected much quicker, after days of testing in stead of months. The testing revealed several major design flaws in the mechanical parts.
Main results from the model driven development approach are that that component testing could be skipped and integration and system test went a lot smoother. When a module is redesigned with MDD the likelihood of defects goes down, which reduces the risk of failures. Reliability testing can be used as a measurement, since less failures are found and the testing will confirm that a product is reliable.
There were several additional benefits gained from using model driven development:
- Reviewing the models turned out to be easier than reviewing code.
- Design discussions could be done earlier which helped to prevent defects.
- Refactoring turned out to be relatively simple and reliable.
- Maintaining the models appeared to be easier than maintaining code.
Model driven development fits well into an agile approach. A sprint would deliver hand written code, parts of the model that could be used to generate code, or both. The model can be demonstrated to the stakeholders in the demo.
