InfoQ did an interview with Yaniv Yehuda, Co-Founder and CTO of DBmaestro, about how they are doing agile development and using DevOps, how they implemented continuous delivery, agile practices that turned out to be difficult to implement, and the benefits that they are getting for using agile and DevOps practices.
InfoQ: Can you share some of your experiences to explain to the InfoQ readers how you are doing agile development at DBmaestro?
Yehuda: A few years ago, we were working in a waterfall model. We were generating large versions with long development cycles, and long test periods for those big features.
It was very hard to issue improvements, as they were always bundled into the "next release". Customers waited months for requested features or improvements. Often, releases did not meet their deadlines – and not because people did not do their jobs as they should – things always proved to be more complex than anticipated, as they were large and hard to break down in advance.
It was also hard to get feedback from customers because "on paper" we suggested a feature that they thought made sense, but they had no way to "feel" what we were building. On at least one occasion, a sizable feature was built never to be used. It made sense when it was presented, but it was implemented in a way that rendered it impractical.
We decided to move to an agile development process, practicing two week sprints. Doing so helped us focus on smaller units of development, breaking work into manageable sized tasks. We saw immediate benefits in three major areas:
- Our ability to better estimate timelines: Tasks were now smaller and related to more focused sprints, with less of a chance for severe mistakes.
- Significantly improved test cycle and feedback to developers: The ability to efficiently test shorter sprints, quickly find out what’s working and what’s not. In addition, being able to get feedback to developers relatively quickly regarding issues, bugs or challenges made it easier for them to fix things.
- Better product quality and overall better visibility of the development process: This was a result of the above.
We ended up practicing a flexed version of Scrum – actual sprint length is determined before it begins according to planned tasks. We usually stick to two weeks sprints, but sometime change it to one or three weeks. That has proven to be more efficient for us than following fixed length sprints. Some purists say "that’s not Scrum" but it doesn’t really matter – we found that dynamically balancing between the focus on achieving the quickest time–to-market and efficient development of larger features, works best for us.
InfoQ: What made you decide to start with DevOps practices?
Yehuda: We really had no choice. Moving to agile development alone definitely improves some things, but we were not happy with the bottom line.
We were unable to issue short development cycles to our customers. True, development cycles were short and efficient, but nothing was "releasable". Overall products could not be realistically tested in a short period of time, and we ended up accumulating sprints in order to deal with "release to production" for our customers.
Implementing DevOps was divided into two categories:
- Making sure development is highly connected to production aspects, customers’ needs, long term goals etc. That connection has proven to be very important, as customer value became the main discussion company wide.
- Implementing continuous delivery:
- Essentially automating as much of the process as possible: the build process, testing process and final packaging.
- Making sure we have a clear definition of done for each releasable feature, which in turn has a clear definition of value to the customer.
DBmaestro is a database source control and deployment automation solution, so we’ve had the chance to eat our own dog food. We are using our own solution to build its own next release. That by itself assured we spend not just theoretical quality assurance time, but also practical, end-to-end usage of each of the features, as they are being developed, raising the bar as high as we can.
InfoQ: Can you elaborate on how you implemented continuous delivery. Which practices and tools do your teams use?
Yehuda: We practice trunk based development, which means that all of our changes are done on the current trunk. We try not to branch, but if branches are required, all changes are first introduced into the trunk and then piggybacked into the relevant branch.
We use a combination of MS-TFS build server, Jenkins and our own DBmaestro for the database side of things. Our build process is initiated, testing matrix is performed and if everything passes correctly, we have a stable package to upload to our website.
InfoQ: Are there any agile practices that turned out to be difficult to implement? Any that you decided to stop doing? Why?
Yehuda: Automated testing. No doubt about it:
We tried unit testing, and reached the conclusion that it would slows down development too much. Creating unit tests for a new project can be effective, but trying to retrofit our large codebase did not make sense from time & costs perspective.
We tried using test driven/automation tools, but discovered again, it takes too much time to build each process in, and by the time a scenario was working properly in the test tool, changes were made to the product which made it obsolete. Neglecting to update the test scenarios in order to deal with some urgent deadlines, proved to be the bullet that killed it in the end.
We reached the conclusion that as most of our product is back-end processing, less GUI intensive, and changes are usually implemented in the "engine", we should try and make some concessions where possible: instead of trying to test everything automatically, we would manually test the smaller, less impacted user interface bits, and focus on regression tests in order to make sure that nothing breaks when we introduce changes. We constructed very large array of scenarios, and ran our product through that array. The test was constructed in such a way that running it start to finish, resulted in a clear overall pass or fail, and if it failed, it would be very clear why.
InfoQ: Which kinds of benefits are you getting for using agile and DevOps practices? Can you give some examples?
Yehuda: We take great pride in customer centric approach and our ability to react quickly; Making quick enhancements, changes or fixes - and handling customer’s needs.
A few weeks ago, one of our customers approached us with a specific request to handle an issue differently than our usual approach, it was very important for him that it would be done to his specifications because he hoped to use it in his next agile sprint. An ad-hoc team was assigned to the task, making required changes, testing and regression testing everything. They succeeded in delivering a next-day hotfix package.
That was a clear victory for a value-to-customer, goal oriented team, with a great process, and efficient automation to make it all happen.
In my eyes the benefits of agile development and DevOps practices are all rolled into that one simple but efficient experience – being an agile company.