Agile helped a data science team to better collaborate with their stakeholders and increase their productivity. As priorities became clear, the team was able to focus and deliver. Buy-in of the data science team by taking them through a journey of agile was crucial to making it work.
Snigdha Satti, a product owner and senior business analyst at News UK, shared her experiences from using agile with a data science team at Agile Tour London 2020.
When starting her assignment with the data science team, Satti mentioned that the first thing she did was to speak to each and every member of the team and their stakeholders to understand each individual’s pain points:
It was quite clear that the team didn’t have a clear direction and motivation, which led to loss in productivity and eventually missing the deadlines. Similarly, stakeholders were frustrated that things were being promised but not delivered.
Satti expected that the team and stakeholders could benefit from agile:
Agile is an iterative process; so is Data Science. Data Science involves the four steps in each iteration- investigation, exploration, testing and tuning.
The idea for applying agile to data science was that all four steps would be completed in each sprint and there would be a demo at the end. When applied this way, they could understand together if the agile model was feasible or not.
Satti conducted agile ways of working sessions with the team to teach them the importance of collaboration, interactions, respect, ownership, improvement, learning cycles and delivering value. The team had to go through a cultural and mind shift change because they believed that agile in data science would only work if data scientists understood and trusted the advantages of agile, Satti said.
The main benefit of introducing agile to the team was that they saw an immediate increase in productivity, as the team members were clear on their priorities and were able to focus on the specific task, Satti said. Due to this, the team was able to commit to deliverables and timelines. Most of the time the committed deadlines were met, making the stakeholders happy, hence increasing the confidence in the team.
Having the buy-in of their Data Science team was quite crucial and they had to be taken through a journey of agile instead of forcing it on them, Satti mentioned. She did this by having 1-1 conversations with the team to understand their challenges and then explaining to them how agile would help with their issues.
The team could see it working as quickly as sprint 1, when they saw the backlog forming, all the ceremonies being scheduled, and communication improving within the team.
InfoQ interviewed Snigdha Satti about how data scientists can use agile.
InfoQ: What challenges were you facing?
Snigdha Satti: Couple of years ago, I was working in a Data Technology programme within News UK. Within the programme, one of the teams was a Data Science team. The team consisted of Data Scientists (based in London and Bangalore) and a Scrum Master; they were working closely with another senior member of the team who was managing the stakeholders and who also had exceptional data skills.
One of the biggest challenges was that the team did not interact with the business stakeholders and didn’t have the knowledge of the organisation’s vision, goals and priorities. They were just getting direct requests from the senior member of the team on what models needed to be built and which datasets were to be used. This made the team a bit demotivated, as they were unable to understand how their work was contributing to the bigger picture.
Another big challenge was that the team was getting many requests at the same time. Instead of saying denying requests, the team was picking them all up and trying to juggle them all at the same time. This was also because they didn’t understand the priority and no one in the team helping them understand this.
This was causing lack of focus for the team.
Trying to work on multiple things at once also meant that they were either missing the committed deadline or unable to give the right estimates.
This led to friction between the stakeholders and the team.
InfoQ: What made you decide to apply agile?
Satti: I was working in a different team within Data Technology and I was asked to come into the team and help resolve the issues that the Data Science team was having. After interviewing everyone individually, I realised that what was lacking in the teams was a clear set of goals, individuals interacting with each other and collaboration, and responding to change.
InfoQ: How did you apply agile for data science?
Satti: In the agile ways of working sessions we covered the fundamentals of agile methodology, such as:
- The iterative nature of agile and its advantages
- The roles within the agile team
- The basics of SCRUM and KANBAN frameworks
- Ceremonies within agile and how they can be useful to the team
The team was quite optimistic towards the upcoming changes, as they knew that things were not working as expected.
Then, the next step was to have a chat with the senior member of the team to explain the issues faced by the team and suggest to not go directly to the team with new requests. The idea was to put in all requests in the backlog so that their priorities could be discussed during the planning sessions without overwhelming the data scientists. The senior member of the team and myself were responsible for speaking to the stakeholders to understand the goals and priorities of the requests.
Then I set up all the ceremonies for the team - standups, planning, demo and retrospectives.
All these sessions added value to the team.
InfoQ: What worked and what didn’t work? What did you learn?
Satti: We found the planning sessions and retrospectives to be the most useful out of all of the ceremonies. Planning gave the team a good overview of the upcoming tasks, as well as the ones that were needed to be picked up in the current sprint. By the time the sprint started, they had clarity on priority, complexity and effort needed.
Retrospectives were quite useful in celebrating the success. Also, the team was able to give feedback to each other and talk about the challenges faced. One of the examples is that the team in Bangalore was working outside of their working hours and no one was aware of it until it was mentioned in one of the retrospectives. From then on, other team members were conscious about sending the Bangalore team messages during their out of office hours. Little things like these helped the team morale a lot.
One of the main things that didn’t work for the team was that we couldn’t estimate the tasks. This is mainly due to the fact that when the problem is given, there isn’t always a clarity on what data to use, if the data is available, if it is clean etc. All this exploration has to be done as part of working on the Data Science Algorithm.
My biggest lesson was that one needs to be flexible, and that there are no hard or fast rules. Do what works for the team.
InfoQ: What benefits did you get?
Satti: The main benefit was an immediate increase in productivity, as the team members were clear on their priorities and able to focus and commit to deliverables and timelines. When committed deadlines were met, it made stakeholders happy and increased their confidence in the team.
Demos were also quite useful to keep the stakeholders updated on the progress of the work being done by the team, again, increasing the confidence in the team.
Having daily standups improved communication within the team and gave them the opportunity to catch the anomalies in time.
Overall, the team benefitted by winning internal awards and accolades for producing some of the best Data Science models. One of the models that was used by the Marketing team resulted in them winning an award at an annual marketing conference.