Jesse Anderson, managing director of the Big Data Institute and author of Data Teams, recently published the latest findings from his annual Data Teams Survey of 81 data leaders and engineers across a range of enterprise types. The 2024 survey shows a slow adoption of LLMs by data teams, a neutral impact from remote work, and an overall decline in the perception of delivered value. The survey also calls out that data-operations capabilities are missing from half of the respondents.
The Data Teams Survey’s analysis emphasised the importance of spreading skills and cognitive load in the data domain across teams with data science, engineering, and operations capabilities. Anderson wrote that while data science and engineering teams are well-represented, "operations teams are lacking in half of the respondents." Further, he also highlighted inconsistencies in how data teams tend to define the role of operations engineers, with only 20% aligning on a definition which covers systems engineering skills. He wrote:
The individual contributors must meet the criteria and definitions to represent the job title. We see well-represented responses for data scientists and data engineers with operations lagging.
Anderson also published another article looking at the evolution of results from Data Teams Surveys between 2020 and 2024, in which he commented on the methodologies and practices used by data teams. Anderson wrote that over time he’s seen a "decline in DataOps and an increase in teams not using a methodology" of any kind. Anderson wrote that "choosing a methodology is still essential" for successful teams.
Commenting on differentiators between high and low-value teams, Anderson wrote that high performers followed "more best practices." The survey showed that these practices include engineering norms such as unit testing, close business collaboration, picking the right technologies, and having appropriate cross-functional skills.
A recent episode of the Data Engineering Podcast discussed the need for increased engineering and operational rigour in data teams with Petr Janda, founder of Synq.io. He started Synq, a data operations reliability and observability platform, following his experience with data teams that measured incident recovery times in weeks, compared to hours for other engineering teams. Janda talked about the need for incident resolution processes which factor in a granular, data specific, impact on stakeholders and downstream systems. Calling out missing processes, he said:
Everyone was focusing on the problem of detecting issues … So once we detect that a certain table is missing data or that a certain test is now failing because some business validation is not met, what happens afterwards? What is the workflow from the perspective of finding the right team to deal with that issue, and assessing the impact on the company? Is this even an issue we have to deal with right now, or is that something that can wait and be dealt with later?
Anderson also reviewed participants' responses to questions about their teams' delivery of value to the business. Looking back over previous surveys, he observed a downward trend that he said, "should give everyone in the industry pause." Anderson noted that many respondents "highlighted a disconnect between data teams and business needs." This was attributed to insufficient "domain knowledge" and issues with "communicating data value to stakeholders." Anderson wrote:
I’ve been seeing this lower value creation trend anecdotally, and now we see it in the data. For a nascent industry like data teams, we should gradually increase the amount of value created or, at a minimum, stay the same. This downward trend has been troubling me for some time … It is concerning to me that there wasn’t an increase in value creation over three years."
On the subject of remote work, the survey shows that the majority of respondents indicate that remote work has not impacted their team’s productivity. The survey also showed a slow and shallow adoption of LLMs, with only 12% of surveyed teams using LLMs for data processing, which he described as their "ideal" use case. After accounting for 24% of teams not using LLMs at all, Anderson explained that the remainder primarily use them "for code generation, ideation or copy creation, and code debugging."
Janda concluded by reflecting on his hope that data teams would become more embedded into organisations’ operational radar, rather than "seeing data as this thing on the side." He also called out the need for more cross-functional skills on data teams, saying:
I hope to see a lot more teams being cross-functional in the way I remember we removed boundaries between frontend and backend and infra teams, by creating cross-functional units.