On February 23rd, 2016 Spotify announced it is moving its technical infrastructure and data services from its existing owned and leased data centers to Google Cloud Platform.
Spotify is a popular music steaming service that powers 2 billion playlists, more than 20 billion hours of music for more than 75 million listeners. As the business has grown year over year, Spotify questioned whether or not they wanted to continue to run their own data centers. In previous years, the company felt the public cloud services could not meet the quality, performance and cost expectations required to make the shift to the cloud. In a recent blog post, Nicholas Harteau, VP engineering and infrastructure at Spotify, explained some of the rationale for moving to the cloud now: “Storage, compute and network services available from cloud providers are as high quality, high performance and low cost as what the traditional approach provides. This makes the move to the cloud a no-brainer for us.”
In a recent Rightscale Cloud Survey, Google Cloud Platform ranked third in Public Cloud adoption behind leader Amazon Web Services and Microsoft Azure. Even though Google is not a leader in this space, Harteau explains why Spotify ended up choosing Google: “What really tipped the scales towards Google for us, however, has been our experience with Google’s data platform and tools. Good infrastructure isn’t just about keeping things up and running, it’s about making all of our teams more efficient and more effective, and Google’s data stack does that for us in spades.”
Google, also very excited about the partnership with Spotify, revealed some specific details about how Spotify will be using their services in their own blog post. From a compute standpoint, Spotify will be relying upon high performance IOPS SSD and local SSD storage capabilities. They will also be taking advantage of auto scaling capabilities to ensure their infrastructure is able to respond to burst scenarios. One such burst scenario was last November 13th when Justin Bieber set a record for the most streams in a single day. Over 36 million streams were listened to on that day. Using a public cloud model, Spotify can now rely upon Google to provide that elasticity, instead of Spotify building out their own infrastructure to support peak loads.
Spotify will also be using Google’s networking services such as Direct Peering, Cloud VPN and Cloud Router in order to efficiently transfer petabytes of data between the two organizations.
From a data services perspective, Spotify will be moving away from Hadoop, MapReduce, Hive and some custom dashboards in favor of Google Cloud services including Google Cloud Pub/Sub, Google Cloud Dataflow, Google BigQuery and Google Dataproc. Guillaume Leygues, lead sales engineer at Google Cloud Platform, explains why Spotify is moving to these Google Services: “With BigQuery and Cloud Dataproc, data teams can run complex queries and get answers in a minute or two, rather than hours. This lets Spotify perform more frequent in-depth, interactive analysis, guiding product development, feature testing and more intelligent user-facing features.”