Recently, AWS announced customizable image support for Amazon EMR on Amazon Elastic Kubernetes Service (Amazon EKS) that allows customers to modify the Docker runtime image that runs their analytics application using Apache Spark on their EKS cluster.
At AWS re:Invent 2020, the company made Amazon EMR on EKS, a new deployment option for Amazon EMR that allows customers to automate the provisioning and management of Apache Spark on Amazon EKS, generally available. In addition, now customizable image support has been added, allowing customers to create a container containing both their application and its dependencies, based on the performance-optimized EMR Spark runtime, using their own continuous integration (CI) pipeline.
Data engineers in organizations leveraging Amazon EMR on EKS can create a base image, add their corporate standard libraries, and store it in Amazon Elastic Container Registry (Amazon ECR). Subsequently, they can customize the image to include their application-specific dependencies - the resulting immutable image can be vulnerability scanned, and deployed to test and production environment.
Using the custom image is straightforward – users can specify the custom image they want to use in their job using the parameter spark.kubernetes.container.image in the Start-Job-Run API. Moreover, they can also use the parameters spark.kubernetes.driver.container.image and spark.kubernetes.executor.container.image to specify a different image for Spark driver and executor pods.
With customizable image support, AWS fulfills the wish of customers running Apache Spark jobs using EMR and pre-built container images on EKS. Damon Cortesi, a principal developer advocate for analytics at AWS, said in a recent video on AWS Build Custom Images for Apache Spark on Kubernetes:
One of the first things that we heard from customers when we released this functionality was that they wanted to customize those container images. Whether it was different dependencies, whether it was, you know, changing some environment variables or something like that. So they wanted a way to take the base image that we provided and make some modifications on top of it. So what we have done is we have allowed for a way to create self-contained immutable container images as part of your CI process to simplify application development.
Currently, customizable images are available in all AWS Regions where Amazon EMR on EKS is available at no additional charge. And lastly, more details are available in the Amazon EMR on EKS Development Guide and the Amazon EMR on Amazon EKS detail page.