AWS recently announced the preview of OpenSearch Serverless, a new option of OpenSearch service that automatically provisions and scales the resources for data ingestion and query responses. The minimum capacity required for the serverless option raised some concerns in the community.
Allowing developers to run large-scale search and analytics workloads without managing clusters, OpenSearch Serverless supports the latest versions of OpenSearch and previous versions of Elasticsearch, as well as OpenSearch Dashboards and Kibana. Channy Yun, principal developer advocate at AWS, explains:
Before the launch of OpenSearch Serverless, you created a managed cluster, specifying instance types, counts, and storage options, and then managed the lifecycle and shard strategy for indices within that cluster. With OpenSearch Serverless, you create a Collection, which manages a group of indices that work together to support a specific workload. You no longer need to specify the hardware or manage the indices directly.
Collections can either be time series, the log analytics segment that focuses on analyzing large volumes of semistructured data in real-time, or search, a full-text search that powers applications in internal networks and internet-facing applications.
Source: https://aws.amazon.com/opensearch-service/features/serverless/
Yun adds:
In the case of time-series data, you can ingest data with all of the streaming ingestion options, such as native OpenSearch streaming APIs, Amazon Kinesis Data Firehose, AWS Glue, and a wide range of open-source streaming ingestion pipelines like Logstash, FluentBit, Fluentd, and Data Prepper.
Search and indexing work independently: when a workload receives a sudden burst of logging activities, OpenSearch Serverless instantly scales the resources to ingest and store the data without impacting query response times.
Source: https://aws.amazon.com/blogs/aws/preview-amazon-opensearch-serverless-run-search-and-analytics-workloads-without-managing-clusters/
According to the cloud provider, the new service helps customers manage variable workloads, meet sensitive service-level agreements and create development or test environments. During his keynote at re:Invent, Adam Selipsky highlighted that OpenSearch was until today the exception, as other AWS analytics services, including QuickSight, Glue for data integration, EMR, MSK, and Redshift, had already serverless options.
The compute capacity is measured in OpenSearch compute units (OCUs), a unit corresponding to a fixed amount of CPU, memory, EBS storage, and I/O resources required to index data or execute queries. With a minimum of 4 OCUs for the first collection in an AWS account, approximately 700 USD monthly, the pricing model left many developers and serverless advocates unhappy. Corey Quinn, chief cloud economist at The Duckbill Group, questions:
I was wondering whether OpenSearch Serverless was real serverless or fake serverless. With this model, AWS should just go all in and call EC2 "Serverless" at this point.
Ben Kehoe, AWS Serverless Hero, writes:
"Serverless" was always destined to become as meaningless as "cloud", and it's always been a spectrum anyway. Don't argue about what is or isn't serverless, it's to talk about what the desired *benefits* of serverless are, and how much a given technology provides them.
Carl Meadows, product manager at AWS, tweets:
This is preview so there is lots of room for us to innovate. Scale-to-zero, smaller cheaper OCUs for lower entry, etc. But also note that OCUs can be shared across collections lowering cost with multiple smaller workloads.
OpenSearch Serverless supports a subset of OpenSearch plugins and indexing, search, and metadata API operations available in OpenSearch. The preview is available in a few AWS regions, including Northern Virginia and Ireland.