CSIRO has developed Serverless Beacon (sBeacon), a serverless architecture for querying genomic variant data without copying large genomic datasets into a central database. The system uses Amazon S3, AWS Lambda, Amazon Athena, and Amazon DynamoDB to provide an implementation of the Global Alliance for Genomics and Health (GA4GH) Beacon standard, supporting genomic queries across large cohorts while keeping the underlying data in its original storage.
The architecture separates genomic data from the metadata needed to search it. Variant Call Format (VCF) files remain in Amazon S3, while metadata such as disease status, age, and location is indexed separately. During onboarding, sBeacon creates indexes and Athena tables for the metadata, while ontology terms are handled through CSIRO's Ontoserver to support queries combining genomic and phenotypic information.
For variant queries, API Gateway routes requests to Lambda functions that identify relevant datasets and genomic regions. The Variant Querying Module uses bcftools and its htslib library to retrieve specific portions of VCF files through HTTP byte range requests. This allows the system to read the required genomic region without transferring the complete file. Results are aggregated and returned through the Beacon API.
Anuradha Wickramarachchi, a former solutions architect for Bioinformatics Products at CSIRO, highlighted data compression in a personal LinkedIn post:
One key consideration we have exercised carefully is having the data in the most compressed form. And querying while they are in the compressed form is greatly beneficial.
CSIRO evaluated sBeacon using chromosome 1 from the 1000 Genomes Project, comprising 2,504 samples in a 1.1 GB compressed VCF file stored in Amazon S3. The system ingested the dataset in 18 seconds for USD 0.00052, creating index files rather than copying the genomic data. A query covering 10,000 base pairs took an average of 1.52 seconds and cost USD 0.00013. CSIRO reported that query time remained broadly constant as the number of variants returned increased 100-fold, from four to 400.
The design builds on earlier sBeacon research published in Nature Biotechnology, which reported support for up to 40 million individuals and 3 billion genomic locations. The original Beacon work described a federated model that lets organizations retain control of their genomic data while exposing standardized query interfaces. The GA4GH protocol was designed to operate across distributed data sources without prescribing an underlying storage technology. The 2026 implementation extends that model with cloud services that provide elastic query execution while retaining genomic files in object storage.