InfoQ Homepage S3 Content on InfoQ
-
Reddit Unveils REV2: Modernised Rule-Execution with Kubernetes, Kafka, and Flink Stateful Functions
Reddit's Safety Engineering team recently published how it modernised its Rule-Execution system, which detects and acts on policy-violating content in real time. The new architecture includes improvements like transitioning from legacy EC2-based systems to Kubernetes, better rule version control with Github and S3 storage, and the capability to scale more efficiently with Flink Stateful Functions.
-
Cloudflare Sippy: Incrementally Migrate Data from Amazon S3 to Reduce Egress Fees
Cloudflare recently announced the open beta of Sippy, an incremental data migration service that copies data from Amazon S3 to Cloudflare R2 only the first time the data is requested. Sippy is designed to minimize migration-specific egress fees by leveraging requests within existing application flows while simultaneously copying objects to R2.
-
Mountpoint for Amazon S3 Now GA to Access Bucket Like Local File System
During the latest AWS Storage Day event, Amazon announced the general availability of Mountpoint for Amazon S3. The new open-source file client provides through a file interface the elastic storage and throughput of Amazon S3, supporting data transfer at up to 100 Gb/second between each EC2 instance and the object storage.
-
Running Apache Flink Applications on AWS KDA: Lessons Learnt at Deliveroo
Deliveroo introduced Apache Flink into its technology stack for enriching and merging events consumed from Apache Kafka or Kinesis Streams. The company opted to use AWS Kinesis Data Analytics (KDA) service to manage Apache Flink clusters on AWS and shared its experiences from running Flink applications on KDA.
-
Inside InfluxDB 3.0: Exploring InfluxDB’s Scalable and Decoupled Architecture
InfluxData recently unveiled the system architecture for InfluxDB 3.0, its newest time-series DB. Its architecture encompasses four major components responsible for data ingestion, querying, compaction, and garbage collection and includes two main storage types. The architecture caters to operating the DB on-premise and natively on major cloud providers.
-
Amazon Introduces AWS HealthImaging to Store and Analyze Medical Imaging Data
At the recent AWS Summit in New York, Amazon announced AWS HealthImaging. The new HIPAA-eligible service helps healthcare providers to store, analyze, and share medical imaging data at scale.
-
Pfizer Uses Serverless Architecture on AWS to Scale Processing of Digital Biomarkers
Pfizer upgraded the serverless architecture for processing digital biomarker data at scale to make it more flexible and configurable. They created a framework that uses a file processing pipeline built with AWS Step Functions and other serverless services, as well as a custom Python package for data ingestion and processing.
-
AWS Launches Amazon S3 Dual-Layer Server-Side Encryption with Keys Stored in AWS KMS
Recently AWS launched Amazon S3 dual-layer server-side encryption with keys stored in AWS Key Management Service (DSSE-KMS), a new encryption option in Amazon S3 that applies two layers of encryption to objects when they are uploaded to an Amazon Simple Storage Service (Amazon S3) bucket.
-
Datadog Creates Scalable Data Ingestion Architecture
Datadog created a dedicated data ingestion architecture offering exactly-once semantics for their third-generation event store, Husky. The event-driven architecture (EDA) can accommodate bursts in traffic in the multi-tenant platform with reasonable ingestion latency and acceptable operational costs.
-
AWS Introduces Athena Provisioned Capacity
AWS recently announced a new feature Provisioned Capacity for Athena, that allows users to run SQL queries on fully-managed compute capacity for a fixed price and no long-term commitments.
-
AWS Announces Open Source Mountpoint for Amazon S3
During the latest Pi Day, AWS announced Mountpoint for Amazon S3, an open-source file client to deliver high throughput access on Amazon S3. Currently in alpha, the local mount point provides high single-instance transfer rates and is primarily intended for data lake applications.
-
Amazon S3 Encrypts All New Objects with AES-256
Since January 5th, Amazon S3 encrypts all new objects by default with AES-256 to protect data at rest. S3 automatically applies server-side encryption using Amazon S3-managed keys for each new object, unless a different encryption option is specified.
-
AWS Announces Upcoming Security Changes in April 2023 for Amazon S3
Recently AWS announced it would make two changes to Amazon Simple Storage Service (Amazon S3): all buckets in a region will have S3 Block Public Access enabled and access control lists (ACLs) disabled by default. These changes will take effect in April 2023 and will be rolled out by the company in all AWS regions within weeks.
-
Recap of AWS re:Invent 2022
After a virtual-only event in 2020 and a reduced-size 2021 edition, re:Invent was back last week in Las Vegas with over 50,000 attendees for the 11th edition. During multiple sessions and keynotes at the largest AWS yearly conference, the cloud provider announced new services and features, with the focus more on business solutions and data options than new building blocks.
-
Cloudflare Introduces Super Slurper Feature for Its R2 Object Storage in Private Beta
Cloudflare recently announced the R2 Super Sluper feature, which enables developers to move all their data to R2 in one giant slurp or sip by sip through an intuitive UI and API. The feature is in private beta by signing up.