InfoQ Homepage Orchestration Content on InfoQ
-
QCon London: Mastering Long-Running Processes in Modern Architectures
At QCon London 2024, Bernd Ruecker recommended implementing long-running tasks asynchronously with a process-orchestration platform. Such a platform provides better service boundaries and efficiencies and reduces accidental system complexity and risk. Organizing the platform centrally in an organization eases orchestration adoption by applications.
-
Netflix Uses Metaflow to Manage Hundreds of AI/ML Applications at Scale
Netflix recently published how its Machine Learning Platform (MLP) team provides an ecosystem around Metaflow, an open-source machine learning infrastructure framework. By creating various integrations for Metaflow, Netflix already has hundreds of Metaflow projects maintained by multiple engineering teams.
-
Canonical Launches Charmed MLFlow to Simplify Management and Maintenance of ML Workflows
Based on the open-source MLflow platform, Canonical Charmed MLFlow aims to simplify the task of managing machine learning workflows and artifacts by using alternative packaging system and orchestration engine.
-
Azure Durable Functions Now Supports Storage Backends Microsoft Netherite and MSSQL
Microsoft recently announced that Azure Durable Functions support for the new storage providers, Netherite and Microsoft SQL Server (MSSQL), is generally available.
-
AWS Introduces Step Functions Distributed Map for Large-Scale Parallel Data Processing
AWS recently announced a distributed map for Step Functions, a solution for large-scale parallel data processing. Optimized for S3, the new feature of the AWS orchestration service targets interactive and highly parallel serverless data processing workflows.
-
Kestra: a Scalable Open-Source Orchestration and Scheduling Platform
Kestra, a new open-source orchestration and scheduling platform, helps developers to build, run, schedule, and monitor complex pipelines. The concept of a workflow, called Flow in Kestra, is at the heart of the platform. It is a list of tasks defined with a descriptive language based on yaml.
-
AWS Releases Multi-Cloud Kubernetes Autoscaler Karpenter
AWS recently released Karpenter, their open-source Kubernetes cluster autoscaler. This improves upon their Kubernetes Cluster Autoscaler by providing a easily configurable, fully automated scheduler. Karpenter is able to monitor for unscheduled pods and launch new nodes as well as terminate unneeded infrastructure. Karpenter is designed to work with any Kubernetes cluster in any environment.
-
Karmada 0.7: Next-Gen Multi-Cloud and Multi-Cluster Kubernetes Orchestration
Karmada (Kubernetes Armada) 0.7, featuring a promising Kubernetes management system in the hybrid cloud era, became available on July 12, 2021. It brought multi-cluster service discovery, precise cluster status management, replica scheduling based on cluster resources, and more convenient APIs to divide replicas by weight list.
-
Gremlin Aims to Reduce Kubernetes Noisy Neighbours through Chaos Engineering
Gremlin has released enhancements to its Chaos Engineering platform aimed at DevOps engineers interested in future-proofing Kubernetes clusters by isolating "noisy neighbours". On Kubernetes, the noisy neighbour issue occurs when multiple applications sharing a Kubernetes cluster compete for resources leading to degraded performance.
-
PayPal Standardizes on Apache Airflow and Apache Gobblin for Its Next-Gen Data Movement Platform
PayPal recently described how it standardized on Apache Airflow and Apache Gobblin for implementing its next-gen data movement platform. In a recent blog post, PayPal engineers detail how the existing data movement platform evolved into many tools & platforms in a complex and unmanageable ecosystem and their shift towards a new implementation.
-
Google Cloud Workflows Now Generally Available
Google has recently announced that Google Cloud Workflows, the service to orchestrate Google Cloud and HTTP-based API services with serverless workflows, is now generally available. Workflows Connectors are now in public preview.
-
HashiCorp Announces 1.0 Beta of Distributed Orchestrator Nomad
HashiCorp announced version 1.0 Beta of Nomad - their orchestration framework for deploying and managing containerized and non-containerized applications.
-
Puppet Relay Provides Event-Driven Automation for Common Workflows
Puppet has announced the beta release of Relay, their event driven automation platform. Relay allows for automating processes as code. There are a number of pre-written workflows and there is support for creating additional workflows. Relay has a number of integrations available for use in workflows including AWS, Kubernetes, Azure, PagerDuty, and GitHub.
-
Rook v1.0 Adds Support for Ceph Nautilus, EdgeFS, and NFS Operator
Rook, a storage orchestrator for Kubernetes, has released version 1.0 for production-ready workloads that use file, block, and object storage in containers. Highlights of Rook 1.0 include support for storage providers through operators like Ceph Nautilus, EdgeFS, and NFS. For instance, when a pod requests an NFS file system, Rook can provision it without any manual intervention.
-
GitHub Incident Analysis Shows How to Improve Service Reliability
On October 21, 2018, GitHub users experienced a degraded service during 24 hours due to an incident caused by routine maintenance work. This led to the display of outdated and inconsistent information and to the unavailability of webhooks and other internal services for 24 hours. GitHub post-incident report shows where things failed and suggests how to improve site reliability.