Apache Linkis is a computation middleware that acts as a layer between upper-level applications and underlying engines, such as Apache Spark, Apache Hive and Apache Flink. It started as an Apache Incubator project in 2021 and graduated to a Top Level Project in January 2023.
Linkis facilitates functional applications such as Tableau or Jupyter to access the computation and storage engines by providing standard interfaces like REST/WS/JDBC. It also provides data source and metadata management services through the REST standard interface.
One of the key advantages of Linkis is its flexibility. It facilitates connections between upper-level applications and various engines and lessens the effects of changes to lower layers. It also has a distributed microservice architecture, which makes it easy to scale and add new features.
Linkis also provides powerful task/request governance capabilities. It offers a basic orchestration framework to enable different strategies to manage computing tasks and allows for easy development of functional applications by reducing the amount of work required for connectivity and scalability. It is able to provide fine-grained routing, load balance, multi-tenancy, traffic control, resource control, and orchestration strategies like dual-active, active-standby, etc. It also provides a way to manage and control access to the underlying engines, including identity verification, risk prevention, and record keeping.
Every task is divided into four stages: submission stage, the task is submitted to the Entrance service; preparation stage, the task is scheduled; execution stage, the task is submitted to the underlying engine for execution and result return stage where it returns a result to the caller.
Currently, Linkis supports various computation and storage engines such as Spark, Hive, Flink, Python, Presto, ElasticSearch, JDBC etc. and will support more in the future like Trino and SeaTunnel. Linkis also supports various scripting languages such as SparkSQL, HiveQL, Python, Pyspark, Scala and JDBC.
Furthermore, Linkis has powerful resource management capabilities. Its ResourceManager can manage resources for Yarn and Linkis EngineManager, and provide label-based multi-level resource allocation and recycling. This allows for powerful resource management capabilities across multiple Yarn clusters and multiple computation resource types.
In summary, Apache Linkis is a powerful computation middleware that provides a layer of abstraction between upper-level applications and underlying engines. It facilitates easy connection and access to different engines and resource management. Linkis is used in diverse industries, from finance, banking, telecommunication, and internet companies.
