Apache Kylin is an

open source Open source is source code that is made freely available for possible modification and redistribution. Products include permission to use the source code, design documents, or content of the product. The open-source model is a decentralized sof ...

distributed analytics engine designed to provide a SQL interface and multi-dimensional analysis (OLAP) on

Hadoop Apache Hadoop () is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage an ...

and

Alluxio Alluxio is an open-source virtual distributed file system (VDFS). Initially as research project "Tachyon", Alluxio was created at the University of California, Berkeley's AMPLab as Haoyuan Li's Ph.D. Thesis, advised by Professor Scott Shenker & ...

supporting extremely large datasets. It was originally developed by

eBay eBay Inc. ( ) is an American multinational e-commerce company based in San Jose, California, that facilitates consumer-to-consumer and business-to-consumer sales through its website. eBay was founded by Pierre Omidyar in 1995 and became a ...

, and is now a project of the

Apache Software Foundation The Apache Software Foundation (ASF) is an American nonprofit corporation (classified as a 501(c)(3) organization in the United States) to support a number of open source software projects. The ASF was formed from a group of developers of the A ...

.Apache Software Foundation.
"The Apache Software Foundation Announces Apache Kylin as a Top-Level Project"
8 December 2015

History

The Kylin project was started in 2013, in eBay's R&D in Shanghai, China. In Oct 2014, Kylin v0.6 was open sourced on github.com with the name "KylinOLAP". In November 2014, Kylin joined Apache Software Foundation incubator. In December 2015, Apache Kylin graduated to be a Top Level Project. In March 2016, Kyligence, Inc. was founded by the creators of Apache Kylin. Kyligence provides a commercial analytics platform based on Apache Kylin for on-premise and cloud-based datasets.

Architecture

Apache Kylin is built on top of Apache Hadoop, Apache Hive, Apache HBase, Apache Parquet, Apache Calcite, Apache Spark and other technologies. These technologies enable Kylin to easily scale to support massive data loads. Kylin has the following core components: * REST Server: Receive and response user or API requests * Metadata: Persistent and manage system, especially the cube metadata; * Query Engine: Parse SQL queries to execution plan, and then talk with storage engine; * Storage Engine: Pushdown and scan underlying cube storage (default in HBase); * Job Engine: Generate and execute MapReduce or Spark job to build source data into cube;

Users

Apache Kylin has been adopted by many companies as their OLAP platform in production. Typical users includes eBay, Meituan, XiaoMi, NetEase, Beike, Yahoo! Japan.

Roadmap

Apache Kylin roadmap (from Kylin website): * Hadoop 3.0 support (Erasure Coding) - completed (v2.5) * Fully on Spark Cube engine - completed (v2.5) * Connect more data sources (MySQL, Oracle, SparkSQL, etc) - completed (v2.6) * Real-time analytics with Lambda Architecture - completed (v3.0) * Cloud-native storage (Parquet) - In progress (v4.0.0-alpha) * Ad hoc queries without Cubing

References

Kylin Free software Java (programming language) Relational database management systems {{database-software-stub