Azure Data Explorer is a fully-managed

big data Though used sometimes loosely partly because of a lack of formal definition, the interpretation that seems to best describe Big data is the one associated with large body of information that we could not comprehend when used only in smaller am ...

analytics cloud platform and data-exploration service, developed by Microsoft, that ingests structured, semi-structured (like

JSON JSON (JavaScript Object Notation, pronounced ; also ) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other ser ...

) and unstructured data (like free-text). The service then stores this data and answers analytic ad hoc queries on it with seconds of latency. It is a full text indexing and retrieval database, including time series analysis capabilities and regular expression evaluation and text parsing. It is offered as Platform as a Service (PaaS) as part of

Microsoft Azure Microsoft Azure, often referred to as Azure ( , ), is a cloud computing platform operated by Microsoft for application management via around the world-distributed data centers. Microsoft Azure has multiple capabilities such as software as a ...

platform. The product was announced by Microsoft in 2018.

History

The development of the product began in 2014 as a grassroots incubation project in the Israeli R&D center of Microsoft, with the internal code name 'Kusto' (named after

Jacques Cousteau Jacques-Yves Cousteau, (, also , ; 11 June 191025 June 1997) was a French naval officer, oceanographer, filmmaker and author. He co-invented the first successful Aqua-Lung, open-circuit SCUBA (self-contained underwater breathing apparatus). T ...

, as a reference to "exploring the ocean of data"). The project aim was to address Azure services' needs for fast and scalable log and telemetry analytics. In 2016 it became the backend big-data and analytics service for Application Insights Analytics. The product was announced as a Public Preview product at the Microsoft Ignite 2018 conference, and was announced as a generally available at the Microsoft Ignite conference of February 2019. In March 2021, "Kusto EngineV3", Azure Data Explorer's next generation storage and query engine, became generally available. It was designed to provide unparalleled performance for ingesting and querying telemetry, logs, and time series data.

Features

Azure Data Explorer offers an optimized query language and visualizing options of its data with a SQL-like language called

KQL KQL was a radio station, located in Los Angeles, California, that was licensed to Arno A. Kluge from October 13, 1921 to June 9, 1922. This was the first broadcasting station licensed in the state of California, and one of the first in the United S ...

(Kusto Query Language.). Azure Data Explorer can ingest 200 MB per second per node. Data Ingestion methods are pipelines and connectors to common services like Azure Event Grid or Azure Event Hub, or programmatic ingestion using SDKs. Data visualization can be achieved using thei
native dashboard offering
or with tools like

PowerBI Power BI is an interactive data visualization software product developed by Microsoft with a primary focus on business intelligence. It is part of the Microsoft Power Platform. Power BI is a collection of software services, apps, and connectors ...

or Grafana.

Design

Azure Data Explorer is a distributed database running on a cluster of compute nodes in

. It is based on relational database management systems (RDBMS), supporting entities such as databases, tables, functions, and columns. It supports complex analytics query operators, such as calculated columns, searching and filtering on rows, group by-aggregates and joins. The engine service exposes a

relational data model The relational model (RM) is an approach to managing data using a structure and language consistent with first-order predicate logic, first described in 1969 by English computer scientist Edgar F. Codd, where all data is represented in terms of tu ...

: At the top level (

cluster may refer to: Science and technology Astronomy * Cluster (spacecraft), constellation of four European Space Agency spacecraft * Asteroid cluster, a small asteroid family * Cluster II (spacecraft), a European Space Agency mission to study th ...

) there is a collection of databases, each database contains a collection of tables and stored functions. Each table defines a schema (ordered list of typed fields). In Azure Data Explorer, unlike a typical relational database management systems (RDBMS), there are no constraints like key uniqueness,

primary Primary or primaries may refer to: Arts, entertainment, and media Music Groups and labels * Primary (band), from Australia * Primary (musician), hip hop musician and record producer from South Korea * Primary Music, Israeli record label Works * ...

and

foreign key A foreign key is a set of attributes in a table that refers to the primary key of another table. The foreign key links these two tables. Another way to put it: In the context of relational databases, a foreign key is a set of attributes subject to ...

. The necessary relationships are established at the query time. The data in Azure Data Explorer generally follows this pattern: Creating Database, Ingesting data, Query the database.

References

{{Microsoft Azure Services Platform Database management systems Lists of database management software