Event processing is a method of tracking and
analyzing
Analysis (plural, : analyses) is the process of breaking a complexity, complex topic or Substance theory, substance into smaller parts in order to gain a better understanding of it. The technique has been applied in the study of mathematics a ...
(processing) streams of information (data) about things that happen (events),
and deriving a conclusion from them. Complex event processing, or CEP, consists of a set of concepts and techniques developed in the early 1990s for processing real-time events and extracting information from event streams as they arrive. The goal of complex event processing is to identify meaningful events (such as
opportunities or threats)
in real-time situations and respond to them as quickly as possible.
These events may be happening across the various layers of an organization as sales leads, orders or
customer service
Customer service is the assistance and advice provided by a company to those people who buy or use its products or services. Each industry requires different levels of customer service, but in the end, the idea of a well-performed service is that ...
calls. Or, they may be news items,
text message
Text messaging, or texting, is the act of composing and sending electronic messages, typically consisting of alphabetic and numeric characters, between two or more users of mobile devices, desktops/ laptops, or another type of compatible comput ...
s,
social media
Social media are interactive media technologies that facilitate the creation and sharing of information, ideas, interests, and other forms of expression through virtual communities and networks. While challenges to the definition of ''social medi ...
posts,
stock market feeds,
traffic report
Traffic reporting is the near real-time distribution of information about road conditions such as traffic congestion, detours, and traffic collisions. The reports help drivers anticipate and avoid traffic problems. Traffic reports, especially in ...
s,
weather report
Weather Report was an American jazz fusion band active from 1970 to 1986. The band was founded in 1970 by Austrian virtuoso keyboardist Joe Zawinul, American saxophonist Wayne Shorter, Czech bassist Miroslav Vitouš, American drummer and voca ...
s, or other kinds of data.
An event may also be defined as a "change of state," when a measurement exceeds a predefined threshold of time, temperature, or other value.
Analysts have suggested that CEP will give organizations a new way to analyze patterns in real-time and help the business side communicate better with IT and service departments. CEP has since become an enabling technology in many systems that are used to take immediate action in response to incoming streams of events. Applications are now to be found (2018) in many sectors of business including stock market trading systems,
mobile device
A mobile device (or handheld computer) is a computer small enough to hold and operate in the hand. Mobile devices typically have a flat LCD or OLED screen, a touchscreen interface, and digital or physical buttons. They may also have a physical ...
s, internet operations,
fraud detection
In law, fraud is intentional deception to secure unfair or unlawful gain, or to deprive a victim of a legal right. Fraud can violate civil law (e.g., a fraud victim may sue the fraud perpetrator to avoid the fraud or recover monetary compensa ...
, the
transportation industry
Transport (in British English), or transportation (in American English), is the intentional movement of humans, animals, and goods from one location to another. Modes of transport include air, land (rail and road), water, cable, pipeline, a ...
, and
governmental intelligence gathering.
The vast amount of information available about events is sometimes referred to as the event cloud.
Conceptual description
Among thousands of incoming events, a monitoring system may for instance receive the following three from the same source:
# church bells ringing.
# the appearance of a man in a tuxedo with a woman in a flowing white gown.
# rice flying through the air.
From these events the monitoring system may infer a ''complex event'': a wedding. CEP as a technique helps discover complex events by analyzing and correlating other events: the bells, the man and woman in wedding attire and the rice flying through the air.
CEP relies on a number of techniques, including:
* Event-
pattern detection
Pattern recognition is the automated recognition of patterns and regularities in data. It has applications in statistical data analysis, signal processing, image analysis, information retrieval, bioinformatics, data compression, computer graphics ...
* Event
abstraction
Abstraction in its main sense is a conceptual process wherein general rules and concepts are derived from the usage and classification of specific examples, literal ("real" or "concrete") signifiers, first principles, or other methods.
"An abstr ...
* Event filtering
* Event aggregation and transformation
* Modeling event hierarchies
* Detecting relationships (such as
causality
Causality (also referred to as causation, or cause and effect) is influence by which one event, process, state, or object (''a'' ''cause'') contributes to the production of another event, process, state, or object (an ''effect'') where the cau ...
,
membership
Member may refer to:
* Military jury, referred to as "Members" in military jargon
* Element (mathematics), an object that belongs to a mathematical set
* In object-oriented programming, a member of a class
** Field (computer science), entries in ...
or
timing) between events
* Abstracting
event-driven
Event driven may refer to:
The term event-driven refers to a methodology that focuses on events and event dependencies.
Examples include
* Event-driven finite-state machine, finite-state machine where the transition from one state to another ...
processes
Commercial applications of CEP exist in variety of industries and include the detection of
credit-card fraud,
business activity monitoring
Business activity monitoring (BAM) is software that aids the monitoring of business activities which are implemented in computer systems.
The term was originally coined by analysts at Gartner, Inc. and refers to the aggregation, analysis, and pr ...
, and security monitoring.
History
The CEP area has roots in
discrete event simulation
A discrete-event simulation (DES) models the operation of a system as a ( discrete) sequence of events in time. Each event occurs at a particular instant in time and marks a change of state in the system. Between consecutive events, no change in t ...
, the
active database
An active database is a database that includes an event-driven architecture (often in the form of ECA rules) which can respond to conditions both inside and outside the database. Possible uses include security monitoring, alerting, statistics gat ...
area and some programming languages. The activity in the industry was preceded by a wave of research projects in the 1990s. According to the first project that paved the way to a generic CEP language and execution model was the Rapide project in
Stanford University
Stanford University, officially Leland Stanford Junior University, is a private research university in Stanford, California. The campus occupies , among the largest in the United States, and enrolls over 17,000 students. Stanford is consider ...
, directed by
David Luckham. In parallel there have been two other research projects: Infospheres in
California Institute of Technology
The California Institute of Technology (branded as Caltech or CIT)The university itself only spells its short form as "Caltech"; the institution considers other spellings such a"Cal Tech" and "CalTech" incorrect. The institute is also occasional ...
, directed by
K. Mani Chandy, and
Apama
Apama ( grc, Ἀπάμα, Apáma), sometimes known as Apama I or Apame I, was a Sogdian noblewoman and the wife of the first ruler of the Seleucid Empire, Seleucus I Nicator. They married at Susa in 324 BC. According to Arrian, Apama was the da ...
in
University of Cambridge
, mottoeng = Literal: From here, light and sacred draughts.
Non literal: From this place, we gain enlightenment and precious knowledge.
, established =
, other_name = The Chancellor, Masters and Schola ...
directed by John Bates. The commercial products were dependents of the concepts developed in these and some later research projects. Community efforts started in a series of event processing symposiums organized by the
Event Processing Technical Society
Event Processing Technical Society (EPTS) is an inclusive group of organizations and individuals aiming to increase awareness of event processing, foster topics for future standardization, and establish event processing as a separate academic disc ...
, and later by the ACM DEBS conference series. One of the community efforts was to produce the event processing manifesto.
Related concepts
CEP is used in
operational intelligence
Operational intelligence (OI) is a category of real-time dynamic, business analytics that delivers visibility and insight into data, streaming events and business operations. OI solutions run queries against streaming data feeds and event data t ...
(OI) products to provide insight into business operations by running query analysis against live feeds and event data. OI collects real-time data and correlates against historical data to provide insight and analysis. Multiple sources of data can be combined to provide a common operating picture that uses current information.
In
network management
Network management is the process of administering and managing computer networks. Services provided by this discipline include fault analysis, performance management, provisioning of networks and maintaining quality of service. Network managem ...
,
systems management,
application management and
service management
Service management in the manufacturing context, is integrated into supply chain management as the intersection between the actual sales and the customer point of view. The aim of high-performance service management is to optimize the service- ...
, people usually refer instead to
event correlation
Event correlation is a technique for making sense of a large number of events and pinpointing the few events that are really important in that mass of information. This is accomplished by looking for and analyzing relationships between events.
Hi ...
. As CEP engines, event correlation engines (''event correlators'') analyze a mass of events, pinpoint the most significant ones, and trigger actions. However, most of them do not produce new inferred events. Instead, they relate high-level events with low-level events.
Inference engine
In the field of artificial intelligence, an inference engine is a component of the system that applies logical rules to the knowledge base to deduce new information. The first inference engines were components of expert systems. The typical expert ...
s, e.g.,
rule-based reasoning engines, typically produce inferred information in
artificial intelligence
Artificial intelligence (AI) is intelligence—perceiving, synthesizing, and inferring information—demonstrated by machines, as opposed to intelligence displayed by animals and humans. Example tasks in which this is done include speech re ...
. However, they do not usually produce new information in the form of complex (i.e., inferred) events.
Example
A more systemic example of CEP involves a car, some sensors and various events and reactions. Imagine that a car has several sensors—one that measures tire pressure, one that measures speed, and one that detects if someone sits on a seat or leaves a seat.
In the first situation, the car is moving and the pressure of one of the tires moves from 45
psi
Psi, PSI or Ψ may refer to:
Alphabetic letters
* Psi (Greek) (Ψ, ψ), the 23rd letter of the Greek alphabet
* Psi (Cyrillic) (Ѱ, ѱ), letter of the early Cyrillic alphabet, adopted from Greek
Arts and entertainment
* "Psi" as an abbreviatio ...
to 41 psi over 15 minutes. As the pressure in the tire is decreasing, a series of events containing the tire pressure is generated. In addition, a series of events containing the speed of the car is generated. The car's Event Processor may detect a situation whereby a loss of tire pressure over a relatively long period of time results in the creation of the "lossOfTirePressure" event. This new event may trigger a reaction process to note the pressure loss into the car's maintenance log, and alert the driver via the car's portal that the tire pressure has reduced.
In the second situation, the car is moving and the pressure of one of the tires drops from 45 psi to 20 psi in 5 seconds. A different situation is detected—perhaps because the loss of pressure occurred over a shorter period of time, or perhaps because the difference in values between each event were larger than a predefined limit. The different situation results in a new event "blowOutTire" being generated. This new event triggers a different reaction process to immediately alert the driver and to initiate onboard computer routines to assist the driver in bringing the car to a stop without losing control through skidding.
In addition, events that represent detected situations can also be combined with other events in order to detect more complex situations. For example, in the final situation the car is moving normally and suffers a blown tire which results in the car leaving the road and striking a tree, and the driver is thrown from the car. A series of different situations are rapidly detected. The combination of "blowOutTire", "zeroSpeed" and "driverLeftSeat" within a very short period of time results in a new situation being detected: "occupantThrownAccident". Even though there is no direct measurement that can determine conclusively that the driver was thrown, or that there was an accident, the combination of events allows the situation to be detected and a new event to be created to signify the detected situation. This is the essence of a complex (or composite) event. It is complex because one cannot directly detect the situation; one has to infer or deduce that the situation has occurred from a combination of other events.
Integration with business process management
A natural fit for CEP has been with
business process management
Business process management (BPM) is the discipline in which people use various methods to discover, model, analyze, measure, improve, optimize, and automate business processes. Any combination of methods used to manage a company's business pro ...
(BPM). BPM focuses on end-to-end business processes, in order to continuously optimize and align for its operational environment.
However, the optimization of a business does not rely solely upon its individual, end-to-end processes. Seemingly disparate processes can affect each other significantly. Consider this scenario:
In the aerospace industry, it is good practice to monitor breakdowns of vehicles to look for trends (determine potential weaknesses in manufacturing processes, material, etc.). Another separate process monitors current operational vehicles' life cycles and decommissions them when appropriate. One use for CEP is to link these separate processes, so that in the case of the initial process (breakdown monitoring) discovering a malfunction based on metal fatigue (a significant event), an action can be created to exploit the second process (life cycle) to issue a recall on vehicles using the same batch of metal discovered as faulty in the initial process.
The integration of CEP and BPM must exist at two levels, both at the business awareness level (users must understand the potential holistic benefits of their individual processes) and also at the technological level (there needs to be a method by which CEP can interact with BPM implementation). For a recent state of the art review on the integration of CEP with BPM, which is frequently labeled as Event-Driven Business Process Management, refer to.
Computation-oriented CEP's role can arguably be seen to overlap with Business Rule technology.
For example, customer service centers are using CEP for click-stream analysis and customer experience management. CEP software can factor real-time information about millions of events (clicks or other interactions) per second into
business intelligence
Business intelligence (BI) comprises the strategies and technologies used by enterprises for the data analysis and management of business information. Common functions of business intelligence technologies include reporting, online analytical pr ...
and other
decision-support applications. These "
recommendation applications" help agents provide personalized service based on each customer's experience. The CEP application may collect data about what customers on the phone are currently doing, or how they have recently interacted with the company in other various channels, including in-branch, or on the Web via self-service features, instant messaging and email. The application then analyzes the total customer experience and recommends scripts or next steps that guide the agent on the phone, and hopefully keep the customer happy.
Integration with time series databases
A
time series database
A time series database (TSDB) is a software system that is optimized for storing and serving time series through associated pairs of time(s) and value(s). In some fields, ''time series'' may be called profiles, curves, traces or trends. Several ea ...
is a software system that is optimized for the handling of data organized by time. Time series are finite or infinite sequences of data items, where each item has an associated timestamp and the sequence of timestamps is non-decreasing. Elements of a time series are often called ticks. The timestamps are not required to be ascending (merely non-decreasing) because in practice the time resolution of some systems such as financial data sources can be quite low (milliseconds, microseconds or even nanoseconds), so consecutive events may carry equal timestamps.
Time series data provides a historical context to the analysis typically associated with complex event processing. This can apply to any vertical industry such as finance and cooperatively with other technologies such as BPM.
The ideal case for CEP analysis is to view historical time series and real-time streaming data as a single time continuum. What happened yesterday, last week or last month is simply an extension of what is occurring today and what may occur in the future. An example may involve comparing current market volumes to historic volumes, prices and volatility for trade execution logic. Or the need to act upon live market prices may involve comparisons to benchmarks that include sector and index movements, whose intra-day and historic trends gauge volatility and smooth outliers.
Internet of Things and Smart Cyber-physical systems
Complex event processing is a key enabler in
Internet of Things
The Internet of things (IoT) describes physical objects (or groups of such objects) with sensors, processing ability, software and other technologies that connect and exchange data with other devices and systems over the Internet or other comm ...
(IoT) settings and Smart
Cyber-physical system
A cyber-physical system (CPS) or intelligent system is a computer system in which a Mechanism (engineering), mechanism is controlled or monitored by computer-based algorithms. In cyber-physical systems, physical and software components are deeply ...
s (CPS) as well. Processing dense and heterogeneous streams from various sensors and matching patterns against those streams is a typical task in such cases. The majority of these techniques rely on the fact that representing the IoT system's state and its changes is more efficient in the form of a data stream, instead of having a static, materialized model. Reasoning over such stream-based models fundamentally differs from traditional reasoning techniques and typically require the combination of
model transformation
A model transformation, in model-driven engineering, is an automated way of modifying and creating models. An example use of model transformation is ensuring that a family of Computer model, models is consistent, in a precise sense which the softw ...
s and CEP.
See also
*
Event correlation
Event correlation is a technique for making sense of a large number of events and pinpointing the few events that are really important in that mass of information. This is accomplished by looking for and analyzing relationships between events.
Hi ...
*
Event-driven architecture Event-driven architecture (EDA) is a software architecture paradigm promoting the production, detection, consumption of, and reaction to events.
Overview
An ''event'' can be defined as "a significant change in state". For example, when a consumer p ...
— (EDA) is a software architecture pattern promoting the production, detection, consumption of, and reaction to events.
*
SEDA
Seda or SEDA may refer to:
Acronyms
* Safe and Effective Drug Act, a bill proposed in the United States House of Representatives in 2004
* Seeing Eye Dogs Australia, an Australian organisation
* Staff and Educational Development Association, a p ...
- Staged event-driven architecture decomposes complex, event-driven architectures into stages
*
Event Processing Technical Society
Event Processing Technical Society (EPTS) is an inclusive group of organizations and individuals aiming to increase awareness of event processing, foster topics for future standardization, and establish event processing as a separate academic disc ...
— (EPTS) is an event processing community of interest
*
Event stream processing
In computer science, stream processing (also known as event stream processing, data stream processing, or distributed stream processing) is a programming paradigm which views data streams, or sequences of events in time, as the central input and o ...
— (ESP) is a related technology that focuses on processing streams of related data.
*
Kinetic Rule Language
Kinetic Rule Language (KRL) is a rule-based programming language for creating applications on the Live Web. KRL programs, or rulesets, comprise a number of rules that respond to particular events. KRL has been promoted as language for building p ...
— (KRL) is an event-condition-action rule language with an embedded complex event expression language.
*
Operational intelligence
Operational intelligence (OI) is a category of real-time dynamic, business analytics that delivers visibility and insight into data, streaming events and business operations. OI solutions run queries against streaming data feeds and event data t ...
— Both CEP and ESP are technologies that underpin operational intelligence.
*
Pattern matching
In computer science, pattern matching is the act of checking a given sequence of tokens for the presence of the constituents of some pattern. In contrast to pattern recognition, the match usually has to be exact: "either it will or will not be ...
*
Real-time business intelligence Real-time business intelligence (RTBI) is a concept describing the process of delivering business intelligence (BI) or information about business operations as they occur. Real time means near to zero latency and access to information whenever it ...
— Business Intelligence is the application of knowledge derived from CEP systems
*
Real-time computing
Real-time computing (RTC) is the computer science term for hardware and software systems subject to a "real-time constraint", for example from event to system response. Real-time programs must guarantee response within specified time constrai ...
— CEP systems are typically real-time systems
*
Real time enterprise Real-time enterprise (commonly abbreviated to RTE) is a concept in business systems design focused on ensuring organisational responsiveness that was popularised in the first decade of the 21st century. It is also referred to as on-demand enterprise ...
Vendors and products
*
Apama
Apama ( grc, Ἀπάμα, Apáma), sometimes known as Apama I or Apame I, was a Sogdian noblewoman and the wife of the first ruler of the Seleucid Empire, Seleucus I Nicator. They married at Susa in 324 BC. According to Arrian, Apama was the da ...
by
Software AG
Founded in 1969, Software AG is an enterprise software company with over 10,000 enterprise customers in over 70 countries. The company is the second largest software vendor in Germany, and the seventh largest in Europe. Software AG is traded on t ...
- monitors rapidly moving event streams, detects and analyzes important patterns, and takes action according to rules.
*
Azure Stream Analytics
Microsoft Azure Stream Analytics is a serverless scalable complex event processing engine by Microsoft that enables users to develop and run real-time analytics on multiple streams of data from sources such as devices, sensors, web sites, social m ...
*
Drools Fusion
EVAM Streaming Analytics*
Esper Complex event processing for Java and C# (GPLv2).
*
Feedzai - Pulse
*
Microsoft StreamInsight Microsoft CEP Engine implementation
*
openPDC — A set of applications for processing streaming time-series data in real-time.
*
Oracle Event Processing - for building applications to filter, correlate, and process events in real time.
*
SAP ESP - A low-latency, rapid development and deployment platform that allows processing multiple streams of data in real time
*
SQLstream
SQLstream is a distributed, SQL standards-compliant plus Java stream processing platform. SQLstream, Inc. is based in San Francisco, California and was launched in 2009 by Damian Black, Edan Kabatchnik and Julian Hyde, author of the open source Mo ...
SQLstream's stream processing platform, s-Server, provides a relational stream computing platform for analyzing large volumes of service, sensor and machine and log file data in real-time.
*
TIBCO BusinessEvents & Streambase - CEP platform and High Performance Low Latency Event Stream Processing
*
WebSphere Business Events
*
Apache Flink
Apache Flink is an open-source, unified stream-processing and batch-processing framework developed by the Apache Software Foundation. The core of Apache Flink is a distributed streaming data-flow engine written in Java and Scala. Flink exec ...
Open-source distributed stream processing framework with a CEP API
for Java and Scala.
*
Apache Storm
Apache Storm is a distributed stream processing computation framework written predominantly in the Clojure programming language. Originally created by Nathan Marz and team at BackType, the project was open sourced after being acquired by Twitter. ...
Free and open source distributed realtime computation system. Storm processes unbounded streams of data in realtime.
References
{{reflist
Information systems
Cognition
Events (computing)
Process (computing)