Log management (LM) comprises an approach to dealing with large volumes of
computer
A computer is a machine that can be programmed to Execution (computing), carry out sequences of arithmetic or logical operations (computation) automatically. Modern digital electronic computers can perform generic sets of operations known as C ...
-generated
log messages (also known as
audit records,
audit trail
An audit trail (also called audit log) is a security-relevant chronological record, set of records, and/or destination and source of records that provide documentary evidence of the sequence of activities that have affected at any time a specific ...
s,
event-logs, etc.).
Log management generally covers:
* Log collection
* Centralized log aggregation
* Long-term log storage and retention
*
Log rotation
In information technology, log rotation is an automated process used in system administration in which log files are compressed, moved (archived), renamed or deleted once they are too old or too big (there can be other metrics that can apply her ...
*
Log analysis
In computer log management and intelligence, log analysis (or ''system and network log analysis'') is an art and science seeking to make sense of computer-generated records (also called log or audit trail records). The process of creating such reco ...
(in real-time and in bulk after storage)
* Log search and reporting.
Overview
The primary drivers for log management implementations are concerns about
security
Security is protection from, or resilience against, potential harm (or other unwanted coercive change) caused by others, by restraining the freedom of others to act. Beneficiaries (technically referents) of security may be of persons and social ...
, system and network operations (such as
system
A system is a group of Interaction, interacting or interrelated elements that act according to a set of rules to form a unified whole. A system, surrounded and influenced by its environment (systems), environment, is described by its boundaries, ...
or
network administration
Network management is the process of administering and managing computer networks. Services provided by this discipline include fault analysis, performance management, provisioning of networks and maintaining quality of service. Network managemen ...
) and regulatory compliance. Logs are generated by nearly every computing device, and can often be directed to different locations both on a local
file system
In computing, file system or filesystem (often abbreviated to fs) is a method and data structure that the operating system uses to control how data is stored and retrieved. Without a file system, data placed in a storage medium would be one larg ...
or remote system.
Effectively analyzing large volumes of diverse logs can pose many challenges, such as:
* Volume: log data can reach hundreds of gigabytes of data per day for a large
organization
An organization or organisation (Commonwealth English; see spelling differences), is an entity—such as a company, an institution, or an association—comprising one or more people and having a particular purpose.
The word is derived from ...
. Simply collecting, centralizing and storing data at this volume can be challenging.
* Normalization: logs are produced in multiple formats. The process of
normalization is designed to provide a common output for analysis from diverse sources.
* Velocity: The speed at which logs are produced from devices can make collection and aggregation difficult
* Veracity: Log events may not be accurate. This is especially problematic for systems that perform detection, such as
intrusion detection systems
An intrusion detection system (IDS; also intrusion prevention system or IPS) is a device or software application that monitors a network or systems for malicious activity or policy violations. Any intrusion activity or violation is typically rep ...
.
Users and potential users of log management may purchase complete commercial tools or build their own log-management and intelligence tools, assembling the functionality from various
open-source
Open source is source code that is made freely available for possible modification and redistribution. Products include permission to use the source code, design documents, or content of the product. The open-source model is a decentralized sof ...
components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it.
Logging can produce technical information usable for the maintenance of applications or websites. It can serve:
* to define whether a reported bug is actually a bug
* to help analyze, reproduce and solve bugs
* to help test new features in a development stage
Terminology
Suggestions were made to change the definition of logging. This change would keep matters both purer and more easily maintainable:
* Logging would then be defined as all instantly discardable data on the technical process of an application or website, as it represents and processes data and user input.
* Auditing, then, would involve data that is not immediately discardable. In other words: data that is assembled in the auditing process, is stored persistently, is protected by authorization schemes and is, always, connected to some end-user functional requirement.
Deployment life-cycle
One view of assessing the maturity of an organization in terms of the deployment of log-management tools might use successive levels such as:
# in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.
# with the increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security perimeter.
# at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the
enterprise
Enterprise (or the archaic spelling Enterprize) may refer to:
Business and economics
Brands and enterprises
* Enterprise GP Holdings, an energy holding company
* Enterprise plc, a UK civil engineering and maintenance company
* Enterpris ...
— especially of those information assets whose availability organizations regard as vital.
# organizations integrate the logs of various
business
Business is the practice of making one's living or making money by producing or Trade, buying and selling Product (business), products (such as goods and Service (economics), services). It is also "any activity or enterprise entered into for pr ...
applications into an enterprise log manager for a better
value proposition
In marketing, a company’s value proposition is the full mix of benefits or economic value which it promises to deliver to the current and future customers (i.e., a market segment) who will buy their products and/or services. It is part of a co ...
.
# organizations merge the physical-access monitoring and the logical-access monitoring into a single view.
See also
*
Audit trail
An audit trail (also called audit log) is a security-relevant chronological record, set of records, and/or destination and source of records that provide documentary evidence of the sequence of activities that have affected at any time a specific ...
*
Common Base Event
*
Common Log Format
For computer log management, the Common Log Format, also known as the NCSA Common log format, (after NCSA HTTPd) is a standardized text file format used by web servers when generating server log files. Because the format is standardized, the ...
*
DARPA
The Defense Advanced Research Projects Agency (DARPA) is a research and development agency of the United States Department of Defense responsible for the development of emerging technologies for use by the military.
Originally known as the Adv ...
PRODIGAL and
Anomaly Detection at Multiple Scales
Anomaly Detection at Multiple Scales, or ADAMS, was a $35 million DARPA project designed to identify patterns and anomalies in very large data sets. It is under DARPA's DARPA#Current program offices, Information Innovation office and began in 2011 ...
(ADAMS) projects.
*
Data logging
A data logger (also datalogger or data recorder) is an electronic device that records data over time or about location either with a built-in instrument or sensor or via external instruments and sensors. Increasingly, but not entirely, they ar ...
*
Log analysis
In computer log management and intelligence, log analysis (or ''system and network log analysis'') is an art and science seeking to make sense of computer-generated records (also called log or audit trail records). The process of creating such reco ...
*
Log monitor
Log monitors are a type of software that monitor log files. Servers, application, network and security devices generate log files. Errors, problems, and more information is constantly logged and saved for later log analysis.
In order to detect ...
*
Log management knowledge base The Log Management Knowledge Base is a free database of detailed descriptions on over 20,000 event logs generated by Windows systems, syslog devices and applications. Provided as a free service to the IT community by Prism Microsystems, the aim of t ...
*
Security information and event management
Security information and event management (SIEM) is a field within the field of computer security, where software products and services combine security information management (SIM) and security event management (SEM). They provide real-time ana ...
*
Server log
In computing, logging is the act of keeping a log of events that occur in a computer system, such as problems, errors or just information on current operations. These events may occur in the operating system or in other software. A message or lo ...
*
Syslog
In computing, syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, i ...
*
Web counter
A web counter or hit counter is a publicly displayed running tally of the number of visits a webpage has received.
Web counters are usually displayed as an inline digital image or in plain text. Image rendering of digits may use a variety of ...
*
Web log analysis software Web log analysis software (also called a web log analyzer) is a kind of web analytics software that parses a server log file from a web server, and based on the values contained in the log file, derives indicators about when, how, and by whom a web ...
References
* Chris MacKinnon: "LMI In The Enterprise". ''Processor'' November 18, 2005, Vol.27 Issue 46, page 33. Online at http://www.processor.com/editorial/article.asp?article=articles%2Fp2746%2F09p46%2F09p46.asp, retrieved 2007-09-10
* MITRE: Common Event Expression (CEE) Proposed Log Standard. Online at http://cee.mitre.org, retrieved 2010-03-03
* NIST 800-92: Guide to Security Log Management. Online at http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf, retrieved 2010-03-03
External links
InfoWorld review and comparison of commercial Log Management products
{{DEFAULTSORT:Log Management And Intelligence
Network management
Computer systems
*