NXLog is a multi-platform log collection and centralization tool that offers log processing features, including log enrichment (parsing, filtering, and conversion) and log forwarding. In concept NXLog is similar to
syslog-ng or
Rsyslog
Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IP network. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, q ...
but it is not limited to
UNIX
Unix (; trademarked as UNIX) is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and ot ...
and
syslog
In computing, syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, i ...
only. It supports all major
operating systems
An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs.
Time-sharing operating systems schedule tasks for efficient use of the system and may also inc ...
such as Windows, macOS, IBM AIX, etc, being compatible with many
SIEM Siem is a surname. Notable people with the surname include:
* Charlie Siem (born 1986), British violinist
* Kjetil Siem (born 1960), Norwegian businessperson, journalist, author and sports official
* Kristian Siem (born 1949), Norwegian businessman ...
, log analytics suites and many other platforms. NXLog can handle different log sources and formats, so it can be used to implement a centralized, scalable logging system. ''NXLog Community Edition'' is proprietary and can be downloaded free of charge with no license costs or limitations.
Overview
NXLog can be installed on many operating systems and it is enabled to operate in a heterogeneous environment, collecting event logs from thousands of different sources in many formats. NXLog can accept event logs from
TCP,
UDP, file, database and various other sources in different formats such as
syslog
In computing, syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, i ...
, windows event log, etc.
It can perform log rewrite, correlation, alerting, pattern matching, it can execute scheduled jobs, and can perform log rotation. It was designed to be able to fully utilize modern multi-core
CPU systems. Its multi-threaded architecture enables input, log processing and output tasks to be executed in parallel. Using an I/O layer it is capable of handling thousands of simultaneous client connections and process log volumes above the 100,000 EPS range.
NXLog does not drop any log messages unless instructed to. It can process input sources in a prioritized order, meaning that a higher priority source will be always processed before others. This can further help avoiding
UDP message loss for example. In case of network congestion or other log transmission problems, NXLog can buffer messages on the disk or in memory. Using loadable modules it supports different input sources and log formats, not only limited to
syslog
In computing, syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, i ...
but windows event log, audit logs, and custom binary application logs.
With NXLog it is possible to use custom loadable modules similarly to the Apache Web server. In addition to the online log processing mode, it can be used to process logs in batch mode in an offline fashion. NXLog's configuration language, with an Apache style configuration file syntax, enables it to rewrite logs, send alerts or execute any external script based on the specified criteria.
History
Back in 2009 the developer of NXlog was using a modified version of msyslog to suit his needs, but when he found a requirement to implement a high performance, scalable, centralized log management solution, there was no such modern logging solution available. There were some alternatives to msyslog with some nice features (e.g.
Rsyslog
Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IP network. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, q ...
,
syslog-ng, etc), but none of them qualified. Most of these were still single threaded,
syslog
In computing, syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, i ...
oriented, without native support for MS Windows, and came with an ambiguous configuration syntax, ugly source-code and so on.
He decided to design and write NXLog from scratch, instead of hacking something else. Thus, NXLog was born in 2009 and was a closed source product in the beginning, heavily used in several production deployments. The source code of NXLOG Community Edition was released in November 2011, and has been freely available since.
Design
Most log processing solutions are built around the same concept. The input is read from a source, then the log messages are processed. Finally output is written or sent to a sink in other terminology.
When an event occurs in an application or a device, depending on its configuration, a log message is emitted. This is usually referred to as an "event log" or "log message". These log messages can have different formats and can be transmitted over different protocols depending on the actual implementation.
There is one thing common in all event log messages. All contain important data such as user names, IP addresses, application names, etc. This way an event can be represented as a list of key-value pairs which we call a "field". The name of the field is the key and the field data is the value. In another terminology this meta-data is sometimes referred to as event property or message tag.
The following example illustrates a syslog message:
<30>Nov 21 11:40:27 log4ensics sshd 6459 Accepted publickey for log4ensics from 192.168.1.1 port 41193 ssh2
The fields extracted from this message are as follows:
AuthMethod publickey
SourceIPAddress 192.168.1.1
AccountName log4ensics
SyslogFacility DAEMON
SyslogSeverity INFO
Severity INFO
EventTime 2009-11-21 11:40:27.0
Hostname log4ensics
ProcessID 26459
SourceName sshd
Message Accepted publickey for log4ensics from 192.168.1.1 port 41193 ssh2
NXLog will try to use th
Common Event Expression standardfor the field names once the standard is stable.
NXLog has a special field, $raw_event. This field is handled by the transport (
UDP,
TCP, File, etc) modules to read input into and write output from it. This field is also used later to parse the log message into further fields by various functions, procedures and modules.
Architecture
By utilizing loadable modules, the plugin architecture of NXLog allows it to read data from any kind of input, parse and convert the format of the messages, and then send it to any kind of output. Different input, processor and output modules can be used at the same time to cover all the requirements of the logging environment. The following figure illustrates the flow of log messages using this architecture.
![Nxlog_architecture](https://upload.wikimedia.org/wikipedia/commons/4/4e/Nxlog_architecture.png)
The core of NXLog is responsible for parsing the configuration file, monitoring files and sockets, and managing internal events. It has an event based architecture, all modules can dispatch events to the core. The NXLog core will take care of the event and will optionally pass it to a module for processing. NXLog is a multi-threaded application, the main thread is responsible for monitoring files and sockets. These are added to the core by the different input and output modules. There is a dedicated thread handling internal events. It sleeps until the next event is to be processed then wakes up and dispatches the event to a worker thread. NXLog implements a worker thread-pool model. Worker threads receive an event which must be processed immediately. This way the NXLog core can centrally control all events and the order of their execution making prioritized processing possible. Modules which handle sockets or files are written to use non-blocking I/O in order to ensure that the worker threads never block. The files and sockets monitored by the main thread also dispatch events which are then delegated to the workers. Each event belonging to the same module is executed in sequential order, not concurrently. This ensures that message order is kept and prevents concurrency issues in modules. Yet the modules (worker threads) run concurrently, thus the global log processing flow is greatly parallelized.
When an input module receives data, it creates an internal representation of the log message which is basically a structure containing the raw event data and any optional fields. This log message is then pushed to the queue of the next module in the route and an internal event is generated to signal the availability of the data. The next module after the input module in a route, can be either a processor module or an output module. Actually an input or output module can also process data through built-in code or using the NXLog language execution framework. The only difference is that processor modules are run in another worker thread, thus parallelizing log processing even more. Considering that processor modules can also be chained, this can efficiently distribute work among multiple
CPUs or
CPU cores in the system.
Distributions
* NXLog Community Edition is a proprietary log management tool available at no cost. It is available for various platforms including Windows and Linux. The NXLog Community Edition can be used both as a log collector agent and as a log server. Summary of features:
** Multi-platform - support for
Linux
Linux ( or ) is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged as a Linux distribution, which ...
,
IBM AIX
AIX (Advanced Interactive eXecutive, pronounced , "ay-eye-ex") is a series of Proprietary software, proprietary Unix operating systems developed and sold by IBM for several of its computer platforms.
Background
Originally released for the ...
,
Solaris
Solaris may refer to:
Arts and entertainment Literature, television and film
* ''Solaris'' (novel), a 1961 science fiction novel by Stanisław Lem
** ''Solaris'' (1968 film), directed by Boris Nirenburg
** ''Solaris'' (1972 film), directed by ...
,
HP-UX
HP-UX (from "Hewlett Packard Unix") is Hewlett Packard Enterprise's proprietary implementation of the Unix operating system, based on Unix System V (initially System III) and first released in 1984. Current versions support HPE Integrity Ser ...
,
BSD
The Berkeley Software Distribution or Berkeley Standard Distribution (BSD) is a discontinued operating system based on Research Unix, developed and distributed by the Computer Systems Research Group (CSRG) at the University of California, Berk ...
,
Android and
Microsoft Windows
Windows is a group of several proprietary graphical operating system families developed and marketed by Microsoft. Each family caters to a certain sector of the computing industry. For example, Windows NT for consumers, Windows Server for serv ...
(from
XP through 2012)
** Modular architecture through dynamically loadable plugins
** Scalable, high-performance I/O - collect messages at blazing speeds (can achieve above 500k EPS)
** Message buffering and prioritization - no lost or dropped messages
** Simple configuration format with a powerful language similar to
Perl
Perl is a family of two high-level, general-purpose, interpreted, dynamic programming languages. "Perl" refers to Perl 5, but from 2000 to 2019 it also referred to its redesigned "sister language", Perl 6, before the latter's name was offici ...
** Scheduled tasks and built-in log rotation
** Support for different formats such as
Syslog
In computing, syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, i ...
,
CSV, GELF,
JSON
JSON (JavaScript Object Notation, pronounced ; also ) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other ser ...
,
XML
Extensible Markup Language (XML) is a markup language and file format for storing, transmitting, and reconstructing arbitrary data. It defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. T ...
, Windows EventLog and even custom formats
** Offline processing mode for post processing, conversion or transfer
** Event classification and pattern matching
** Log message rewrite, conversion between different formats
** Event correlation
** Secure network transport over
SSL
** Internationalization for supporting different character sets and on the fly auto-detection of encoding
* NXLog Enterprise Edition: In addition to the features of NXLog Community Edition such as the flexibility, low memory footprint and high performance, the NXLog Enterprise Edition contains several enhancements which can be useful in enterprise deployments. These features are enhanced reliability, support for additional log sources such as CheckPoint LEA and
SNMP
Simple Network Management Protocol (SNMP) is an Internet Standard protocol for collecting and organizing information about managed devices on IP networks and for modifying that information to change device behaviour. Devices that typically ...
events, agent management and monitoring capabilities, remote Windows EventLog collection,
ODBC
In computing, Open Database Connectivity (ODBC) is a standard application programming interface (API) for accessing database management systems (DBMS). The designers of ODBC aimed to make it independent of database systems and operating systems. An ...
input and output modules to read/write data into databases (e.g. Microsoft SQL Server) and many more. Summary of features:
** Multi-platform - support for
Unix
Unix (; trademarked as UNIX) is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and ot ...
/
Linux
Linux ( or ) is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged as a Linux distribution, which ...
,
MS Windows
** Modular architecture through dynamically loadable plugins
** Scalable, high-performance I/O - collect messages above 100k events per second from thousands of sources
** Message buffering and prioritization - no lost or dropped messages
** Simple configuration format with a powerful language similar to
Perl
Perl is a family of two high-level, general-purpose, interpreted, dynamic programming languages. "Perl" refers to Perl 5, but from 2000 to 2019 it also referred to its redesigned "sister language", Perl 6, before the latter's name was offici ...
** Scheduled tasks and built-in log rotation
** Support for different formats such as
syslog
In computing, syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyzes them. Each message is labeled with a facility code, i ...
,
CSV, Windows EventLog, CheckPoint LEA or even custom formats
** Remote administration and statistics with a
WebService API for easier integration to external monitoring tools
** Offline processing mode for post processing, conversion or transfer
** Real-time event correlation and classification
** Log message rewrite, conversion between different formats
**
Secure SSL transport, message integrity and timestamping support
** Internationalization for supporting different character sets and on the fly auto-detection of encoding.
* NXLog Manager: Managing and monitoring a large number of log collector agents can be tough, especially if you have a lot of servers in different roles with multiple teams in charge. NXLog Manager can remotely manage and monitor NXLog Enterprise Edition instances using a centralized web based management console.
Features
Releases
License
NXLog Community Edition is licensed under the NXLOG PUBLIC LICENSE v1.0.
[https://nxlog.co/nxlog-public-license NXLOG PUBLIC LICENSE v1.0]
References
{{Reflist
External links
NXLog official websiteNXLog Community Edition download site
Data management software