DTrace is a comprehensive dynamic

tracing Tracing may refer to: Computer graphics * Image tracing, digital image processing to convert raster graphics into vector graphics * Path tracing, a method of rendering images of three-dimensional scenes such that the global illumination is faithf ...

framework originally created by

Sun Microsystems Sun Microsystems, Inc. (Sun for short) was an American technology company that sold computers, computer components, software, and information technology services and created the Java programming language, the Solaris operating system, ZFS, the ...

for

troubleshooting Troubleshooting is a form of problem solving, often applied to repair failed products or processes on a machine or a system. It is a logical, systematic search for the source of a problem in order to solve it, and make the product or process ope ...

kernel Kernel may refer to: Computing * Kernel (operating system), the central component of most operating systems * Kernel (image processing), a matrix used for image convolution * Compute kernel, in GPGPU programming * Kernel method, in machine learn ...

and application problems on production systems in real time. Originally developed for

Solaris Solaris may refer to: Arts and entertainment Literature, television and film * ''Solaris'' (novel), a 1961 science fiction novel by Stanisław Lem ** ''Solaris'' (1968 film), directed by Boris Nirenburg ** ''Solaris'' (1972 film), directed by ...

, it has since been released under the free Common Development and Distribution License (CDDL) in OpenSolaris and its descendant

illumos Illumos (stylized as illumos) is a partly free and open-source Unix operating system. It is based on OpenSolaris, which was based on System V Release 4 (SVR4) and the Berkeley Software Distribution (BSD). Illumos comprises a kernel, device d ...

, and has been ported to several other

Unix-like A Unix-like (sometimes referred to as UN*X or *nix) operating system is one that behaves in a manner similar to a Unix system, although not necessarily conforming to or being certified to any version of the Single UNIX Specification. A Unix-li ...

systems. DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file. In 2010, Oracle Corporation acquired Sun Microsystems and announced discontinuing OpenSolaris. As a community effort of some core Solaris engineers to create a truly open source Solaris,

operating system An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs. Time-sharing operating systems schedule tasks for efficient use of the system and may also in ...

was announced via

webinar Web conferencing is used as an umbrella term for various types of online conferencing and collaborative services including webinars (web seminars), webcasts, and web meetings. Sometimes it may be used also in the more narrow sense of the peer-l ...

on Thursday, 3 August 2010, as a fork on OpenSolaris OS/Net consolidation, including DTrace technology. In October 2011, Oracle announced the

porting In software engineering, porting is the process of adapting software for the purpose of achieving some form of execution in a computing environment that is different from the one that a given program (meant for such execution) was originally desi ...

of DTrace to

Linux Linux ( or ) is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged as a Linux distribution, which ...

, and in 2019 official DTrace for Fedora is available on

GitHub GitHub, Inc. () is an Internet hosting service for software development and version control using Git. It provides the distributed version control of Git plus access control, bug tracking, software feature requests, task management, continuous ...

. For several years an unofficial DTrace port to Linux was available, with no changes in licensing terms. In August 2017, Oracle released DTrace kernel code under the

GPLv2+ The GNU General Public License (GNU GPL or simply GPL) is a series of widely used free software licenses that guarantee end users the four freedoms to run, study, share, and modify the software. The license was the first copyleft for general u ...

license, and

user space A modern computer operating system usually segregates virtual memory into user space and kernel space. Primarily, this separation serves to provide memory protection and hardware protection from malicious or errant software behaviour. Kernel ...

code under GPLv2 and UPL licensing. In September 2018

Microsoft Microsoft Corporation is an American multinational technology corporation producing computer software, consumer electronics, personal computers, and related services headquartered at the Microsoft Redmond campus located in Redmond, Washing ...

announced that they had ported DTrace from

FreeBSD FreeBSD is a free and open-source Unix-like operating system descended from the Berkeley Software Distribution (BSD), which was based on Research Unix. The first version of FreeBSD was released in 1993. In 2005, FreeBSD was the most popular ...

to Windows. In September 2016 the OpenDTrace effort began o
github
with both code and comprehensiv
documentation
of the system's internals. The OpenDTrace effort maintains the original CDDL licensing for the code from OpenSolaris with additional code contributions coming under a BSD 2 Clause license. The goal of OpenDTrace is to provide an OS agnostic, portable implementation of DTrace that is acceptable to all consumers, including macOS, FreeBSD, OpenBSD, NetBSD, and Linux as well as embedded systems.

Description

Sun Microsystems designed DTrace to give operational insights that allow users to tune and troubleshoot applications and the OS itself. Testers write tracing programs (also referred to as scripts) using the D programming language (not to be confused with other programming languages named "D"). The language, inspired by C, includes added functions and variables specific to tracing. D programs resemble

awk AWK (''awk'') is a domain-specific language designed for text processing and typically used as a data extraction and reporting tool. Like sed and grep, it is a filter, and is a standard feature of most Unix-like operating systems. The AWK lang ...

programs in structure; they consist of a list of one or more ''probes'' (instrumentation points), and each probe is associated with an action. These probes are comparable to a

pointcut In aspect-oriented programming, a pointcut is a set of join points. Pointcut specifies where exactly to apply advice, which allows separation of concerns and helps in modularizing business logic. Pointcuts are often specified using class names or m ...

aspect-oriented programming In computing, aspect-oriented programming (AOP) is a programming paradigm that aims to increase modularity by allowing the separation of cross-cutting concerns. It does so by adding behavior to existing code (an advice) ''without'' modifying th ...

. Whenever the condition for the probe is met, the associated action is executed (the probe "fires"). A typical probe might fire when a certain file is opened, or a process is started, or a certain line of code is executed. A probe that fires may analyze the run-time situation by accessing the

call stack In computer science, a call stack is a stack data structure that stores information about the active subroutines of a computer program. This kind of stack is also known as an execution stack, program stack, control stack, run-time stack, or ma ...

and context variables and evaluating expressions; it can then print out or log some information, record it in a database, or modify context variables. The reading and writing of context variables allows probes to pass information to each other, allowing them to cooperatively analyze the correlation of different events. Special consideration has been taken to make DTrace safe to use in a production environment. For example, there is minimal

probe effect Probe effect is an unintended alteration in system behavior caused by measuring that system. In code profiling and performance measurements, the delays introduced by insertion or removal of code instrumentation may result in a non-functioning appl ...

when tracing is underway, and no performance impact associated with any disabled probe; this is important since there are tens of thousands of DTrace probes that can be enabled. New probes can also be created dynamically.

Command line examples

DTrace scripts can be invoked directly from the command line, providing one or more probes and actions as arguments. Some examples: # New processes with arguments dtrace -n 'proc:::exec-success ' # Files opened by process dtrace -n 'syscall::open*:entry ' # Syscall count by program dtrace -n 'syscall:::entry ' # Syscall count by syscall dtrace -n 'syscall:::entry ' # Syscall count by process dtrace -n 'syscall:::entry ' # Disk size by process dtrace -n 'io:::start ' # Pages paged in by process dtrace -n 'vminfo:::pgpgin ' Scripts can also be written which can reach hundreds of lines in length, although typically only tens of lines are needed for advanced troubleshooting and analysis. Over 200 examples of open source DTrace scripts can be found in the DTraceToolkit, created by Brendan Gregg (author of the DTrace book), which also provides documentation and demonstrations of each.

Supported platforms

DTrace first became available for use in November 2003, and was formally released as part of Sun's

Solaris 10 Solaris is a proprietary Unix operating system originally developed by Sun Microsystems. After the Sun acquisition by Oracle in 2010, it was renamed Oracle Solaris. Solaris superseded the company's earlier SunOS in 1993, and became known for ...

in January 2005. DTrace was the first component of the OpenSolaris project to have its source code released under the Common Development and Distribution License (CDDL). DTrace is an integral part of

and related distributions. DTrace is a standard part of FreeBSD and NetBSD. Apple added DTrace support in Mac OS X 10.5 "Leopard", including a GUI called

Instruments Instrument may refer to: Science and technology * Flight instruments, the devices used to measure the speed, altitude, and pertinent flight angles of various kinds of aircraft * Laboratory equipment, the measuring tools used in a scientific lab ...

. Over 40 DTrace scripts from the DTraceToolkit are included in /usr/bin, including tools to examine disk I/O (iosnoop) and process execution (execsnoop). Unlike other platforms that DTrace is supported on, Mac OS X has a flag (P_LNOATTACH) that a program may set that disallows tracing of that process by debugging utilities such as DTrace and

gdb The GNU Debugger (GDB) is a portable debugger that runs on many Unix-like systems and works for many programming languages, including Ada, C, C++, Objective-C, Free Pascal, Fortran, Go, and partially others. History GDB was first written by ...

. In the original Mac OS X DTrace implementation, this could affect tracing of other system information, as unrelated probes that should fire while a program with this flag set was running would fail to do so. The OS X 10.5.3 update addressed this issue a few months later. However, since El Capitan, System Integrity Protection prevents user from DTracing protected binary by default. The

port of DTrace has been available since 2008; work continues actively to enhance and fix issues. There is also an activ
implementation on github
Standard core providers are available (fbt, syscall, profile), plus a special "instr" provider (some of the Solaris providers are not yet available ). The Linux DTrace implementation is a loadable

kernel module In computing, a loadable kernel module (LKM) is an object file that contains code to extend the running kernel, or so-called ''base kernel'', of an operating system. LKMs are typically used to add support for new hardware (as device drivers) and/o ...

, which means that the kernel itself requires no modification, and thus allows DTrace to avoid CDDL vs. GPL licensing conflicts (in its source form, at least). However, once DTrace is loaded the kernel instance will be marked as tainted. In 2007, a developer at QNX Software Systems announced on his blog that he and a colleague were working on incorporating DTrace into the

QNX QNX ( or ) is a commercial Unix-like real-time operating system, aimed primarily at the embedded systems market. QNX was one of the first commercially successful microkernel operating systems. The product was originally developed in the early ...

operating system. Oracle Corporation added beta DTrace support for

Oracle Linux Oracle Linux (abbreviated OL, formerly known as Oracle Enterprise Linux or OEL) is a Linux distribution packaged and freely distributed by Oracle, available partially under the GNU General Public License since late 2006. It is compiled from Red ...

in 2011, as a technology preview in the

Unbreakable Enterprise Kernel Oracle Linux (abbreviated OL, formerly known as Oracle Enterprise Linux or OEL) is a Linux distribution packaged and freely distributed by Oracle, available partially under the GNU General Public License since late 2006. It is compiled from R ...

release 2, which is under GPLv2 (the DTrace Linux kernel module was originally released under CDDL). General availability was announced in December 2012. On March 11, 2019, Microsoft released their build of DTrace for the Windows 10 insider builds.

Language and application providers

With a supported ''language provider'', DTrace can retrieve context of the code, including function, source file, and line number location. Further, dynamic memory allocation and garbage collection can be made available if supported by the language. Supported language providers include

assembly language In computer programming, assembly language (or assembler language, or symbolic machine code), often referred to simply as Assembly and commonly abbreviated as ASM or asm, is any low-level programming language with a very strong correspondence be ...

, C,

C++ C++ (pronounced "C plus plus") is a high-level general-purpose programming language created by Danish computer scientist Bjarne Stroustrup as an extension of the C programming language, or "C with Classes". The language has expanded significan ...

Java Java (; id, Jawa, ; jv, ꦗꦮ; su, ) is one of the Greater Sunda Islands in Indonesia. It is bordered by the Indian Ocean to the south and the Java Sea to the north. With a population of 151.6 million people, Java is the world's List ...

, Erlang,

JavaScript JavaScript (), often abbreviated as JS, is a programming language that is one of the core technologies of the World Wide Web, alongside HTML and CSS. As of 2022, 98% of Website, websites use JavaScript on the Client (computing), client side ...

Perl Perl is a family of two high-level, general-purpose, interpreted, dynamic programming languages. "Perl" refers to Perl 5, but from 2000 to 2019 it also referred to its redesigned "sister language", Perl 6, before the latter's name was offici ...

PHP PHP is a general-purpose scripting language geared toward web development. It was originally created by Danish-Canadian programmer Rasmus Lerdorf in 1993 and released in 1995. The PHP reference implementation is now produced by The PHP Group. ...

Python Python may refer to: Snakes * Pythonidae, a family of nonvenomous snakes found in Africa, Asia, and Australia ** ''Python'' (genus), a genus of Pythonidae found in Africa and Asia * Python (mythology), a mythical serpent Computing * Python (pro ...

Ruby A ruby is a pinkish red to blood-red colored gemstone, a variety of the mineral corundum ( aluminium oxide). Ruby is one of the most popular traditional jewelry gems and is very durable. Other varieties of gem-quality corundum are called sa ...

, shell script, and

Tcl TCL or Tcl or TCLs may refer to: Business * TCL Technology, a Chinese consumer electronics and appliance company **TCL Electronics, a subsidiary of TCL Technology * Texas Collegiate League, a collegiate baseball league * Trade Centre Limited ...

. ''Application providers'' allow DTrace to follow the operation of applications through system calls and into the kernel. Applications that offer DTrace application providers include

MySQL MySQL () is an open-source relational database management system (RDBMS). Its name is a combination of "My", the name of co-founder Michael Widenius's daughter My, and "SQL", the acronym for Structured Query Language. A relational database o ...

, PostgreSQL, Oracle Database,

Oracle Grid Engine Oracle Grid Engine, previously known as Sun Grid Engine (SGE), CODINE (Computing in Distributed Networked Environments) or GRD (Global Resource Director), was a grid computing computer cluster software system (otherwise known as a batch-queuing ...

, and

Firefox Mozilla Firefox, or simply Firefox, is a free and open-source web browser developed by the Mozilla Foundation and its subsidiary, the Mozilla Corporation. It uses the Gecko rendering engine to display web pages, which implements current ...

Authors and awards

DTrace was designed and implemented by Bryan Cantrill, Mike Shapiro, and Adam Leventhal. The authors received recognition in 2005 for the innovations in DTrace from ''

InfoWorld ''InfoWorld'' (abbreviated IW) is an information technology media business. Founded in 1978, it began as a monthly magazine. In 2007, it transitioned to a web-only publication. Its parent company today is International Data Group, and its siste ...

'' and ''

Technology Review ''MIT Technology Review'' is a bimonthly magazine wholly owned by the Massachusetts Institute of Technology, and editorially independent of the university. It was founded in 1899 as ''The Technology Review'', and was re-launched without "The" in ...

''. DTrace won the top prize in ''

The Wall Street Journal ''The Wall Street Journal'' is an American business-focused, international daily newspaper based in New York City, with international editions also available in Chinese and Japanese. The ''Journal'', along with its Asian editions, is published ...

''s 2006 Technology Innovation Awards competition. The authors were recognized by USENIX with the Software Tools User Group (STUG) award in 2008.

References

* *

Notes

Oracle Solaris Studio Oracle Developer Studio, formerly named Oracle Solaris Studio, Sun Studio, Sun WorkShop, Forte Developer, and SunPro Compilers, is Oracle Corporation's flagship software development product for the Solaris and Linux operating systems. It inclu ...

prior to version 12.4
Exploring Leopard with DTrace
DTrace for debugging and exploration
Tech Talk on DTrace given by Bryan Cantrill

Hidden in Plain Sight
Sun Microsystems, by Bryan Cantrill
dtrace.org
hosts blogs from software developers related to DTrace * Official Oracle DTrace port to Linux: *
DTrace patches merged with recent Linux kernel releases
*
Userspace portion of the DTrace port to Linux
*
Compact Type Format library used by DTrace on Linux

DTrace on Fedora
{{Oracle FOSS Command-line software Sun Microsystems software Free system software Debuggers Aspect-oriented programming Software using the CDDL license Linux kernel features