This is the second part of the 7 part series “The Wonderland of Dynamic Tracing.” This series will consist of updates on the state of art of the dynamic tracing world.

The previous part, Part 1, introduced the basic concepts of dynamic tracing and covered its advantages. This part will continue looking at some of the most popular dynamic tracing frameworks in the open source world.

See also Part 3 and Part 4.

DTrace

We cannot talk about dynamic tracing without mentioning DTrace. DTrace is the earliest modern dynamic tracing framework. Originating from the Solaris operating system at the beginning of this century, it was developed by engineers of Sun Microsystems. Many you may have heard about the Solaris system and its original developer Sun.

A story circulates around the creation of DTrace. Once upon a time, several kernel engineers of the Solaris operating system spent several days and nights troubleshooting a very weird online issue. They originally considered it very complicated, and spared great effort to address it, only to realize it was just a very silly configuration issue. Learning from the painful experience, they created DTrace, a highly sophisticated debugging framework to enable tools which can prevent them from going through similar pains in the future. Indeed, most of the so-called“weird problems”(high CPU or memory usage, high disk usage, long latency, program crashes etc.) are so embarrassing that it is even more depressing after pinpointing the real cause.

As a highly general-purpose debugging platform, DTrace provides the D language, a scripting language that looks like C. All DTrace-based debugging tools are written by D. The D language supports special notations to specify“probes”which usually contain information about code locations in the target software system (being either in the OS kernel or in a user-land process). For example, you can put the probe at the entry or exit of a certain kernel function, or the function entry or exit of certain user mode processes, and even on any machine instructions. Writing debugging tools in the D language requires some understanding and knowledge of the target software system. These powerful tools can help us regain insight of complex systems, greatly increasing the observability. Brendan Gregg, a former engineer of Sun, was one of the earliest DTrace users, even before DTrace was open-sourced. Brendan wrote a lot of reusable DTrace-based debugging tools, most of which are in the open-source project called DTrace Toolkit. Dtrace is the earliest and one of the most famous dynamic tracing frameworks.

DTrace Pros and Cons

DTrace has an edge in closely integrating with the operating system kernel. Implementation of the D language is actually a virtual machine (VM), kinda like a Java virtual machine (JVM). One benefit of the D language is that its runtime is resident in the kernel and is very compact, meaning the startup and quitting time for the debugging tools are very short. However, I think DTrace also has some notable weaknesses. The most frustrating one is the lack of looping language structures in D, making it very hard to write many analytical tools targeting complicated data structures in the target. The official statement attributed the lack to the purpose of avoiding infinite loops, but clearly DTrace can instead limit the iteration count of each loop on the VM level. The same applies to recursive function calls. Another major flaw is its relatively weak tracing support for user-mode code as it has no built-in support for utilizing user-mode debug symbols. So the user must declare in their D code the type of user-mode C language structures used.¹

DTrace has such a large influence that many engineers port it over to several other operating systems. For example, Apple has added DTrace support in its Mac OS X (and later macOS) operating system. In fact, each Apple laptop or desktop computer launched in recent years offers a ready-to-use dtrace command line utility. Those who have an Apple computer can have a try on its command line terminal. Alongside the Apple system, DTrace has also made its way into the FreeBSD operating system. Not enabled by default, the DTrace kernel module in FreeBSD must be loaded through extra user commands. Oracle has also tried to introduce DTrace into their own Linux distribution, Oracle Linux, without much progress though. This is because the Linux kernel is not controlled by Oracle, but DTrace needs close integration with the operating system kernel. Similar reasons have long left the DTrace-to-Linux porting attempted by some engineers to be far below the production-level requirement.

Those DTrace ports lack some advanced features here and there (it would be nice to have the floating-point number support, and they are also missing support for a lot of built-in probes etc.) In addition, they cannot really match the original DTrace implementation in the Solaris operating system.

SystemTap

Another influence of DTrace on the Linux operating system is reflected in the open-source project SystemTap, a relatively independent dynamic tracing framework built by engineers from Red Hat and other companies. SystemTap has its own little language, the SystemTap scripting language, which is not compatible with DTrace’s D language (although it does also resemble C). Serving a wide range of enterprise-level users, Red Hat naturally relies on engineers who have to cope with a lot of“weird problems”on a daily basis. The real-life demand has inevitably prompted it to develop this technology. In my opinion, SystemTap is one of the most powerful and the most usable dynamic tracing frameworks in today’s open source Linux world, and I have been using it in work for years. Authors of SystemTap, including Frank Ch. Eigler and Josh Stone are all very smart engineers full of enthusiasm. I once raised questions through their IRC channel and their mailing list, and they often answered me very quickly and in great detail. I’ve been contributing to SystemTap by adding new features and fixing bugs.

SystemTap Pros and Cons

The strengths of SystemTap include its great maturity in automatic loading of user-mode debug symbols, complete looping language structures to write complicated probe processing programs, with support for a great number of complex aggregations and statistics. Due to the immature implementation of SystemTap and Linux kernels in the early days, outdated criticisms over SystemTap have already flooded the Internet, unfortunately. In the past few years we have witnessed significant improvements in it. In 2017, I established OpenResty Inc. which have also been helping improve SystemTap.

Of course, SystemTap is not perfect. Firstly, it’s not a part of the Linux kernel, and such lack of close integration with the kernel means SystemTap has to keep track of changes in the mainline kernel all the time. Secondly, SystemTap usually complies (or “translates”) its language scripts (in its own language) into C source code of a Linux kernel module. It is therefore often necessary to deploy the full C compiler toolchain and the header files of the Linux kernel in online systems². For these reasons, SystemTap script starts much more slowly than DTrace, and at a speed similar to JVM. Overall, SystemTap is still a very mature and outstanding dynamic tracing framework despite these shortcomings³.

DTrace and SystemTap

Neither DTrace nor SystemTap supports writing complete debugging tools as both lack convenient primitives for command-line interactions. This is why a slew of real world tools based on them have come with wrappers written in Perl, Python, and even Shell script⁴. To use a clean language to write complete debugging tools, I once extended the SystemTap language to a higher-level“macro language”called stap++⁵. I employed Perl to implement the stap++ interpreter capable of directly interpreting and executing stap++ source code and internally calling the SystemTap command-line tool. Those interested please visit GitHub for my open-source code repository stapxx, where many complete debugging tools backed by my stap++ macro language are available.

Conclusion

This part of the series introduced two famous dynamic tracing frameworks, DTrace and SystemTap and covered their strengths and weaknesses. The next part, Part 3, will talk about applications of SystemTap to solve really hard problems. Stay tuned!

A Word on OpenResty XRay

OpenResty XRay is a commercial dynamic tracing product offered by our OpenResty Inc. company. We use this product in our articles like this one to demonstrate implementation details, as well as provide statistics about real world applications and open source software. In general, OpenResty XRay can help users gain deep insight into their online and offline software systems without any modifications or any other collaborations, and efficiently troubleshoot difficult problems for performance, reliability, and security. It utilizes advanced dynamic tracing technologies developed by OpenResty Inc. and others.

We welcome you to contact us to try out this product for free.

About The Author

Yichun Zhang is the creator of the OpenResty^® open source project. He is also the founder and CEO of the OpenResty Inc. company. He contributed a dozen open source Nginx 3rd-party modules, many Nginx and LuaJIT core patches, and designed the OpenResty XRay platform.

Translations

We provide a Chinese translation for this article on blog.openresty.com.cn We also welcome interested readers to contribute translations in other
languages as long as the full article is translated without any omissions. We thank anyone willing to do so in advance.

1. Neither SystemTap nor OpenResty XRay has these restrictions. ↩
2. SystemTap also supports the “translator server” mode which can remotely compile its scripts on some dedicated machines. But it is still required to deply the C compiler tool chain and header files on these “server” machines. ↩
3. OpenResty XRay’s dynamic tracing solution has overcome SystemTap’s these shortcomings. ↩
4. Alas. The newer BCC framework targeting Linux’s eBPF technology also suffers from such ugly tool wrappers for its standard usage. ↩
5. The stap++ project is no longer maintained and has been superseded by the new generation of dynamic tracing framework provided by OpenResty XRay. ↩