How to unbreak LTTng

LTTng is a tracing subsystem; to carry out that sort of task, it must be able to hook into the kernel in a number of fairly deep places. It is unsurprising that LTTng was accessing parts of the kernel that are not deemed suitable for export to modules in general. Losing access to kallsyms_on_each_symbol() deprived LTTng of the ability to find those addresses, thus breaking much of its functionality. That is not welcome news to those who work on — or use — LTTng.

LTTng developer Mathieu Desnoyers has responded to this change with a patch series exporting a number of new symbols; with those available, LTTng can do what it needs to do without using the rather more general kallsyms_on_each_symbol() function. For example, LTTng needs access to stack_trace_save_user() to be able to save user-space stack traces. It also needs access to functions like task_prio(), disk_name(), and get_pfn_blocks_mask(). LTTng obtains kernel information from tracepoints as well, of course, and that usage will increase as tracepoints replace some of the direct internal accesses that were used before. The patch set raises the number of arguments that can be passed to a BPF program from a tracepoint to an eye-opening 13 (to allow more information to be passed out via a specific tracepoint), but that change may prove to be unnecessary in the end.

Anybody who has watched the kernel community for any period of time can probably guess what sort of reception this patch series received. Christoph Hellwig was characteristically blunt: "Which part of every added export needs an in-tree user did you not get?" The kernel community as a whole is strongly resistant to the idea of adding any sort of support for code that is outside of the kernel repository. Much of that resistance comes from a dislike for proprietary kernel modules in general, but there is a bit more to it than that.

LTTng, being free software, should not be affected by any antipathy for proprietary kernel code. But, as Greg Kroah-Hartman explained, there are still reasons to avoid adding support for free, out-of-tree modules. Once those modules are supported in some way, they add constraints to what kernel developers can do. Internal kernel interfaces can be changed as needed; since all of the users of those interfaces are present in the same code base, they can be changed at the same time. If external modules have to be supported, though, it becomes harder to make such changes, since the users cannot be changed to match. Indeed, it becomes difficult to even know when a change might cause problems elsewhere.

Thus, Kroah-Hartman said:

This all suggests that there is not much of a path forward for LTTng. It is unable to function without access to kernel internals, and that access is being expressly denied.

There is, of course, one other option that was first raised by Steve Rostedt: "I guess we should be open to allowing LTTng modules in the kernel as well". If LTTng were actually a part of the mainline kernel, there would no longer be problems with giving access to the resources that it needs.

This is not a new idea. Numerous attempts have been made to get the LTTng code into the mainline kernel, without success. In the early days, before the kernel had any sort of tracing capability at all, adding that feature was a hard sell. Kernel developers now are heavily dependent on tracing for their own work and would strongly resist any attempt to take that capability away, but it was not that long ago that many of the same developers were unconvinced that tracing was needed at all. During that time, getting any tracing features into the kernel was not easy.

Over time, some low-level LTTng code found its way in, but LTTng as a whole has not followed. More recently, in 2011, LTTng was brought into the staging tree by Kroah-Hartman as a first step toward merging it. That move brought about a great deal of hostility, some of which seems familiar; a rather lengthy thread was set off by an attempt to export task_prio(), for example. In the end, LTTng was pushed back out of the staging tree — as it was before and has been ever since.

So LTTng would appear to be in a difficult position: unable to function outside of the kernel, and unable to be merged. Leaving LTTng broken would cause serious harm to a lot of users, though, and seems unlikely to advance the cause of Linux or free software in general. So perhaps the time has come for something to give. If a handful of symbols truly cannot be exported for this subsystem, perhaps some space could be found in the mainline for a widely used tracing subsystem, even if it somehow duplicates some of the functionality that is already there.