File monitoring with Mortadelo and SystemTap

SystemTap is a tool to help gather information about running Linux systems which has been available for some time now. But applications that use the tool have been few and far between. Mortadelo is a GUI tool that uses SystemTap to observe and record system calls. It is more of a proof-of-concept than a complete application—though it is useful in its current form—but it does start to show some of the things that can be done using SystemTap.

Mortadelo specifically intercepts system calls that deal with accessing files, collecting the arguments to the calls as well the return codes. It is patterned after the Windows Filemon program, which is used in much the same way that a Linux user might use strace—only with a GUI. Problems with permissions or files that do not exist are the kinds of things that Mortadelo could be used to diagnose.

The data collected is displayed in a list in the GUI (shown at left), which can then be filtered using regular expressions to pull out the information of interest. Because it uses SystemTap, Mortadelo gathers information from all running processes at once, allowing the user to choose which parts they are interested in. The filtering is somewhat primitive, in that particular fields cannot be chosen to filter on, but still useful because it searches each entry fully.

System calls that return an error are highlighted in red making it easy to pick them out. By choosing appropriate strings to filter on, all permission errors in the system or every access of a particular filename can be seen. The GUI allows one to start and stop the recording as well as to save the captured data to a file. Each entry includes a timestamp, the process name and pid, the system call, return code, and arguments.

The application is written in C#, using the Mono framework; one of the authors has an interesting weblog entry comparing Mono and Python for developing this kind of tool. Mortadelo's interface to SystemTap is fairly straightforward, it spawns a stap command and sends it the probe points and code via stdin. It then reads the stap output, parsing it and displaying it in the window.

There were some tricks to getting it to build and run, but Eugene Teo's instructions for running it on Fedora 8 were quite helpful. Part of the problem was in getting SystemTap going on the system, which is a problem we have mentioned before. There were some other small hurdles as well, but Teo's hints and proper application of grep were enough to get past those.

Mortadelo's impact isn't so much in the application itself as it is in some of the ideas behind it. Using SystemTap for GUI tools will help users and administrators, especially those who are not command-line savvy. If Mortadelo, or some descendant of it, becomes popular, that will help make SystemTap use more widespread. Distributors will start packaging it in more readily usable forms, perhaps installing it by default. That will in turn help anyone tasked with keeping a Linux system smoothly functioning, whether they are GUI-centric or not.

Is there some redundancy between Audit and SystemTap ?

Audit can monitor a system call .. SystemTap does so
Audit can monitor single files .. Again, System tap does so

The uncommon thing now is the ability to audit LSMs by ,say, a MAC subject label and giving
LSMs an easy structure to report violations.

It'll be interesting to see how will they both react once SystemTap gains more popularity.

> Is there some redundancy between Audit and SystemTap ?

Sure.  Other than logistical (installation) issues though,
there is the potential for more interesting differences.

Audit is a single system-wide facility, so only a single
configuration (set of trace points) can be active at a time.
Systemtap is per-session, so many different probing sessions
collecting different sorts of data can run at the same time.

Mortadelo represents only a basic use of systemtap at the
present (an unconditional trace record for a bunch of
systemcalls, system-wide).  It could do something richer,
like dynamically adjusting the target process/syscall list
to reduce trace data quantity (-> improve performance, reduce
system impact); to encode user-specified filters; to change
these even during systemtap probe run-time using a /proc file
interface.

Aha .. Thanks for this great explanation.

There seem to be some hints and allegations of IronPython running on Mono.

I'm a Ruby guy, but still enjoyed the Mono/Python thoughts. Especially his final conclusion:

  Overall:
  I'm not writing new programs in C ever again. 

Yes, I know C is great for super-high-performance code, and for super-low-memory (embedded)
footprints, and for writing super-portable libraries. So for any of those, I might consider
it. Fortunately for me, I haven't had to write any of those for about 20 years.

It's hard to imagine a "normal" app that would best be written in C.

Mortadelo & Filemon is a famous comic here in Spain.
http://www.mortadeloyfilemon.com

It's not really right to suggest that the biggest difference between SystemTap and strace is
the GUI.

Strace can only trace one manually selected process or process tree. That makes it less than
useful once you start to involve D-BUS, HAL, kernel interaction, and other sorts of
inter-process communication.

By contrast, systemtap (and FileMon on Windows) can trace a whole system's actions then filter
out uninteresting data.

I like, and use, both.

My point is that the system-wide aspect of systemtap is a vastly more important difference
between it and strace than the Mortadelo GUI is.