Improving EXPORT_SYMBOL()

The kernel's EXPORT_SYMBOL() directive is used to make kernel symbols (functions and data structures) available to loadable modules; only the symbols that have been explicitly exported can by used by modules. This directive is a simple macro that has, since the beginning, had a couple of annoying limitations, but nobody has ever gotten around to fixing them until now. Al Viro's patch set is a good opportunity to look at symbol exports and how they work.

The actual implementation of EXPORT_SYMBOL() can be found in include/linux/export.h. Whenever this macro is invoked, it declares a kernel_symbol structure for the exported symbol:

    struct kernel_symbol
    {
	unsigned long value;
	const char *name;
    };

As one might expect, the name field is set to the name of the symbol, while value becomes its address. When the code is compiled, though, this structure is not placed with the rest of the surrounding object code; instead, it goes into a special ELF section called __ksymtab (or __ksymtab_gpl for GPL-only symbols). The kernel binary contains these sections; any module that exports symbols of its own can also have them. When the kernel boots (or a module that exports symbols is loaded), that section is read and the symbol table is populated from the structures found there. The symbol table can then be used to satisfy references from modules to exported symbols.

In theory, the symbol-export mechanism limits the API available to loadable modules. There was once hope that this API could be kept to a relatively small and well-defined set. A quick grep in the kernel repository reveals, though, that there are currently over 27,000 exported symbols — not exactly a small set. When you have that many symbols, simply maintaining them all becomes a bit of a challenge.

One rule of thumb meant to help with the maintenance of exported symbols is that the actual EXPORT_SYMBOL() directive should appear next to the function or data structure that it exports. That allows the function and its export declaration to be modified together. This rule is not always observed, though. Sometimes it's just a matter of old code that predates the adoption of this rule but, more often, it is actually the result of a couple of limitations in the export mechanism:

• The macros found in <linux/export.h> are written in C and use GCC-specific extensions; they do not work in assembly-language code. So the export declarations for any functions written in assembly must appear in a separate, C-language file.

• Code that is built into a separate library prior to being linked into the kernel image has a potential surprise of its own. If nothing that is built into the main kernel image uses the exported object, the linker will leave it out of the build. Later, when a module is loaded needing that symbol, the load will fail because that symbol is not actually present. One way to work around this problem is to put the export declaration in code that's known to be built in — away from the object actually being exported.

Addressing these limitations is the goal of Al's patch set. Fixing the first one is relatively easy; it is mostly just a matter of writing a version of <linux/export.h> that uses the necessary assembler directives to create the kernel_symbol structures in the proper section. There are some details related to alignment requirements on some architectures, but they do not appear to have been that hard to get around. Once Al's patches are applied, assembly code can include <asm/export.h> and use EXPORT_SYMBOL() in the usual way.

The solution to the second problem is a bit of scripting trickery. As part of the build process, objdump is run on any library objects to obtain a list of exported symbols. A dummy object file (lib-ksyms.o) is then created with an explicit reference to each exported symbol; that object file is linked directly into the kernel. That will cause the linker to pull in all of the exported functions as well, ensuring that they will be available later on when a module is loaded. That eliminates an annoying trap that can spring on unsuspecting users years in the future when they happen on a configuration that fails to pull in objects they will need.

The bulk of the patch set is a set of cleanups enabled by the above changes; in particular, a lot of EXPORT_SYMBOL() and EXPORT_SYMBOL_GPL() declarations are moved into assembly code next to the objects they are exporting. In the process, Al found a number of dusty corners where unused functions could be removed; as he put it: "I'm fairly sure that more junk like that is out there; that's one of the reasons why exports really ought to be near the definitions."

Not all of those cleanups need to be merged anytime soon, though; they can happen anytime after the enabling patches go into the mainline. So that part of the patch set will likely be left in the hands of the specific architecture maintainers (assembly-language code, by its nature, is found in the architecture-specific parts of the kernel tree). The core changes are straightforward and uncontroversial; there is unlikely to be much keeping them out of the mainline. So, in the near future, one longstanding build-system annoyance should be history.

Index entries for this article
Kernel	Modules/Exported symbols

---

to post comments

Improving EXPORT_SYMBOL()

Posted Feb 4, 2016 16:04 UTC (Thu) by viro (subscriber, #7872) [Link] (1 responses)

For what it's worth, there turned out to be another interesting problem in EXPORT_SYMBOL() - while alignment issues are easy to handle, some of the architectures with function descriptors are trickier. That had been dealt with in the second iteration of the patchset, but it went out late on Wednesday, apparently too late for this article.

Usually a pointer to function is represented as the address of its entry point. However, on some architectures that's not the only thing required for calling an arbitrary function. Consider e.g. ppc64 - a register (r2) is dedicated to accessing variables local to an object file. As the result, the calling sequence includes "set r2 to the right value"; in case of local calls it doesn't require doing anything since r2 is already pointing where needed, but indirect calls absolutely must set it right. Which means that pointer to function needs to encode both where we jump and what we put into r2. Since a lot of code assumes that function pointers and data pointers are of the same size, going for 128bit function pointers would've been a bad idea and they introduced a small static structure associated with function. That structure (function descriptor) contains all information needed for a function call and pointer to function is actually an address of that structure.

Very similar things exist on ia64 and 64bit parisc; several MMU-less architectures (blackfin, frv, sh) also have such ABI variants (selected by -mfdpic, usually). However, frv, sh and blackfin kernels are built with -mno-fdpic, so while the kernel needs to be aware of userland code doing that kind of stuff, it's limited to dealing with signal handlers. ppc64, ia64 and parisc64, OTOH, have kernels built that way.

Suppose we have
int foo(void);
int bar;
and are exporting a function foo(). Initializing struct kernel_symbol needs the right thing stored in its ->value. And for our 3 architectures that's the address of function descriptor. That's where the things get nasty: in ppc64 assembler it will be spelled out as foo, in ia64 one - @fptr(foo) and in parisc - P%foo. ppc64 case is fine - it looks exactly as if no function descriptors existed, but both for itanic and parisc we are not that lucky.

The real trouble is that if next to our EXPORT_SYMBOL(foo) we have EXPORT_SYMBOL(bar), ia64 will need to put @fptr(foo) in one and bar in another.
Compiler can tell that foo is a function and bar isn't; assembler obviously can't, so we end up having to supply that information manually. Since there's a lot more function exports than data ones, I prefer to keep EXPORT_SYMBOL for the former and add EXPORT_DATA_SYMBOL for the latter; on normal architectures they expand to the same thing. but on ia64 and parisc we need to use the right one in assembler files.

Improving EXPORT_SYMBOL()

Posted Feb 5, 2016 3:27 UTC (Fri) by jschrod (subscriber, #1646) [Link]

Thanks for this comment (and thanks for your (comment) contributions to lwn.net :-)). Clear, understandable, and made me appreciate again the work of you and your peers on the kernel.

Comments like this (and the accompanying article ;-)) are one of the reasons why lwn.net is *THE* source of information about the Linux eco-system. Thank you again for participating. I, as a user of your code, appreciate it a lot; it allows us users to learn something.