The Novell Partner Linux Driver Process

[Posted May 17, 2006 by corbet]

Every so often, somebody shows up on the linux-kernel list with the same bright idea: separate the device drivers from the rest of the kernel and release them independently. Then drivers could be installed or updated without having to change the entire kernel. This idea never gets very far; among other things, it implies the creation of a stable driver API, which is just not in the plans. But the idea keeps coming back anyway.

When Novell's breathless press release, describing a "device driver breakthrough" which "solves Linux device driver compatibility issues," your editor's first thought was that this old idea had returned yet again. This breakthrough process "allows customers to obtain drivers independently of Novell kernel updates," after all, and is said to make life easier for vendors. As it turns out, however, Novell has no plans for defining any sort of stable kernel API; instead, it has created a mechanism making it easier for vendors to cope with the existing, dynamic API.

Essentially, a vendor with a driver for its hardware can approach Novell, pay whatever fee is required to become a "partner," and have its driver distributed through the SUSE YaST mechanism. If the partner supplies versions of the driver which work with distributed SUSE kernels, Novell will make sure that each user gets the right version. Novell will provide API change notifications, helping vendors to keep their drivers working with current kernels. If the vendor becomes an Extra Special partner, Novell will take care of much of the driver updating work themselves.

To some, this program looks for a way for Novell to help vendors who ship proprietary drivers. And there may be some truth to that view. But the real customer base may be elsewhere. Imagine that you are a vendor selling products into a highly competitive market. When your new widget comes out, you do the right thing and contribute a driver to the mainline tree. Even if the driver is accepted on the same day (a relatively unlikely course of events), it will not appear in a released kernel for a month or two, and it will not show up in released distributions for some months (or years) after that. By the time normal users can install the driver, the device is already obsolete and being replaced by something newer, shinier, and faster. And, in any case, having the driver in new distributions is of little help to customers who are running older kernels and don't want to change that.

The Novell program will make it easy for this vendor to make drivers available for the range of currently installed SUSE systems, without forcing a kernel upgrade on their customers. If the program is done right, it could change the landscape for the better: vendors would have an easier time supporting the range of distributor kernels, and users would get current drivers, even on older systems. If done wrong, it could lead to more out-of-tree drivers, but Novell appears to have anticipated that concern. From the driver partner FAQ:

As an active member of the open source community, Novell's position is clear: The best place for partners to develop kernel drivers is upstream in the kernel.org source tree, where kernel driver code benefits from thorough review and community involvement. Novell promotes having all Linux device drivers be a part of the official kernel.org source tree.

As long as vendors use this program as a backporting mechanism, it will do nothing but good for everybody involved. If they use it as a way to avoid the kernel development process or the need to release their code, the benefits will be rather less. The initial signs are good enough, however, that it is worth wishing Novell luck in this endeavor.

Index entries for this article
Kernel	Device drivers

The Novell Partner Linux Driver Process

Posted May 18, 2006 6:52 UTC (Thu) by pimlott (guest, #1535) [Link]

Unless I misunderstand the current situation, the best service Novell could provide would be to maintain backports of the current kernel API to older kernels. I've heard that many experienced driver maintainers do this themselves--create a translation layer so that they can write to the latest and greatest API (for submission to the mainline), but still build on older kernels (possibly with loss of functions that those kernels don't support). There is no central maintenance of these translation shims, though, so everyone hacks up his own. If Novell took this over, they could tell their hardware partners, "write to the current API and you'll get earlier kernels for free (and for Novell kernels, we guarantee it will work)"--freeing them from the ifdef hell that supporting many API revisions usually entails.

kernel API - a little bit less bias, please

Posted May 18, 2006 19:22 UTC (Thu) by wilck (guest, #29844) [Link] (7 responses)

I wish that LWN was a little bit less biased in the "stable module API" vs. "all drivers must be in-tree" discussion.

The LWN editor shares Greg KH's position. That's fine. But the way this opinion is expressed ("is just not in the plans", "you do the right thing ...") is not argumentative; it simply states that people who think differently are on the wrong side, end of discussion.

That is unfortunate, because I, for one, have never been convinced by Greg's arguments. In particular, most of the arguments in the part on "stable kernel source interfaces" can be shaken by realizing that e.g. for the USB module API, one could introduce a counter like #define USB_API_VERSION 123 and someone changing the USB API would simple increase that counter (well, a slightly more elaborate scheme might be necessary, you get the idea). That wouldn't strictly be a "stable API" but one that driver authors could easily adapt to.

This simple approach can hardly have escaped the highly intelligent Linux hackers' attention. Yet it is never mentioned. It appears that the developers don't want it, because a) they outright refuse to even think about backwards compatibility, and b) they want to make life tough for the despised developers of non-GPL modules - actually, for all external developers.

Greg's advice for vendors to push their code into mainline can turn out be an extremely tough ride. His praise of the "drivers-in-mainline" model ignores its disadvantages:

the kernel gets bloated with drivers that 99% of users never need,
small and exotic drivers can be easily forgotten when changes are made (it has happened that drivers didn't even compile),
driver development is slowed down by the necessity of getting changes accepted,
the mainline developers' claim to be able to maintain all drivers for every piece of hardare out there might be just a bit too big even for their giant boots.

That said, IMO Novell is targetting its own (SLES) users in the first place. Kernel updates, in particular, are a mess if external module packages are needed. AFAICS, the Novell driver process (which consists mainly of guidelines for packaging drivers in rpm's and how to distribute these rpm packages) is primarily meant as a remedy to that problem for SLES, for both GPL and non-GPL add-on drivers. If all works well, people will be able to use YOU for the kernel just like any other package, even if they need 3rd party drivers.

Novell mentions kABI in the FAQ, a concept that goes beyond a stable source-code API. That suggests that Novell actually wants to provide a stable API (or at least much less volatile than mainline) during the lifetime of SLES10. SUSE did the same for earlier SLES releases, although less explicitly, and RedHat does for RHEL as well.

I believe that Novell is on the right track. If the driver process proves to be good for other goals, as outlined in the article, great. Not to mention that if other distributors adopted the model, it would make it much easier for vendors to support Linux distributions, and would improve the driver availability for Linux in general.

kernel API - a little bit less bias, please

Posted May 18, 2006 21:23 UTC (Thu) by hazelsct (guest, #3659) [Link] (1 responses)

"I, for one, have never been convinced by Greg's arguments [supporting in-tree drivers]."

"Kernel updates, in particular, are a mess if external module packages are needed."

So if in-tree drivers are bad, and external module packages are bad, what exactly do you propose? A set of independent drivers full of #ifdefs?

Thanks, but no thanks!

kernel API - a little bit less bias, please

Posted May 19, 2006 8:05 UTC (Fri) by wilck (guest, #29844) [Link]

I am not saying that in-tree drivers are generally "bad", just that goes too far to demand that everything must be in-tree. I expect Novell's driver process to simplify the "mess" I was talking about considerably.

What I propose is: to realize that a stable or at least predictable API (on the source code level) is not such a bad thing as some people claim it to be.

Btw, in-tree drivers are no guarantee against #if's, you just don't see them anymore. Some drivers maintained by vendors are kept portable between kernels in their private development version, and the #if's necessary for portability that are just stripped before merging the driver into the kernel tree.

If a vendor wants to keep full-featured drivers for different upstream and vendor kernel series, #if's are hard to avoid. The alternative "vanilla + patches" approach of DKMS may be more sympathetic to the in-tree camp, but it is neither easier to handle nor easier to understand.

kernel API - a little bit less bias, please

Posted May 19, 2006 8:19 UTC (Fri) by eru (subscriber, #2753) [Link] (1 responses)

I, too find the arguments against a stable source level device interface unconvincing (while I fully agree that binary ABI is not desirable). Not wanting to stabilize the device interface made lots of sense when Linux was young, but now that it is a mainstream system, with lots of people wanting to use it with lots of different devices, refusal to provide a stable API and require inclusion of all devices in the kernel tree will simply not work for much longer. The number of devices for which open-source support is desirable and possible is simply too large for that now.

Just the kernel configuration program by itself is beginning to look like an encyclopedia of computer peripherals, and despite that many devices for which an open-source driver exists are missing, or the kernel version of the driver is out-dated.

There are people who may be willing and able to create a free driver, but are not interested in updating it forever merely because the kernel interface changes: Isn't it a pointless activity when the driver works and the hardware itself has not changed? (might even be out of production, but still be useful to support for older machines).

A carefully designed source API leaves a lot of flexibility with respect to the underlying implementation and extension possibilities. Provide "operators" (which can in reality be macroes or functions), "opaque" data types whenever possible, and rules about their use. To shake out inwarranted dependencies, you could even make a point of always putting some changes to the underlying implementation at each release, so that drivers that use the abstraction correctly are OK, but misusers break - this would also weed out attempts to use the source API as a binary ABI!

Incompatible changes cannot of course be always avoided, but they should be rare, and limited to times when a major version change occurs (like 2.6 to 2.8, if it ever happens).

A carefully designed source API

Posted May 25, 2006 15:05 UTC (Thu) by jimwelch (guest, #178) [Link]

>> A carefully designed source API

I still see too much churn in the drivers and kernel to lock down an API. I do not want to stagnate Linux drivers yet. I do that too much at work. True, I am looking from a birds eye view, not being a driver writer for Linux, just a home user. I've worked in the embedded world for 30 years where hardware is "stable" and supported for 10 years or more.

kernel API - a little bit less bias, please

Posted May 19, 2006 12:48 UTC (Fri) by corbet (editor, #1) [Link] (2 responses)

I'm sorry if you find the reporting "biased." It is true that I feel a writer's position should be clear, rather than hidden behind some sort of bogus show of neutrality. And my position, formed by years of digging through and messing with kernel code, is that trying to cast the internal ABI in cement would slow and destabilize the whole process.

BUT when I report that a stable internal ABI is "just not in the plans," I am being flat-out factual. It is not in the plans, and there is no serious discussion about changing that. How could I have expressed that better?

kernel API - a little bit less bias, please

Posted May 19, 2006 14:25 UTC (Fri) by wilck (guest, #29844) [Link] (1 responses)

I agree that a writer should state his position. Generally, I like LWN's way of discussing things.

To me, your article conveyed something like "Thy Driver Shall Be In the Tree and Shame on You Who Disagree", and I took it as a cause to dispute that statement.

I may have misinterpreted the tone of your article. I apologize if it is so.

kernel API - a little bit less bias, please

Posted May 19, 2006 21:10 UTC (Fri) by giraffedata (guest, #1954) [Link]

Maybe it's just because I'm familiar with this writer's style, but I did not take "you do the right thing and ..." to be advice from Jon, but rather his description of the position that the folks in control hold.

When LWN reports on a fresh controversy, it usually "argues" both sides in that same style.

The Novell Partner Linux Driver Process

Posted May 19, 2006 7:47 UTC (Fri) by tzafrir (subscriber, #11501) [Link] (1 responses)

It should be interesting to compare this "breakthrough" to the "kernel interface numbers" of Debian kernel packages numbers. I'll describe here the system of Debian Stable. I figure SuSE must have something similar as they have so many module sources packages.

The version name of the current kernel images of Debian Stable are 2.4.27-3-<configname> and 2.6.8-3-<configname> . <configname> is e.g. 386, 686 or k7-smp .

This is not the thir revision of the 2.4.27 kernel package or the 2.6.8 kernel package. Actually current revision number os something in the lines of 16.sarge2 . "3" here is a sort of "kernel ABI version number". It changes if a change in the kernel broke its interface to userspace or to external modules.

There are numerous "foo-source" packages from which you can build (normally using m-a: modules-assistant) a package called foo-modules-<kernelver> . That package include the modules as installed under /lib/modules/<kernelver> .

Thus as long as the ABI is not broken, old modules keep working. If someone noticed that a patch breaks certain userspace/modules, it may require bumping up the "kernel ABI version number". Now the new kernel will have a different version string (e.g: 2.6.8-4-386 instead of 2.6.8-3-386) . Existing module packages will simply not load. They will have to be rebuilt with the new kernel's headers.

So mechanics for "stable kernel ABI" for a distro are already there. Naturally actually testing that a new kernel update does not break all the existing devices requires some work. And work doesn't come free...

The Novell Partner Linux Driver Process

Posted May 19, 2006 14:09 UTC (Fri) by wilck (guest, #29844) [Link]

RedHat is handling ABI version numbers similarly (first number after kernel version denotes major changes, additional numbers are added to denote pure bug fixes that don't affect any interfaces).

Novell/SuSE have not handled kernel version numbers that consequently in the past, AFAICT.

The new Novell process, apart from their promise to reflect ABI changes properly, also warrants that update module packages (foo-modules-<kernelver> in your example) will be available from vendors for drivers that adhere to the model, so that no manual download and no recompilation will be necessary for users after a kernel update. To my understanding, vendors participating in the process must commit themselves to provide such packages in due time after kernel updates have been released.

Using Yast doesn't actually make sense

Posted May 21, 2006 8:09 UTC (Sun) by error27 (subscriber, #8346) [Link] (2 responses)

It doesn't actually make sense to use yast for drivers.

Let's say I worked for a RAID manufacturer. I sign up for the free email service. That's nice but it doesn't help me that much. I'm already in the beta program so I already know what new releases are coming out.

So let's say I become a full partner. Now I can just say, "Use yast to install the SCSI driver." But obviously, the customer is going to ask, "How can I use yast when I don't even have the RAID driver to install the blasted system?"

So hopefully SuSE is going to create driver disks for me I guess.

The customer completes the install then upgrades their kernel so now they can't boot. And I say, "You have to install the new driver using yast when you upgrade the kernel. Boot to the old kernel and take care of that."

But the thing is that yast doesn't normally save the old kernel after an upgrade, so the customer is righteously screwed. Since the system isn't bootable they decide to install Windows XP Server Edition instead.

It would be nice if SuSE set up a driver disk page though. That would be useful.

Using Yast doesn't actually make sense

Posted May 25, 2006 14:46 UTC (Thu) by niner (guest, #26151) [Link] (1 responses)

> The customer completes the install then upgrades their kernel so now they can't boot. And I say, "You have to install the new driver using yast when you upgrade the kernel. Boot to the old kernel and take care of that."

But when driver updates are available through yast online update, like the kernel is, the user would get both, the new kernel and the updated driver at the same time. So where's the problem?

Using Yast doesn't actually make sense

Posted May 25, 2006 20:41 UTC (Thu) by error27 (subscriber, #8346) [Link]

Under the old method you would would install with a driver disk. That means you don't have an foo_raid_driver.rpm. So when you upgrade the kernel there are no depends that will force you to upgrade the foo_raid_driver.rpm.

Maybe under the new process the driver disk will install a dummy rpm... I guess that would be amazing and wonderful...

The way I'm reading it, drivers would be in a seperate package from the base kernel. It's still an out of tree driver but it's distributed by SuSE. That's silliness obviously. It would make a lot more sense to include it in the stock SuSE kernel. That way you wouldn't have to worry about dummy RPMs or anything like that.