KS2008: When should drivers be merged?

By Jonathan Corbet
September 16, 2008

LWN's 2008 Kernel Summit coverage

The rough consensus in the kernel development community over the last couple of years has held that device drivers should be merged into the mainline as soon as possible. Even if these drivers have significant problems, it is better to get them into the mainline where they are more likely to be fixed. This approach was reconsidered at a Kernel Summit 2008 session, but the group left the policy essentially unchanged.

There are two fundamental lines of thought on this subject. James Bottomley started off the session with his feeling that the time before merging presents the best opportunity to get driver authors to improve their work. The possibility of merging the code provides a motivational incentive which vanishes once the code goes in. So James likes to hold code submissions out of the mainline until the worst problems have been addressed.

On the other hand, Arjan van de Ven doesn't like the idea of "holding code hostage" in this way. From this point of view, about the only reason to hold drivers out of the mainline is obvious security or user-space API problems. In the absence of those, getting the code merged into the mainline, where it will be more accessible for others to fix, is the best way to improve bad drivers.

Linus is clearly in the second camp. Drivers which are out of the mainline tree, he says, simply do not get better. People just do not spend much time looking at out-of-tree code. Additionally, not accepting drivers from vendors may put us into a position of having no real traction with those vendors; each of their subsequent drivers will have the same problems. By getting those drivers into the tree and fixing them, we may be able to push them toward producing better code. Otherwise, says Linus, we may be "shooting ourselves in the foot."

On the other hand, Greg Kroah-Hartman reported some strong successes with his linux-staging tree. That tree currently hosts some 15 drivers, most of which are steadily improving over time. Being in linux-staging is apparently enough to draw some attention to a driver, and that helps to get it into better shape.

Much of the discussion was devoted to an attempt to set a line dividing drivers which can be merged from those which cannot. There was not a whole lot of success, though. It really appears to be a case-by-case sort of problem. For example, what about one vendor driver which reads a configuration file directly from /etc? Such behavior is normally frowned upon. But, if the driver is already out there and being used, putting it into the mainline will not make things worse - we already have the problem. So, especially when the driver is already in widespread use, we might as well just merge it.

Some ways of mitigating problems with drivers were discussed. Some of the worst behaviors could be configured out, allowing the merging of a barely functional driver which can then be improved in place. Really nasty drivers can set a taint bit in the kernel as a warning to developers trying to track down bugs on the affected systems. Another idea involves outfitting badly-written drivers with strong warnings to keep other developers from copying the code found therein.

It was suggested that the distributors could ship drivers from the linux-staging tree, perhaps with the taint feature added. The answer to that was that, if the drivers are being shipped by distributors, they might as well be in the mainline. Linus stated that anything the distributors ship should really be merged as well. There are practical difficulties, though; Fedora ships the Nouveau driver, which still has not committed to a stable user-space API. Until that API stabilizes, Nouveau cannot be merged into the mainline, but there is still value in getting the driver tested by Fedora users.

There were a few conclusions from the discussion. The taint flag for substandard drivers will probably be added. There might be a drivers/staging directory for such drivers as well. Greg will take responsibility for getting some of those linux-staging drivers into the mainline; he has, it was suggested, just become the official crap maintainer.

Index entries for this article
Kernel	Development model/Driver merging

KS2008: When should drivers be merged?

Posted Sep 16, 2008 15:13 UTC (Tue) by sbergman27 (guest, #10767) [Link] (7 responses)

"""
Greg will take responsibility for getting some of those linux-staging drivers into the mainline; he has, it was suggested, just become the official crap maintainer.
"""

One common complaint about Linux is its lack of support for newer hardware. Never mind that it supports every ISA card ever sold. What people care about is the new stuff that they can buy *now*. It seems to me that the kernel devs consider new drivers, and by extension, the skills of the programmers writing them, to be crap. I'm not sure that attitude will be good for Linux in the long term.

But determining to try to get the stuff into the mainline tree is a positive action, regardless.

New drivers

Posted Sep 16, 2008 15:26 UTC (Tue) by corbet (editor, #1) [Link]

I don't think I wrote anything to give that impression. There is no such general feeling. It is true, though, that some new drivers are poorly written. That's what the discussion was about; nobody worries about the well-written drivers, dozens of which go into every kernel release.

KS2008: When should drivers be merged?

Posted Sep 16, 2008 19:31 UTC (Tue) by tialaramex (subscriber, #21167) [Link] (1 responses)

Most in-house stuff is crap. Never designed to be looked at by the general public, still less criticised by dozens of expert kernel developers. It is very noticeable, for example, that an in-house program which I got cleared to be released as GPL, is still sat there waiting for me to take action because I realised after getting the OK to release it that it needs at least a couple of days of extra work doing to it before outsiders will be able to use it sensibly. It mistakenly relies on an extension to a standard that it implements, and (a colleague pointed out when I enquired) nobody except us implements that extension. Whoops. From any outsider's point of that program is currently crap, so I won't be releasing it until it's not crap.

Developing the drivers for yet another brand of 100Mbit network card is not most people's dream job, so it should be no surprise that the developers working for a lot of hardware outfits are not the best in the business. That doesn't mean that /they/ are crap, but it does mean that their code isn't necessarily going to be up to the standard people expect from Linux.

Old in-house drivers weren't any better, I've seen the code that made the Video Toaster go. It's awful, you'd never have guessed that the Video Toaster was a roaring success if you tried to judge based on the quality of source code.

But actually today's drivers have even more opportunities to be terrible than did those from the Video Toaster era. You can screw up suspend and resume, you can fail when there are multiple CPU cores, there are countless novel ways to screw up. Microsoft has a QA process for new drivers, which must be passed before they'll certify the drivers for use with ordinary Windows PCs. Perhaps their QA process doesn't use the word "crap" but the judgement is the same, "this driver isn't good enough, try harder next time".

KS2008: When should drivers be merged?

Posted Sep 19, 2008 23:43 UTC (Fri) by giraffedata (guest, #1954) [Link]

Much of the crappiness being referred to here is not general crappiness that you might get from an incompetent or rushed engineer -- its just failure to conform to standards of the Linux kernel. It could be a different coding style. It could be stuff that was designed to meet requirements unrelated to the requirements of mainline kernels (e.g. the code was ported from Windows). It could be code that doesn't work in some of the myriad scenarios it might find itself in, in which it was never needed to work before.

KS2008: When should drivers be merged?

Posted Sep 17, 2008 17:10 UTC (Wed) by iabervon (subscriber, #722) [Link] (3 responses)

There's the sort of fundamental problem that vendors don't want to announce unreleased hardware, let alone publish drivers for it, and then they obsolete the hardware sooner than the latency in keeping a kernel release into users' hands. They're doing their release cycle based on the idea that they'll ship drivers, which they write internally and which don't have to be maintainable (i.e., only have to work with one or two already-released OS versions), with their hardware.

The solution is probably for them to design hardware that functions okay with drivers that aren't specifically for that hardware release, such that Linux users get hardware that works fine today and performs better over time (as newer drivers notice more hardware features and use them).

KS2008: When should drivers be merged?

Posted Sep 17, 2008 19:53 UTC (Wed) by mjthayer (guest, #39183) [Link] (2 responses)

Perhaps this would be a case to ship the vendor's drivers "until something better is available", probably disabled by default in the main kernel tree. Or alternatively, to have a private mailing list with relevant kernel drivers that they can submit to, so that their code will not be made public before they are ready.

KS2008: When should drivers be merged?

Posted Sep 17, 2008 22:16 UTC (Wed) by iabervon (subscriber, #722) [Link] (1 responses)

Releasing vendor drivers wouldn't really help much, since it's pretty late by the time users get anything released by the kernel team. Getting drivers for pre-release hardware to kernel reviewers under NDA could be a big help, though, particularly if that included distro representatives who could get a package for the module through QA and ready to release simultaneously with the hardware. I wonder if big hardware vendors would be interested in the linux driver project that Greg KH runs, for the express purpose of collecting acks from reviewers (and, of course, comments, but those would presumably be uninteresting before they could be posted) in advance of the code going out to the world.

KS2008: When should drivers be merged?

Posted Sep 19, 2008 1:27 UTC (Fri) by njs (subscriber, #40338) [Link]

IIRC it's common for (clueful) vendors to provide docs under NDA with the actual release embargoed until the hardware is announced. Or there are weird tricks, like drivers landing in the mainline kernel under code names, and then when the press release goes out so does the rename-all-the-symbols-in-that-driver patch.

What about when the author is disinclined to merge?

Posted Sep 17, 2008 8:15 UTC (Wed) by chrispe (subscriber, #4442) [Link]

I rely on 2 long-standing unmerged drivers on a daily basis...

yaffs2 (http://www.yaffs.net/) is an excellent NAND flash filesystem with all necessary features and acceptable mount times for large devices. It is widely used in embedded systems shipping today. A barrier to merging may be the existence of a compatibility layer allowing the filesystem to compile on other platforms and bare metal (eg: RTOS's & bootloaders).

drbd (http://www.drbd.org/) is a distributed block device driver used in high-availability systems and an integral part of the Linux-HA architecture. A barrier to merging may be existing features in the main line with partially overlapping goals.

I've asked on the mailing lists of both these projects when they will be merged quite some time ago and got vague promises, but no visible effort towards inclusion. They are both backed by small companies which may be unwilling to give up control. They may simply have other priorities.

I'd like to see a way for these sorts of projects to get merged. Perhaps some body could be persuaded to provide the authors with a financial incentive to get the merge work done.

Both of the examples above are large bodies of work which I'm very glad exist in any state of merging. Given the effort and generosity of their authors over a long period, I wouldn't like to see the projects forked for inclusion.