An update on the Android problem

By Jonathan Corbet
November 7, 2017

2017 Maintainers Summit

Android has been a great boon to the kernel community, having brought a great deal of growth in both the user and the development communities. But Android has also been a problem in that devices running it ship with kernels containing large amounts (often millions of lines) of out-of-tree code. That fragments the development community and makes it impossible to run mainline kernels on this hardware. The problematic side of Android was discussed at the 2017 Maintainer Summit; the picture that resulted is surprisingly optimistic.

Greg Kroah-Hartman started by saying that he has been working for some time with the system-on-chip (SoC) vendors to try to resolve this problem, which he blames primarily on Qualcomm for having decided not to work upstream. Qualcomm has since concluded that this decision was a mistake and is determined to fix it, but the process of doing so will take years. The other SoC vendors are also committed to closing the gap between the kernels they provide and the mainline but, again, getting there will take a while.

Google's new rules requiring the use of long-term support kernels with Android and keeping up with updates should also help. If vendors do not follow those rules, he said, he will eventually stop maintaining the LTS releases. For now, though, he is running an experiment where he will support the 4.4.x kernels for a period of six years. Vendors are coming around to using those updates, he said, but there is a new problem in the form of carriers who are proving unwilling to ship those updates. He is trying to get carriers to put one out every six months for now.

Rom Lemarchand, Google's Android kernel manager, said that newer devices are shipping with 4.4 kernels now. The SoC market cycle is such that these chips will always run a two-year-old kernel. The two-year support lifetime for LTS kernels thus didn't work well for SoC vendors; just about the time that they ship something, the support goes away. Hopefully the six-year support period will work better. Updates are still a problem, though; vendors still are working under the mentality that they only need to take patches that have CVE numbers attached to them, which is not the case. Kroah-Hartman added that they weren't even taking all of the patches with CVE numbers. Kees Cook said that none of the vendors have decent testing for their kernels and don't want to merge any changes at all. They don't, he said, want to admit that they are bringing in LTS patches.

Along the lines of testing, there was some discussion of the Linux Test Project (LTP). This project has tended to be viewed dismissively by kernel developers, but it is evidently the recipient of more resources and has been getting better. There may eventually be value in integrating LTP into the kernel self tests. Linus Torvalds said that even an improved LTP is not that interesting compared to real workloads, though, so he would much rather see Android running on mainline kernels. This is evidently being worked on, but is not there yet. Lemarchand said that the HiKey boards are staying as close to mainline and can boot a 4.9 kernel, but Arnd Bergmann pointed out that the HiKey boards are no longer being produced.

Somebody asked: has any Android phone ever done a major kernel upgrade after it has been shipped? That is evidently a difficult proposition, since there a number of regulatory certifications that must be redone. But the Galaxy Nexus and Galaxy S phones both saw major kernel upgrades, so it is possible. Torvalds noted that there are a lot of Android devices that are not phones, tablets for example, that might prove to be better development devices. It would be nice if mainline developers could run their own kernels on real devices. Bergmann said that the gap is shrinking on some devices, and Kroah-Hartman repeated that he is working toward this goal with the SoC vendors, but the process should be expected to take about six years.

Cook said that applying the larger updates involved in following the LTS kernels completely should eventually make vendors more comfortable with larger kernel changes in general. Sean Paul said that running mainline kernels on Android devices may well become possible soon, but phones still probably will not jump to new major releases. Even that would be good, though, Bergmann said; the current out-of-tree code problem defeats the goal of building a single ARM kernel for all devices. Fixing that would enable third-party distributors to ship systems for multiple phones. Torvalds said that, even if vendors don't upgrade their devices, the ability to do so would enable some useful regression testing. James Bottomley said that the whole situation is a repeat of the enterprise Linux problem from many years ago.

Ted Ts'o asked if there were any ARM Chromebooks that could be used as development machines; Paul answered that the ones based on Rockchip SoCs were close. Torvalds asked about the status of the Mali GPU driver; Bergmann responded that there had been one person working on reverse-engineering that device, but he didn't work well with other developers. Now somebody else is making progress with the older GPUs, but nobody is working on current-generation devices. It was said that everybody within ARM is in favor of solving the problem by open-sourcing ARM's driver — except for one recalcitrant high-level manager.

Torvalds said that, if the Mali problem could be solved, the community as a whole would be in good shape. Bergmann said that there are currently four ARM GPUs with good free-software support, but they are all older. Going forward, Mali seems to be the GPU of choice for Android devices, so that is the problem that needs to be solved. Lemarchand said that pressure is being applied from the Android side as well.

The final conclusion of this session was that, while the Android problem has not gone away, the situation is far better than it was one year ago.

[Your editor would like to thank the Linux Foundation, LWN's travel sponsor, for supporting his travel to this event].

Index entries for this article
Kernel	Android
Conference	Kernel Maintainers Summit/2017

to post comments

Carriers

Posted Nov 7, 2017 9:39 UTC (Tue) by epa (subscriber, #39769) [Link] (12 responses)

I don't understand why carriers have to get involved in shipping updated kernels. Surely the device can connect to Google's update servers, see that a new version is available, and then prompt the user to upgrade? What is the phone company's involvement in this?

Carriers

Posted Nov 7, 2017 9:57 UTC (Tue) by gby (guest, #23264) [Link] (3 responses)

In many countries the prime channels to get smartphones to the end of customers is by said customers purchasing a carrier branded, often subsidized and sometime customized phone. The carriers must OK each software update to such devices.

Carriers

Posted Nov 7, 2017 11:28 UTC (Tue) by epa (subscriber, #39769) [Link] (2 responses)

Hmm, but the carriers don't have to OK each software update to iPhones, surely? Even if the iPhone was subsidized and locked to that carrier. And if it can be done for iOS it can be done for Android.

Carriers

Posted Nov 7, 2017 13:21 UTC (Tue) by nhippi (subscriber, #34640) [Link] (1 responses)

The carriers go "We'd like to order x million phones, but we want brand/customize them". Apple can say "No, we don't customize", and carriers will order iphones anyways, as else customers who want apple phones will go to other carriers. Android makers can't afford to say no, the carrier will simply order android phones from another vendor.

It wouldn't be so bad, if the branding/customization were just .apk:installed on top of Generic Android. But now the whole stack is compiled into one image that must be updated and tested as whole. Project Treble is the first step google is taking to get out of the mess.

Carriers

Posted Nov 10, 2017 8:47 UTC (Fri) by marcH (subscriber, #57642) [Link]

While iOS customization is certainly nowhere near Android's, I was recently surprised to see the following popup on a friend's iPhone: "To enable tethering on this account, contact AT&T at 611 or visit...

Carriers

Posted Nov 7, 2017 17:13 UTC (Tue) by nim-nim (subscriber, #34454) [Link] (5 responses)

Connecting a phone to a cellular network requires a handshake and downloading network-specific parameters. The same mechanism is used for firmware updates, the phone asks the network 'is there an update for me, I am model X with firmware Y', and if the network infra answers yes and you accept the update the firmware is downloaded from network distribution nodes without being billed as 'internet access'.

That means carriers can block the distribution of updates even after they've been made available by the manufacturer, if they feel they are not ready, fear the result is incompatible with their infra, or are just lazy. That's why people with the same model, in the same country, can see different updates depending on their carrier.

I don't see carriers giving up this power easily. A bad update would cause lots of problems for them, especially if it hits a large number of phones at once. Trying to shortcut carriers with direct internet updates, would just result in carrier refusal to accept connexions from corresponding phones.

Just because carriers are very lax, on phones that received an update on another network, does not mean they are not ready to exert mass blocking, if it gets out of hands. Carriers trust the choices of other carriers. The number of devices that plug in their network after receiving updates somewhere else is necessarily limited (making their users good guinea pigs to shake out eventual incompatibilities). They do not trust you or any internet update mechanism, that could result on mass firmware updating of existing customer phones, outside their control. If there is an incompatibility customers would blame them. Right now if you roam inside a network, with an update the carrier didn't approve, and it does not work, you blame the roaming.

Carriers

Posted Nov 7, 2017 20:13 UTC (Tue) by jem (subscriber, #24231) [Link] (3 responses)

A thought experiment: replace each occurrence of "carrier" and "phone firmware update" with "internet service provider" and "PC operating system update". Why do these have to be different? In large parts of the world there is no difference.

Carriers

Posted Nov 7, 2017 22:20 UTC (Tue) by smurf (subscriber, #17840) [Link] (2 responses)

Because in large swathes of the world bandwidth is still rather expensive.
The obvious solution is updating via WLAN … but that idea doesn't fit the typical carriers' mindset.

Carriers

Posted Nov 8, 2017 7:40 UTC (Wed) by gioele (subscriber, #61675) [Link] (1 responses)

> Because in large swathes of the world bandwidth is still rather expensive.

Also fixed-line bandwidth can be expensive (think Australia or rural EU). This does not stop Windows from downloading gigabytes of updates.

> The obvious solution is updating via WLAN … but that idea doesn't fit the typical carriers' mindset.

In my own experience, EU carriers makes you pay for the bandwidth used for the updates (and this is how it should be, otherwise it would be against net neutrality principles). For this reason, everybody I know updates their phones only via WiFi.

Carriers

Posted Nov 17, 2017 22:34 UTC (Fri) by JanC_ (guest, #34940) [Link]

I don't see how making phone update traffic free would be against net neutrality, provided this is true for all "phone firmware distributions". Even better, expand that to (security) updates of firmwares & operating systems of all internet-connected devices.

Carriers

Posted Nov 10, 2017 17:02 UTC (Fri) by marcH (subscriber, #57642) [Link]

> The same mechanism is used for firmware updates, the phone asks the network 'is there an update for me, I am model X with firmware Y', and if the network infra answers yes and you accept the update the firmware is downloaded from network distribution nodes without being billed as 'internet access'.

I really don't think this is happening when you don't buy the phone from a telco. By the way most phones recommend you switch to wifi before accepting to download.

> That means carriers can block the distribution of updates even after they've been made available by the manufacturer, [...] Trying to shortcut carriers with direct internet updates, would just result in carrier refusal to accept connexions from corresponding phones.

Again this is only for phones bought from the telco (which is very rare in some countries)

> I don't see carriers giving up this power easily. A bad update would cause lots of problems for them, especially if it hits a large number of phones at once

Updates from Google are always staggered (Apple may be different)

Carriers

Posted Nov 9, 2017 18:57 UTC (Thu) by jreck (guest, #96413) [Link] (1 responses)

Typically consumers buy phones through or for a carrier to some degree. Then when they have problem with said phone, they call the carrier for support. Updates always cause new problems, which means updates cause new volume of support calls, which means updates cost carriers lots of money and makes some percent of their users unhappy.

So updates:
Make X% of population happier (good press, less bad press, positive karma, whatever); which brings in $X' in future revenue from people now buying from said carrier
Make Y% of population unhappy (something changed! you broke it! etc...); which costs $Y' in future lost revenue from people leaving said carrier
Cost $Z on support calls, bandwidth, and repairs/replacements.

If (Y' + Z) > X', then updates are simply a net-loss for the carrier.

Why is this different from laptops/desktops/whatever? Well quite simply you don't buy your computer from whoever your ISP is, so you don't call them when some OS update results in your PC no longer booting. You call Apple. You call Dell. You call Best Buy. Whatever. And they all have support plans and similar to recoup those costs by charging for being the support center you call.

Carriers

Posted Nov 9, 2017 19:52 UTC (Thu) by jem (subscriber, #24231) [Link]

> Typically consumers buy phones through or for a carrier to some degree.

Typically in your country, maybe. I have never bought a phone from a carrier. I have received my updates from Nokia and Motorola directly.

The whole point of SIM cards is (or at least was originally) that you get your telephone subscription from the carrier, and buy a handset from whichever retailer you want. Put in the SIM card and go. When you want a new phone, just buy a new one. When a competing operator has a better deal for you, toss out the old SIM card and put in the new one you got from your new operator.

The Carriers Problem

Posted Nov 7, 2017 9:55 UTC (Tue) by darwish (guest, #102479) [Link] (2 responses)

Steve Jobs was right, he mandated that Apple controls all software on the device as a hard condition to make it compatible with the Carriers networks. This has really bode well for iPhone devices so far, with most of devices always up-to-date.

One day Google or other Android manufactures should do the same.

The Carriers Problem

Posted Nov 7, 2017 10:35 UTC (Tue) by moltonel (subscriber, #45207) [Link]

Except if Google had tried being as controling as Apple when it brought Android to the limelight, there'd have been a lot more pushback. Android probably wouldn't be the dominant platform today and it's ecosystem wouldn't be as diverse.

The Carriers Problem

Posted Nov 7, 2017 16:34 UTC (Tue) by drag (guest, #31333) [Link]

> One day Google or other Android manufactures should do the same.

Google and Android is more successful then Apple. Apple has their high priced niche in the West and are quite satisfied to sit and roll in the profits there while Android powers the devices for the rest of humanity.

Mali

Posted Nov 7, 2017 11:38 UTC (Tue) by phh (guest, #112196) [Link] (3 responses)

The mali question has different facets.

At the moment, if you are:
- On a supported SoC (Rockchip, Allwinner (thanks to free-electron))
- On mainline kernel + dtb 4.15 (or was it merged on 4.14?)

You'll be able to build a mali.ko (this is fully opensource), and you'll need a userspace libmali.so which does libEGL/libGLES/libvulkan/... as one blob.

The mali.ko driver and libmali.so are currently version-dependent, you're supposed to use the same version.
The mali.ko/libmali.so are platform independent, so if you've got binaries for Mali-400, it should work on all Mali-400 platforms.
(Note that there are hardware revisions of Mali, and drivers are hardware-revision-specific...)

This driver is compatible with modern tools, like dma-buf, gbm, kms, but is not based on Mesa, or DRM

So at the moment, installing mali driver to a mainline kernel is as easy as installing a nVidia driver.
If it is supported by the distribution and everything goes well, then it will just work, so that's pretty good imo.

To recap, the current caveats:
- libmali.so is environment-specific: there is one for fbdev, one for X, one for gbm (well gbm version should be able to rule them all)
- libmali.so and mali.ko are version-tied, so mali.ko can't really be mainlined
- mali.ko will be broken when kernel API changes (because it is out-of-tree)

Now, if the driver gets open-sourced... None of those problems are gone!

Mali

Posted Nov 7, 2017 14:41 UTC (Tue) by TheLessThanAmazing (guest, #119480) [Link] (1 responses)

> Now, if the driver gets open-sourced... None of those problems are gone!

I'm confused.

Mali

Posted Nov 7, 2017 17:02 UTC (Tue) by k3ninho (subscriber, #50375) [Link]

>> - libmali.so and mali.ko are version-tied, so mali.ko can't really be mainlined
>> ...
>> Now, if the driver gets open-sourced... None of those problems are gone!
>I'm confused.
Yeah, I'd like to know how the versioned shim, when open-sourced, isn't going to enable a generic free-software shim. Any hint?

(I say this as someone who's an utter fan of magic numbers for -- eg hashes of -- published interfaces, mostly because any magic number and its compatible interface should be interchangeable with anything else which meets the interface and has the magic number. I don't see how that can't work here -- help me out?)

K3n.

Mali

Posted Nov 10, 2017 22:25 UTC (Fri) by edeloget (subscriber, #88392) [Link]

I haven't checked but can't you also have any kind of libc issue? In the embedded world there is a lot of libc fragmentation (multiple versions of glibc, musl, bionic... are in use). If libmali.so is bound to a particular glibc ABI, it won't help you much on a musl-based system.

An update on the Android problem

Posted Nov 7, 2017 12:52 UTC (Tue) by toddpoynor (guest, #19973) [Link] (1 responses)

Galaxy Nexus was the first of the Nexus / Google-partnered devices to not receive a kernel upgrade, I believe (the Motorola Xoom tablet was probably the last to do so). Galaxy Nexus was largely a victim of the SoC vendor trying to do the right thing: TI engaged the kernel community to upstream much of the SoC support code, plus the remoteproc framework and other stuff written as part of that project. But the resulting large differences in code bases between the tested shipping code and the upstream code, and the likelihood of requiring a substantial effort to get the upstream version to product quality, caused a kernel upgrade to be deemed not worth the effort. (I believe the XDA folks wrestled with this for their firmware and ended up reverting a bunch of stuff done upstream.)

This situation reportedly continues to be the case for most if not all major SoC vendors: the internal trees used to produce products differ heavily from the code that undergoes community back-and-forth and gains acceptance upstream. The two codebases just can't be allowed to diverge very far, and that requires an ongoing level of investment that can be difficult to justify.

An update on the Android problem

Posted Nov 7, 2017 20:46 UTC (Tue) by drag (guest, #31333) [Link]

This sort of thing is why Dell created DKMS.

They discovered that it's a huge pain in the rear to work on out of tree patches on older kernels and then forward port them to upstream. However it is relatively easy to go the other direction... work on upstream kernels and then back port drivers to older kernels.

Therefore if you are working on getting new SoC support done for a kernel it is then probably a bad idea to work on it in private on a 'Long Term Release' kernel then try to release patches to upstream. The good idea is to work on the patching the latest kernels, getting that to work first, and then backporting the changes to whatever kernel you have to ship with.

An update on the Android problem

Posted Nov 9, 2017 4:10 UTC (Thu) by jstultz (subscriber, #212) [Link]

Regarding the line "HiKey boards are no longer being produced." I don't believe that's accurate.
The HiKey boards are still in stock on Amazon and my understanding was another batch was being built.

An update on the Android problem

Posted Nov 10, 2017 4:55 UTC (Fri) by pabs (subscriber, #43278) [Link] (2 responses)

Re Mali, there are two recent efforts I'm aware of:

https://github.com/yuq/mesa-lima https://github.com/yuq/linux-lima

https://notabug.org/cafe/chai

An update on the Android problem

Posted Nov 17, 2017 19:02 UTC (Fri) by pboddie (guest, #50784) [Link] (1 responses)

It is interesting to read the related part of the article:

Torvalds asked about the status of the Mali GPU driver

This I interpret as the usual "are the peasants unhappy?" from the upper echelons.

Bergmann responded that there had been one person working on reverse-engineering that device, but he didn't work well with other developers.

I don't have any opinion on people not working well with others, but the story I've heard has a lot more to it than that, very little of it making various companies look very good. But then again, those two GitHub repositories perhaps indicate what narrowmindedness and destructive behaviour can bring: the involvement of another corporation with a more established presence in the sector in question; one who is less likely to be intimidated by random threats about "intellectual property concerns".

An update on the Android problem

Posted Nov 21, 2017 17:46 UTC (Tue) by flussence (guest, #85566) [Link]

Intel and nVidia may have an awful FOSS track record, but I have never heard comparable horror stories about them (or some cult of personality surrounding them) actively trying to cause ruin to people writing libre drivers for their chips…

Something is disturbingly wrong with that other corner of the hardware industry, and all those perpetrating it need to be dragged out into the light kicking and screaming sooner or later.

An update on the Android problem

Posted Nov 10, 2017 17:11 UTC (Fri) by marcH (subscriber, #57642) [Link]

> Greg Kroah-Hartman started by saying that he has been working for some time with the system-on-chip (SoC) vendors to try to resolve this problem, which he blames primarily on Qualcomm for having decided not to work upstream [...] Ted Ts'o asked if there were any ARM Chromebooks that could be used as development machines;

I understand Android implies ARM most of the time, however these are still different from each other. One is software and the other hardware!

For instance Intel Chromebooks boot upstream kernels without any major issue/regression. Yet the Android container inside doesn't boot with an upstream kernel (In some cases/versions the Android container may even bring the *entire system* down: lock up, or reboot loop,...) *This* is an actual "Android problem" which has nothing to do with Qualcomm or any other ARM vendor.