Android gets a toybox

By Jake Edge
January 14, 2015

The toybox project has been around for a while, and now it looks to be achieving one of its longtime goals: inclusion into Android. Toybox is a replacement for many Linux command-line utilities that is targeted at embedded systems—similar to BusyBox. It was created by Rob Landley, who is also a former BusyBox maintainer, and has a BSD license rather than BusyBox's GPLv2—something that was likely important for the notoriously GPL-averse Android project.

Landley started toybox in 2006 under the GPLv2, but soon switched to the BSD license. The project was largely dormant until he restarted development in 2011 with the goal of being a drop-in replacement for Android's toolbox that, like BusyBox and toybox, encompasses multiple command-line utilities in a single binary.

Toolbox, though, has long been a source of frustration for Android developers, some of whom replace it with BusyBox as one of their first acts on any new Android system. Developers have found that toolbox lacks many of the commands they need, while also providing commands with different behavior and command-line options than they are used to from Linux.

Landley released toybox 0.5.1 on November 19. That release is a bug fix update to 0.5.0 that we looked at in October.

Prior to that, though, Jason Spiro filed a bug in the Android Open Source Project (AOSP) bug tracker asking that the project incorporate toybox. There was some back-and-forth in the bug comments between Bionic (Android's C library) maintainer Elliott Hughes and Landley that resulted in a toybox that would build for Android. After that, toybox was added to the AOSP Git repository.

But there is more than just adding the code. Hughes posted a status report on December 18 to the toybox mailing list (which was updated on January 1) that listed various utilities in the AOSP master branch that use toybox as the underlying binary. The list has quite a number of new commands that were not available in toolbox, as well as more than two dozen commands where the toybox version replaces the one in toolbox. There are also some notes for commands that need to be looked at or fixed in toybox before Android can switch to using them.

For example, Hughes lists missing command-line arguments for several commands (e.g. cat, cmp, grep), but patches are pending for many of those. Other commands, such as ifconfig and inotifyd, appear to provide a superset of the functionality required by Android, but more testing needs to be done.

There are a couple of advantages that toybox brings to the table. The utilities work more like the GNU versions, which should be more familiar to Linux users than the NetBSD-based versions that are in toolbox. It also neatly avoids the GPL that comes with both the GNU utilities and BusyBox, which is frowned upon (at least) for Android user space.

In fact, the license difference is one reason that another embedded Linux distribution, Tizen, is looking at toybox. It currently uses the GNU coreutils package that is GPLv3-licensed. The anti-Tivo-ization language in that version of the GPL evidently worries some device manufacturers that have a "managed platform" (e.g. phones, TVs, in-vehicle infotainment or IVI). Tizen test systems have been built to use toybox instead without any problems, as reported on the wiki page, but it appears that more testing needs to be done.

Landley's larger vision is to use stock Android phones as a development platform by adding USB devices for keyboard, mouse, and display. There are hurdles, however, since Android kernels tend to only support USB gadget mode—plugging in a hub with input and output devices does not work on stock Android devices. He also described his plans, with links to a talk he gave about them, in a comment on the AOSP bug. It's not at all clear that Google (or even some Google developers such as Hughes) share those goals, but it is clear that Android will be moving away from toolbox and toward toybox in coming releases. That certainly gives the project a nice boost.

Shortly after the project restarted, it came under fire from Matthew Garrett and others because it was perceived as providing a way to avoid the license-compliance efforts that surround BusyBox. While Google is not circumventing the GPL in its work—other than the kernel, it tries to avoid the license entirely—the Android downstreams have sometimes played fast and loose with the GPL-licensed kernel (and possibly other GPL code). But other devices that currently ship BusyBox could switch to toybox to avoid compliance efforts based on BusyBox, which is Garrett's complaint.

The BusyBox compliance front has been quiet lately, at least visibly, so the Software Freedom Conservancy (SFC) (which has been leading those efforts) may have turned its attention elsewhere. There may also be ongoing compliance efforts that have not been made public, which is the norm unless a lawsuit is filed. The SFC also has a compliance project for the Linux kernel that could be used against vendors that are not publishing their kernel source as required by the GPL. Once again, though, any kernel compliance work from that project is proceeding quietly at this point.

Landley has been a vocal opponent of the BusyBox lawsuits since he ceased being one of the plaintiffs in those suits. So any reduction in those kinds of actions due to the adoption of toybox will likely be a satisfying outcome for him. In any case, more free code seeing wider adoption is certainly a good thing for the community. It may mean that certain kinds of GPL enforcement will need to change, but enforcing the license on the actual code of interest, rather than using BusyBox as a proxy, is likely to work out better in the long run.

to post comments

Android gets a toybox

Posted Jan 15, 2015 17:36 UTC (Thu) by flussence (guest, #85566) [Link] (1 responses)

Nice to see another project run by nice people gaining popularity at the expense of a nasty one.

Now if only Gentoo would let me *uninstall* busybox, I'd be busy switching all my systems over...

Android gets a toybox

Posted Jan 15, 2015 22:13 UTC (Thu) by ABCD (subscriber, #53650) [Link]

> Now if only Gentoo would let me *uninstall* busybox, I'd be busy switching all my systems over...

You can uninstall busybox in Gentoo, it just takes a bit more effort. Specifically, you can create a file named "/etc/portage/profile/packages" with the contents "-*sys-apps/busybox", which will remove sys-apps/busybox from @system. At that point, if nothing else installed pulls it in (and it's not likely that anything would), you can just do "emerge --depclean sys-apps/busybox" to get rid of it.

Android gets a toybox

Posted Jan 16, 2015 9:53 UTC (Fri) by rsidd (subscriber, #2582) [Link] (14 responses)

Landley's larger vision is to use stock Android phones as a development platform by adding USB devices for keyboard, mouse, and display

As Landley notes, one can equally use bluetooth for keyboard and mouse, something like chromecast for display (some phones also have HDMI), but does he plan to port compilers, editors, etc to android? The easier way is to use a linux chroot, it works now, busybox helps but if toybox was part of the system that would be enough. I have actually been using my 5" phone in this way quite actively. The display is large enough for terminal-based work, and together with this keyboard (which unfolds to almost full size), it is pretty much a fully functional computer. And the whole thing fits in my pocket.

Android gets a toybox

Posted Jan 16, 2015 18:42 UTC (Fri) by dlang (guest, #313) [Link] (12 responses)

Yes, Rpb's intent is to have compilers etc on Android as well. However, it's not a case of having to port the compilers. If they can already run in a chroot and can compile native android code there, then it seems to me that the problem is one of organization, not porting.

Getting Android to have a fully capable shell rather than a crippled one is more work. That's where toybox replacing toolbox helps.

Android gets a toybox

Posted Jan 16, 2015 22:01 UTC (Fri) by rsidd (subscriber, #2582) [Link] (11 responses)

Android uses mksh, hardly a crippled shell. And not sure why busybox would help in porting bash or some more heavyweight shell.. It's probably portable already if anyone wants it.

Android gets a toybox

Posted Jan 16, 2015 22:19 UTC (Fri) by dlang (guest, #313) [Link] (10 responses)

toolbox/toybox/busybox aren't the shell, they replace full-blown versions of apps to save space and ram (minor things like ls, sed, etc)

Android won't use busybox by default, they were using toolbox, but it didn't implement a lot of commands that people want, so the move to using toybox, which does implement a lot more (and is aimed at supporting a full self-hosting environment) is a move forwards.

Android gets a toybox

Posted Jan 17, 2015 1:02 UTC (Sat) by rsidd (subscriber, #2582) [Link] (9 responses)

I know all that but you said something about a 'crippled shell' above.

Busybox/toolbox do make life easier but there's much more to a development environment, unless one goes the chroot way.

Android gets a toybox

Posted Jan 18, 2015 8:29 UTC (Sun) by landley (guest, #6789) [Link] (8 responses)

> Busybox/toolbox do make life easier but there's much more
> to a development environment, unless one goes the chroot
> way.

Actually no, there isn't. Years ago I got a self-bootstrapping environment down to 7 packages (busybox, uClibc, linux, make, gcc, binutils, bash), and I still maintain the project here:

http://landley.net/aboriginal/about.html

That project is not just my current main toybox test harness, it's how I got into busybox development in the first place (eventually becoming maintainer because I was doing more work than Erik, so he handed over the reigns). I started with Linux From Scratch (circa 3.0 or so) and tried replacing all the gnu packages, one by one, with busybox. They broke spectacularly (or were missing), so I fixed them, or completely rewrote things like "sed" and "sort" so the new implementation would actually work in a development environment.

I actually used to keep a list of which gnu packages I'd replaced with busybox. Search this page for "coreutils" to see the old list:

http://landley.net/aboriginal/old/

(Yeah, I renamed my project years ago because "firmware" wasn't quite a correct description and is impossible to google for anyway. The above link is from before the restart where I switched from User Mode Linux to QEMU as my native build environment. Started over from scratch, the current project's repository only goes back to the start of the qemu-targeted stuff.)

I got Aboriginal Linux to 1.0 years ago with an automated Linux From Scratch build that runs under it (http://landley.net/aboriginal/control-images) which I run under each new toybox release. (Currently LFS 6.8, upgrading it is on my todo list but not a high priority.) I got it to work with busybox years ago, and now I'm replacing each busybox command with toybox, one by one. This config file lists all the commands needed by a development environment to rebuild itself:

https://speakerdeck.com/landley/developing-for-non-x86-ta...

(Experimentally determined using the record-commands stuff, see "command logging wrapper" in http://landley.net/aboriginal/FAQ.html#debug_logging)

The config symbols before the comment containing the word "toybox" are still used from busybox. The ones afterwards are now provided by toybox. It's in "miniconfig" format, ala:

http://landley.net/aboriginal/FAQ.html#dev_miniconfig
https://lwn.net/Articles/161086/

This is why I keep pushing patches to remove perl from the kernel build, I do _not_ want to add perl as an 8th package required to get a minimal development environment capable of rebuilding itself under itself from source code:

https://lkml.org/lkml/2008/2/15/548

I gave a long presentation on Aboriginal Linux at three different conferences about 5 years back. My 260-slide deck is at:

http://landley.net/aboriginal/downloads/presentation.pdf

(In theory it's also at https://speakerdeck.com/landley/developing-for-non-x86-ta... but speakerdeck appears to be down at the moment...)

Rob

Android gets a toybox

Posted Jan 18, 2015 10:47 UTC (Sun) by rsidd (subscriber, #2582) [Link] (7 responses)

Sure it's possible, but is it useful or is it mainly a technology demonstration? Android takes about 8GB RAM and huge amounts of disk space to compile. Yes, phones will get there soon enough, but desktops aren't going away or getting slower...

As noted above, I do use my phone for work when I'm waiting somewhere (and that foldable bluetooth keyboard is very comfortable). But if it's serious programming stuff, I open a few terms, ssh into my work machine, and get into the GNU screen sessions that I have running there.

Android gets a toybox

Posted Jan 18, 2015 12:28 UTC (Sun) by magnus (subscriber, #34778) [Link]

Desktops are not getting slower, but since the price depends heavily on economy of scale it's possible they get more expensive over time. It sounds to me like a good idea to make sure there is a path to general-purpose computing with local data storage on whatever devices are available for the average person (smartphones, SBCs or whatever). At least it's nice to know the possibility is there.

Android gets a toybox

Posted Jan 19, 2015 20:44 UTC (Mon) by landley (guest, #6789) [Link] (3 responses)

> Sure it's possible, but is it useful or is it mainly a technology
> demonstration?

My use case is an 8 year old girl in rural india who inherited an old phone from her parents and should be able to learn to program with just that.

Having a phone should be sufficient to program for the phone, and not just ROM BASIC du jour but actually digging into the thing. Linus started Linux because he had a minimum level of knowledge and equipment, and way more time than money. He's no longer that guy, and it's starting to show. But the _next_ generation is gonna do it on phones, not PCs.

Android is a long way from that, but responds to gentle nudges. (Meanwhile iOS is actively locking people out. No user serviceable parts inside, shut up and subscribe to your feeds.)

> Android takes about 8GB RAM and huge amounts of disk space to compile.

AOSP can build in less memory when you do a non-parallel build, but doing so takes about 5 hours.

Fixing this is a "meet in the middle" sort of thing, Moore's Law marches on and it should be possible to pare AOSP down to a less monstrous version. (Right now AOSP only functions as a complete distro build in a giant integrated monolith with hundreds of packages. It should be possible to extract a base system build and make the rest more configurable. Remember we had a bunch of releases of Red Hat before we got Linux From Scratch...)

> As noted above, I do use my phone for work when I'm waiting somewhere
> (and that foldable bluetooth keyboard is very comfortable).

The keyboard linked elsewhere in the comments is very nice, but almost $200. I started learning to program when I was 11 on shared family resources, I didn't have $200 (even inflation adjusted) to spend on anything (even counting indirect purchasing power from birthdays and christmas) for like 6 years after that.

Ideally having the device should be sufficient for programming the device. Ergonomically bigger keyboard and display are really cool, but to matter they need to be cheap and ubiquitous. (I'm tracking how long until USB 3 hubs are available used. Old keyboard, old mouse, easy to come by already. People throw _out_ ones that still work. Old USB->HDMI adapter allows you to plug into an old television, but that requires USB 3. USB2 only gives you usb->vga adapter, and old monitors aren't nearly as ubiquitous as old TVs. Where "old" is "today's technology in 5-10 years".)

Still, a teenager's _already_ going to be a lot more comfortable with an on-screen phone keyboard than I'm ever likely to get. "How can you live with that cramped horrible thing, my fingers hurt just looking at you" is what an old person (like me) says to a 12 year old who has no problem with it. And there are tablets. (Did you ever notice that successful tablets are big phones, not small PCs? Except a phone you carry with you 24/7 and a tablet you generally don't, so that's not a solution unless you transiently associate your phone to the tablet just like you would to any other larger display/input devices you happened to be near...)

(Pity Google's glasshole spycams poisoned the well for glasses as an I/O device. Oh well, maybe that's temporary or doesn't apply to Korea or something...)

> But if it's serious programming stuff, I open a few terms,
> ssh into my work machine, and get into the GNU screen sessions
> that I have running there.

Microsoft's Altair BASIC was cross compiled (well, cross-assembled) from a PDP-10. Ken and Dennis cross-assembled the first unix code on a GE 645 mainframe. Looking at the previous generation as a "real computer" and the new one as "just a toy" is a common stage, and old programmers usually stick with what they know until they retire still writing COBOL. Meanwhile the new kids come up to speed on the hardware in front of them, and consider it a thing in itself rather than an extension of something else.

I'm trying to allow serious programming to be done natively on a phone. Yes, an android phone is about 7 years behind the PC in hardware specs. But I was building Linux systems 7 years ago on the hardware of the day, the fact I could get a much more powerful mainframe and just use that is not actually a new argument, it was made about minis and micros before it was made about phones.

Honestly, we've been through all this before:

mainframe -> minicomputer -> microcomputer (I.E. PC) -> smartphone.

This is the fourth iteration of the same old story. Too many people ask "is this worth doing today" rather than "would this be nice to have in 5-10 years".

I can't fix all the problems, but I can chip away at obvious bottlenecks to try to steer the narrative through the paths I want it to take rather than yet more DRM lockdown nonsense (replaying the lost decade of shrinkwrap software turning into abandonware the PC went through).

Android gets a toybox

Posted Jan 20, 2015 6:03 UTC (Tue) by rsidd (subscriber, #2582) [Link] (2 responses)

>The keyboard linked elsewhere in the comments is very nice, but almost $200.

Lazy google link on my part. Newegg, where I got mine, lists it for $23 (plus shipping which was significant in my case, but still a total of perhaps $40).

If you can indeed get kids to program on their phones, that would be great. I get what you're saying about power, but with phones it's not just the power, but the interface. For programmers, the mainframe->mini->desktop transition came with improved interfaces (punch card -> keyboard -> keyboard+mouse, teletype -> vt200->cga->vga->modern monitors). Phones are a huge step back, and if you are going to hook them up to external keyboards and displays, you lose most of the benefits of a phone. But I'm happy to be proved wrong.

Android gets a toybox

Posted Jan 21, 2015 4:25 UTC (Wed) by landley (guest, #6789) [Link] (1 responses)

> I get what you're saying about power, but with phones
> it's not just the power, but the interface.

I'm sad I can't find the article I wrote for linuxandmain.com back around 2002 on exactly this issue. (I was arguing that the next logical step beyond laptops was hooking a palmtop (newton/ipaq/palm) up to cell tower modems, the reply was that the ergonomics made it impossible for that to ever grow beyond a tiny power user niche. Alas the site went down years ago and archive.org's copy is swiss cheese. The iPhone was announced about 4 years later; presumably it was already in development when I wrote that article)

Here's a project that put a linux dev environment chroot on an android phone a year or two back:

https://play.google.com/store/apps/details?id=champion.gn...

That's the current baseline of sad ui that we have to improve upon. (And of course the above can't come preinstalled because it's full of GPL stuff Android has an official policy about not shipping. And I'd prefer a version that didn't cause my phone to reboot if I left it running more than an hour.)

> For programmers, the mainframe->mini->desktop transition
> came with improved interfaces (punch card -> keyboard ->
> keyboard+mouse, teletype -> vt200->cga->vga->modern monitors).

Minicomputers didn't start with improved UI, the first one to take off was the Altair which had toggle switches and LEDs as its input and output. The old PBS computer history documentary "Triumph of the Nerds" starts describing the altair at 18 minutes 30 seconds into here:

https://www.youtube.com/watch?v=2tsBAS91lTI

It goes on to describe the first thing an Altair ever actually _did_, which was use a series of timed loops to generate radio interference, so you could put an AM radio on top of it and it would play a song.

Intel's 8008 processor was designd to go in a "glass tty", I.E. a video display terminal hooked up to a minicomputer via serial port. The 8080 in the Altair was a sequel to the 8008. (The Intel section of http://landley.net/history/mirror/ starts with interviews with all four of the principals in the 4004->8008->8080 (and z80).)

From the Altair, we went to various ROM Basic systems hooked up to _televisions_ (max possible horizontal resolution about 340 pixels due to the NTSC signal, see http://landley.net/history/scans/icpug-1986-jul/page290-2... so with 8 pixel characters you have 42 columns max), and only years later managed DOS and CP/M (single-tasking, booted from a floppy, not a step up from a PDP-11 with a vt100). The PC _grew_ better UI as it went along, but it started out with monochrome video (then EGA graphics, four colors! One of which was Cyan!)

Meanwhile, Ken Thompson wrote unix on a PDP-7 that he _originally_ started using because it had a vector graphics terminal hooked up to it, so he ported an early version of lunar lander to it. (I don't believe unix used the graphics terminal, but the vector graphics terminal is why he started poking at that mothballed machine.) The reason it was mothballed and abandoned it was a cheap low-end system, nothing special.

The first 3D card (SGI's first product) was an add-on board for a VAX that powered a display terminal. (The Iris 1000: http://www.nekochan.net/wiki/Silicon_Graphics#First_gener...)

So by the time the PC grew 3D cards (ala Voodoo and such) circa 1995, minicomputers already had that for a decade (and tron was rendered on minicomptuers in 1982).

The convenience of the PC was you didn't need to sign up for a timeslot on the terminal down the hall when you could have your own dedicated system on your desk. Now you don't need the computer on your desk when you have a comptuter in your pocket.

Oh, the founder of DEC (Ken Olsen) was proud that the PDP-1 had a built-in video display terminal in the operator's console. That's the machine spacewar" was written for. Here's an interview with the Ken Olsen where he talks about that: http://landley.net/history/mirror/interviews/olsen.html

And the wikipedia article about spacewar (in 1961, 14 years before the Altair shipped):

http://en.wikipedia.org/wiki/Spacewar_(video_game)

> Phones are a huge step back,

The advantage the phone has is that there are billions of them deployed today and people carry them around 24/7. We don't have to make it a better programming environment than big iron, we just have to make it _possible_ to program on them. Having a device should be sufficient to program the device, and not in a second class "you can write BASIC programs but to make real programs like the one it comes with you need a real computer which this isn't".

Hobbyists developers learn to program on the machines they have. I'm not trying to put phones in people's hands, that's _today_. I'm trying to make those phones fully programmable without being second-class citizens dependent on external big iron. (And these days, a PC is big iron. Already happened, past tense. That was what the http://youtu.be/SGmtP5Lg_t0 talk was about.)

> and if you are going to hook them up to external keyboards
> and displays, you lose most of the benefits of a phone.

The USB keyboard/mouse/monitor allows old farts like you and me to switch over from our big iron VAXes to the new micro-er computers. I doubt your average 12 year old will find an onscreen keyboard "too small and cramped" if that's what they grew up with, and the UI should naturally evolve in response to what people are doing with it.

We are the unix greybeards singing "all the world's a vax" and squinting at those strange kids with their 16 bit PCs going "what a silly toy". Except computer history's a hobby of mine, and this is the fourth generation of this particular transition. I don't have to know how the problems will be fixed to know they will, I'm just trying to steer the future towards android and away from iOS because one is a multivendor commodity system with a theoretically modular software architecture, and the other is a hermetically sealed and locked down system produced by a monopoly that sells the hardware too.

> I'm happy to be proved wrong.

I'm operating on the assumption that there are already 10x as many smartphones in the world as PCs, that people already carry them around 24/7, and that historically economies of scale have always strongly favored the system with the higher unit volume.

Given that smartphones exist, they should be fully natively programmable. If you're going to write code for a phone, you should be able to do it on the phone.

Rob

Android gets a toybox

Posted Jan 21, 2015 5:46 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

> I doubt your average 12 year old will find an onscreen keyboard "too small and cramped"
They certainly do - children are not some kind of alien species.

Android gets a toybox

Posted Jan 22, 2015 23:27 UTC (Thu) by xslogic (guest, #97478) [Link] (1 responses)

That's 8GB of RAM to compile the whole OS.

Most people don't compile their whole OS that much - unless they're updating it.

Mostly, people will (and do) compile applications.

Android gets a toybox

Posted Jan 23, 2015 3:45 UTC (Fri) by rsidd (subscriber, #2582) [Link]

Rob mentioned a "self-bootstrapping" environment, which has to mean a working OS. Or it could be a chrooted environment, but that's already possible on android.

Also, most people developing on android devices would want to develop android apps. The development environment for that (sdk+ide etc) is pretty heavyweight, and I'm not sure it even exists on ARM. But it would definitely be brilliant if you could compile, debug and run an app right on your mobile device without needing a computer. Even better if you could recompile the kernel and install it while keeping the earlier one as a backup -- we are all used to doing that on desktops, but doing it on a mobile device is much trickier.

Android gets a toybox

Posted Jan 18, 2015 8:04 UTC (Sun) by landley (guest, #6789) [Link]

> but does he plan to port compilers, editors, etc to android?

Yes, I gave a talk about this at ELC in 2013:

video: http://youtu.be/SGmtP5Lg_t0
outline: http://landley.net/talks/celf-2013.txt

I was also interviewed about it on a podcast a few months back:

http://linuxluddites.com/shows/episode-11/

Rob

Android gets a toybox

Posted Jan 28, 2015 17:56 UTC (Wed) by nye (subscriber, #51576) [Link] (1 responses)

>Android kernels tend to only support USB gadget mode

Since when? Is this a new development as of 5.0, or does it refer to vendor-customised (ie non-stock) versions?

I know that in the past I've plugged a USB hub into my stock Nexus 7 and connected USB mouse, keyboard, and mass storage device into it (if you want to mount a mass storage device you will need root, but the others work without any customisation at all), and I'm pretty certain I've done so since 5.0.

USB On-The-Go

Posted Feb 9, 2016 3:36 UTC (Tue) by jasonspiro (guest, #38047) [Link]

nye, your Nexus 7 clearly supports USB On-The-Go ("OTG" for short). I own just one single Android device, and it supports OTG as well.

Jake Edge claims in the article that Android kernels tend not to support OTG. I don't know whether or not his claim is correct. But I do suspect that the majority of Android devices sold in the past few years do support OTG.