Debian considering automated upgrades

Posted Dec 15, 2016 6:20 UTC (Thu) by tnoo (subscriber, #20427)
Parent article: Debian considering automated upgrades

This looks nice, but is a really bad idea, especially on servers. What if this is a compute node that runs some self-compiled code which requires certain libraries. Changing these libraries while the process is running, or upon a new start of the software (triggered by scripts or events), will eventually lead to problems that are unpredictable and difficult to recover from.

And on desktops: do we want reboots during presentations, or major changes in software before an extended trip without reasonable internet connection? Don't follow the path of commercial dummy-user systems, we're all adults here.

to post comments

Debian considering automated upgrades

Posted Dec 15, 2016 13:15 UTC (Thu) by mikapfl (subscriber, #84646) [Link] (1 responses)

Well, on a compute node that runs some brittle self-compiled code, you just opt out of the automated upgrades. We are only talking about the default value changing, debian will certainly not make it impossible to change the configuration.
And for your fear of desktop usage breaking due to "major changes in software": We are talking about debian stable updates here. There are no major changes in software in debian stable, that's the point of debian stable. By the way: On the desktop, all the big desktop environments already have their own automated or semi-automated updating systems, so for the desktop this is not even new, it is the reality already.

Debian considering automated upgrades

Posted Dec 16, 2016 0:57 UTC (Fri) by pabs (subscriber, #43278) [Link]

> There are no major changes in software in debian stable, that's the point of debian stable.

The browsers are updated to new major versions in Debian stable, since there is no sane way to provide normal security updates for browsers in 2016. Luckily Firefox has ESR, but still, that will eventually lead to having to backport Rust to be able to compile Firefox ESR.

Debian considering automated upgrades

Posted Dec 15, 2016 13:44 UTC (Thu) by itvirta (guest, #49997) [Link] (1 responses)

> Changing these libraries while the process is running,

Library upgrades are usually done by unlinking the old version and creating a new one, instead of overwriting the file,
which would corrupt the binary image in memory.

> or upon a new start of the software (triggered by scripts or events), will eventually lead to problems that are unpredictable and difficult to recover from.

If a stable update contains more than bugfixes or the like, thus breaking any sensible application, I would count that as a bug with the updated package itself.
Of course, if you're running something very sensitive, disable the automatic updates.

> do we want reboots during presentations

Of course not. But unattended-upgrades doesn't reboot, it merely sends an email suggesting nicely that you might want to.

Debian considering automated upgrades

Posted Dec 16, 2016 4:04 UTC (Fri) by zlynx (guest, #2285) [Link]

Library updates have, in the past, inconvenienced me. Of course, I do blame it on bad developers.

For example, some binary data file that is used by a set of applications. It is accessed through a library. The main application keeps on running, using the old library and the old data file.

But wait! What happens when some innocent DBUS event launches one of these small utility programs? It spins up, the new library determines that it should upgrade the binary data file, it locks it, updates it IN PLACE, and continues on. Everyone should be happy right?

No! Suddenly the old app that is still running has a new format to deal with and it has no idea what just happened.

Sometimes I think certain developers never actually use their own stuff. And if I recall correctly this was "fixed" by having the install script just kill everything that could have the file open.

Debian considering automated upgrades

Posted Dec 15, 2016 15:22 UTC (Thu) by Seegras (guest, #20463) [Link]

> What if this is a compute node that runs some self-compiled code
> which requires certain libraries.

If this really is your problem, the automated upgrades are not your problem.

I usually pummel those who deploy "self-compiled code which requires certain libraries." with some kind of heavy implement, in order to hammer in some sanity into their brains.

Debian considering automated upgrades

Posted Dec 15, 2016 23:38 UTC (Thu) by mcatanzaro (subscriber, #93033) [Link] (3 responses)

Fedora tried this years ago and the result was bricked desktops when users powered off or rebooted during an update that they didn't know was even in progress. Some very careful attention should be paid to user-interface requirements in order to avoid rehashing this problem (not that I expect this to actually happen...). At least for the GNOME, which is dead-set on supporting only offline updates to improve reliability, that's all going to have to happen downstream.

I'm surprised that Debian is going the complete opposite direction of Fedora here on update reliability. It's one thing to automatically *prompt* users to install updates, or to do so automatically during a reboot... but completely unattended seems to make more sense for servers that are actually unattended.

Debian considering automated upgrades

Posted Dec 16, 2016 1:00 UTC (Fri) by pabs (subscriber, #43278) [Link]

The discussion was mostly about cloud stuff.

Debian considering automated upgrades

Posted Dec 16, 2016 4:10 UTC (Fri) by zlynx (guest, #2285) [Link] (1 responses)

This could be done somewhat safely with BTRFS or LVM. Make /usr read-only. Create a read-write snapshot of /usr, apply the updates to that snapshot and then somehow atomically replace /usr with the new snapshot. Just mount over it? Is there a way to unmount the old /usr after, or would this plan just result in an eventual pile of 100 copies of /usr?

Debian considering automated upgrades

Posted Dec 17, 2016 9:31 UTC (Sat) by Cato (guest, #7643) [Link]

Snapshotting can help, but btrfs still seems not ready for production, and LVM snapshots are quite buggy - see http://serverfault.com/questions/279571/lvm-dangers-and-c...

Debian considering automated upgrades

Posted Dec 15, 2016 23:58 UTC (Thu) by st (guest, #96477) [Link]

I would expect that either pinning the necessary libs to specific versions using apt pinning or building the custom code into a .deb package that depends on specific versions of the libs would solve this problem while still allowing the system in question to benefit from automated updates of the rest of the packages.

Debian considering automated upgrades

Posted Dec 16, 2016 1:42 UTC (Fri) by JanC_ (guest, #34940) [Link] (2 responses)

That shouldn't really be a problem with 'unattended-upgrades':

* by default it only upgrades for security issues; you could also configure it to only upgrade from your own private tested repository, if you think that's necessary on certain critical systems
* it allows for a configurable blacklist of packages that will never be upgraded
* it can try to fix interrupted upgrades automatically (or you can disable this)
* you can configure it to delay upgrades until shutdown/reboot
* automatic reboot is possible but optional, can be scheduled to happen immediately or at a particular time, and when disabled it's easy for other programs or scripts to check if a reboot is wanted at a more opportune moment

etc.

And in any case, removing or disabling it is easy if that's what you really want/need.

Debian considering automated upgrades

Posted Dec 16, 2016 1:48 UTC (Fri) by JanC_ (guest, #34940) [Link] (1 responses)

Also see: https://github.com/mvo5/unattended-upgrades (the default configuration files are under /data/ )

Debian considering automated upgrades

Posted Dec 17, 2016 9:34 UTC (Sat) by Cato (guest, #7643) [Link]

A fairly nice Ansible role that installs and configures unattended-upgrades for Debian and Ubuntu: https://github.com/jnv/ansible-role-unattended-upgrades.git

Ansible is lightweight compared to Puppet/Chef/Salt etc - only requires SSH and Python on target server (and you can bootstrap to install Python if needed), and is easy to learn since its scripts are executed sequentially. See http://docs.ansible.com/