A filesystem corruption bug breaks loose

Posted Dec 11, 2018 1:21 UTC (Tue) by ken (subscriber, #625)
Parent article: A filesystem corruption bug breaks loose

I was one of the unlucky ones hitting this on ubuntu 18.10.

All important files was on NFS so not a huge problem but I found out that it no longer works to run fsck on boot in ubuntu . So now your forced to boot from a usb live system just to do fsck on root.

That upset me more than the actual filesystem bug.

to post comments

A filesystem corruption bug breaks loose

Posted Dec 11, 2018 1:28 UTC (Tue) by pr1268 (guest, #24648) [Link] (3 responses)

it no longer works to run fsck on boot

Was that because of this bug?

A filesystem corruption bug breaks loose

Posted Dec 11, 2018 1:46 UTC (Tue) by ken (subscriber, #625) [Link] (2 responses)

No I was booting 4.12 kernel when I tried to fix the drive.

tried the good old. touch /forcefsck but that creates a warning
Please pass 'fsck.mode=force' on the kernel command

but that did not help I did still got the
"warning: mounting fs with errors, running e2fsck is recommended"

That only stopped once I run fsck manually from a live usb boot.

I could not find any trace fsck was run at all on boot.

A filesystem corruption bug breaks loose

Posted Dec 11, 2018 12:59 UTC (Tue) by mgedmin (guest, #34497) [Link] (1 responses)

You can check /run/initramfs/fsck.log for a full log of fsck execution from the initramfs.

Perhaps fsck -f -a was unable to fix the corruption automatically? In which case you needed to pass fsck.repair=yes as well, to change that command to fsck -f -y. I'm not sure where that's documented, I discovered it by grepping through the initramfs shell scripts in /usr/share/initramfs-tools.

Anyway, booting a livecd and running fsck interactively sounds like a good plan, where you can tell what's going on without digging through a maze of little scripts in /usr/share/initramfs-tools.

A filesystem corruption bug breaks loose

Posted Dec 20, 2018 17:07 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]

These command-line parameters were first implemented by systemd and are documented in systemd-fsck(8). Since initramfs-tools took over running fsck on / and /usr, it is meant to support the same command-line parameters that control fsck in either initscripts or systemd.

A filesystem corruption bug breaks loose

Posted Dec 11, 2018 19:30 UTC (Tue) by zblaxell (subscriber, #26385) [Link] (5 responses)

I copied e2fsck into initramfs years ago. Also dropbear (to diagnose and repair the root filesystem remotely), mkfs and rsync (in case the diagnosis and repair does not end well, and a restore from backups over the network is required).

The old Unix way of running fsck on a live root filesystem so that it will be potentially modifying its own program text (or block maps thereof) was, at best, a workaround for not having a usable initramfs subsystem to run fsck from. That era ended 15 years ago on Linux, and even earlier on Solaris and other commercial Unixes.

I'm surprised Linux distros that boot with initramfs today still try to fsck after / is mounted.

A filesystem corruption bug breaks loose

Posted Dec 11, 2018 20:16 UTC (Tue) by lkundrak (subscriber, #43452) [Link] (3 responses)

Uh? A fresh Fedora install:

[root@nedofet lkundrak]# lsinitrd |grep -i fsck
-rwxr-xr-x   1 root     root        28512 Oct 25 09:14 usr/lib/systemd/systemd-fsck
-rw-r--r--   1 root     root          671 Oct 25 09:14 usr/lib/systemd/system/systemd-fsck@.service
-rwxr-xr-x   2 root     root            0 Jul 13 04:35 usr/sbin/e2fsck
-rwxr-xr-x   1 root     root        55952 Jul 16 13:51 usr/sbin/fsck
-rwxr-xr-x   2 root     root       339424 Jul 13 04:35 usr/sbin/fsck.ext4
[root@nedofet lkundrak]#

A filesystem corruption bug breaks loose

Posted Dec 11, 2018 20:18 UTC (Tue) by zblaxell (subscriber, #26385) [Link] (2 responses)

OK, that's one...

A filesystem corruption bug breaks loose

Posted Dec 12, 2018 3:45 UTC (Wed) by nybble41 (subscriber, #55106) [Link] (1 responses)

Debian also includes e2fsck in the initramfs:

[~]$ lsinitramfs /boot/initrd.img-4.18.0-2-amd64 | fgrep fsck
usr/sbin/e2fsck
usr/sbin/fsck
usr/sbin/fsck.ext4
usr/sbin/reiserfsck

A filesystem corruption bug breaks loose

Posted Dec 20, 2018 17:10 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]

Indeed, we've been doing this by default since Debian 8 "jessie". But if you build a custom kernel with no initramfs then of course fsck gets run in the old way.

A filesystem corruption bug breaks loose

Posted Dec 14, 2018 23:41 UTC (Fri) by nix (subscriber, #2304) [Link]

Presumably you also install iproute2, or you'll have trouble bringing the network up to rsync over it (unless it's already up for netconsole or something, I suppose).