KHB: A Filesystems reading list

Posted Aug 22, 2006 1:14 UTC (Tue) by nlucas (subscriber, #33793)
Parent article: KHB: A Filesystems reading list

Another excelent article by Valerie Henson for those like me who don't really want to be filesystem experts but always want to know a little more.

I'll be waiting for the next! :-)

One thing I miss is some info about any generic filesystems for the now common flash disks, or even CD+-RW/DVD+-RW. Last time I checked jffs/yafs can't be used on generic block devices, only on memory devices (or with some wrapper driver). Couldn't we be using some variation of jffs2 for CD+-RW/DVD+-RW? Maybe with the help of FUSE, as the kernel doesn't know how to write to the CD/DVD?

It seems ideal for incremental backups, at least...

to post comments

KHB: A Filesystems reading list

Posted Aug 22, 2006 17:45 UTC (Tue) by nix (subscriber, #2304) [Link] (9 responses)

CD-RW et al are what UFS is designed for. JFFS has completely different
constraints (i.e. zero-seek-time filesystems rather than
very-high-seek-time filesystems). I haven't tried it but I'd wager the
performance of JFFS on CD-RW would be utterly awful.

KHB: A Filesystems reading list

Posted Aug 22, 2006 23:25 UTC (Tue) by nlucas (subscriber, #33793) [Link] (8 responses)

CD-RW et al are what UFS is designed for.

Yes I know, but last I checked it's read-only, i.e. you can't use it on a partition as a normal filesystem, like the old msdos filesystem.

You have no easy way (AFAIK) to just add a file to a DVD-RW, like copy/paste or drag a file to a DVD-RW folder.

I would be happy enough when I can do something like this from the command line:

$ ufs-cp somefile.txt /media/cdrw/somedir/

and let it do the rest in the background (and not wait an eternity, off course). Much like the old days with DOS formated floppy disks.

My home PC still have an USB 1.1 hub and so I'm used to wait until it flushes all writes to the USB pen when unmounting. I only ask for something similar for a DVD-RW (the speed should not be that much different than for USB 1.1).

And you have the advantage that you can actually prevent the user from ejecting the CD/DVD before all writes are flushed, so less risk of user generated filesystem corruption (at least for users that don't use that cute little hole to open the "cup-holder" ;-).

KHB: A Filesystems reading list

Posted Aug 23, 2006 1:24 UTC (Wed) by nlucas (subscriber, #33793) [Link] (1 responses)

s/UFS/UDF/

It seemed weird, but followed your path without thinking more about it ;-)

KHB: A Filesystems reading list

Posted Aug 23, 2006 16:34 UTC (Wed) by nix (subscriber, #2304) [Link]

Um. Yes. Brainfart, sorry.

KHB: A Filesystems reading list

Posted Aug 23, 2006 16:37 UTC (Wed) by nix (subscriber, #2304) [Link] (2 responses)

What? Packet-written CD-RWs use UDF filesystems, and act just like a normal filesystem (except that the device is really rather slow and the filesystem overhead is ridiculous). In modern CD-RWs and all DVD-RWs this is even in hardware (while I haven't yet seen a CD rewriter that has packet-writing firmware that works well, this may not be true of DVD-RW drives).

It's ISO-9660 that has to be built in one lump.

KHB: A Filesystems reading list

Posted Aug 24, 2006 21:51 UTC (Thu) by nlucas (subscriber, #33793) [Link] (1 responses)

I now understand the problem has nothing to do with filesystems, except the UDF overhead problem (and ISO-9660 compatibilty), as it's the only usable CD/DVD RW filesystem (because it can be read everywhere).

It seems the problem is only on the software side (for modern RW drives, I mean), and maybe cdrecord "flame wars" have delayed the adoption of a standard linux solution for the typical user cases (like maybe a FUSE DVD UDF filesystem that works the same as a USB pen, i.e. can add/delete files on the fly).

Well, I still have a lot of doubts about the process, but the truth is this thread is becoming off topic so I'll refrain my thoughts for latter times.

KHB: A Filesystems reading list

Posted Aug 27, 2006 15:53 UTC (Sun) by nix (subscriber, #2304) [Link]

We already have exactly that feature: the packet-writing driver. If you're
packet-writing to CD-RW (or DVD-RW) you don't *need* cdrecord, or FUSE.

(It does indeed have significant overhead, as you say.)

KHB: A Filesystems reading list

Posted Aug 24, 2006 2:12 UTC (Thu) by wookey (guest, #5501) [Link] (2 responses)

growisofs meets your command line criteria, I think.

growisofs -M /dev/dvd files

will add files to a disk (CD or DVD).

KHB: A Filesystems reading list

Posted Aug 24, 2006 7:36 UTC (Thu) by drag (guest, #31333) [Link]

growisofs even supports pipes...

My favorite backup technic for my own home directory is to use growisofs with splitpipe.

Splitpipe is a nice utility that splits files just like 'split' does, but it performs a action on each section it makes.

Check it out:
tar c /home | splitpipe -s dvd -o 'growisofs -Z /dev/dvd=/dev/stdin'

It'll burn a DVD until it gets finished, pause, prompt you for you to press enter so you can have time to pop in a new DVD and off you go.

Then they have a 'joinpipe' command for rebuilding the data.

Supports md5sum'ng the data and makes sure you don't accidently try to join the data back together in the wrong order and such.

Of course if you have a file that is less then 4 megs you could just go:

cat filenmae |growisofs -Z /dev/dvd -

And I think that should work out fine. Then you should be able to 'cat /dev/dvd > whatnot' to get it back.

Haven't tried it personally like that, but I expect it to work fine.

KHB: A Filesystems reading list

Posted Aug 24, 2006 21:57 UTC (Thu) by nlucas (subscriber, #33793) [Link]

I'm almost certain I had read the growisofs manual page some moons ago, but somehow missed that command.

Thanks for the heads-up. I'm an happier man now :-)

KHB: A Filesystems reading list

Posted Aug 22, 2006 18:27 UTC (Tue) by plougher (guest, #21620) [Link] (3 responses)

>Couldn't we be using some variation of jffs2 for CD+-RW/DVD+-RW? Maybe >with the help of FUSE, as the kernel doesn't know how to write to the >CD/DVD?
>
>
Try squashfs - this is a compressed filesystem which is being used a lot for liveCDs. It also achieves better compression than JFFS2.

>
>It seems ideal for incremental backups, at least...

Squashfs is read-only, but it does support appending to the filesystem. This was added specifically to support incremental backups.

KHB: A Filesystems reading list

Posted Aug 22, 2006 23:49 UTC (Tue) by nlucas (subscriber, #33793) [Link] (2 responses)

Try squashfs - this is a compressed filesystem which is being used a lot for liveCDs. It also achieves better compression than JFFS2.

I really don't care about compression, but I suppose it doesn't hurt.

Squashfs is read-only, but it does support appending to the filesystem. This was added specifically to support incremental backups.

But how is this used on a DVD-RW?

1) You have a squashfs disk image and append to it, writing the final image to the CD/DVD when a certain size is reached?

2) You use a CD/DVD-RW (with UFS or whatever) and update a squashfs image file on the disk?

3) Other...

If it's 1) then it doesn't matter what filesystem is used. If it's 2), it's nice for incremental backups, but it will not allow you to take advantage of the fact that you can delete files on a -+RW disk (but good for -+R disks).

Hope it's 3)...

KHB: A Filesystems reading list

Posted Aug 23, 2006 1:25 UTC (Wed) by nlucas (subscriber, #33793) [Link]

s/UFS/UDF/

KHB: A Filesystems reading list

Posted Aug 24, 2006 18:27 UTC (Thu) by plougher (guest, #21620) [Link]

>If it's 1) then it doesn't matter what filesystem is used. If it's 2), >it's nice for incremental backups, but it will not allow you to take >advantage of the fact that you can delete files on a -+RW disk (but good >for -+R disks).

You could do 2) and store the Squashfs filesystem image inside a UFS/ISO9660 filesystem. If the UFS/ISO9660 filesystem was writable, then the Squashfs file could be updated and extended in place.

>Hope it's 3)...

I always write the Squashfs filesystem directly to the disk (not within a UFS/ISO9660 filesystem). Mounting is then done by mounting directly from the disk (i.e. mount -t squashfs /dev/hda /mnt). Appending to this can be done, but it requires the ability to update the superblock (block 0), and to re-write the directory structure. I've never tried to do this, and so don't know if it is possible.

Pure appending to the Squashfs filesystem (so no blocks in the original filesystem are touched) isn't possible, because of the need to update the superblock and directory structure (stored at the end of the filesystem). Extending Squashfs to do this isn't impossible, it requires Squashfs to append the new superblock (and changed metadata) to the end of the filesystem, and then scanning for any new superblocks on mounting - i.e. what log structured filesystems do.

KHB: A Filesystems reading list

Posted Aug 24, 2006 2:26 UTC (Thu) by wookey (guest, #5501) [Link] (1 responses)

Last time I checked jffs/yaffs can't be used on generic block devices, only on memory devices (or with some wrapper driver).

Yes. This is the case. It would be nice to have the robustness of log-strucutreed filesystems on flash devices accessed as block devices, but it's actually very difficult to achieve. The problem is that although USB sticks and SD/MMC cards and CF cards use flash, they hide it behind a controller which does who-knows-what behind the scenes in order to make it look enough like a disk that cameras can pretend it is a DOS FAT disk. The controllers remap blocks to retire bad blocks and (on the better ones) to avoid premature wear in the cluster tables at the start of the FAT format. But precisely because they do these things there is little benfit to using a filesystems like YAFFS/JFFS which knows about flash directly and does the right thing itself. You might as well just use something designed for disks.

So you can use YAFFS/JFFS on smartmedia and xD cards because they expose raw NAND flash, but not on all the others.

Various sorts of shims have been considered but the general consensus seems to be that it's not worth bothering with.

KHB: A Filesystems reading list

Posted Aug 24, 2006 22:08 UTC (Thu) by nlucas (subscriber, #33793) [Link]

If we don't know what the controller does, is there any filesystem that is better than others on this kind of disks?

What I mean is that we could conceive one of those micro-atx VIA boards with an IDE flash disk (I believe they use a normal Compact Flash memory) running a linux system, but FAT seems awfull for such a system (if it works at all). I tried to google about it but couldn't find a definitive answer.