The PuzzleFS container filesystem

Posted Sep 26, 2023 13:40 UTC (Tue) by hsiangkao (subscriber, #123981)
In reply to: The PuzzleFS container filesystem by bluca
Parent article: The PuzzleFS container filesystem

> Are there any plans to bring file-based runtime dedup to EROFS? Being able to combine that with dm-verity would be awesome

Hi! Using EROFS + dm-verity + blockdevice + overlayfs (composefs-like model), you could already implement a clean file-based runtime dedup by using overlayfs.

I understand that you mean EROFS self-contained file-based runtime dedupe across images as you mentioned in some previous article. Actually we already have an internal version for internal products by Jingbo, but it's still unclean for now. We will try to clean up and post it to fsdevel mailing list later, but not quite sure it could land smoothly, anyway.

to post comments

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:17 UTC (Tue) by bluca (subscriber, #118303) [Link] (8 responses)

Yes I mean directly in EROFS, that would be great to have and we'd use it immediately - unless I'm missing something, you can't really do this together with dm-verity in the overlayfs model, as the data storage is not actually in dm-verity, but it's in the blob storage layer?

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:30 UTC (Tue) by alexl (guest, #19068) [Link] (3 responses)

You can use dm-verity to protect a composefs-style erofs image, but you would have to use fs-verity to verity the blob storage layer.

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:52 UTC (Tue) by hsiangkao (subscriber, #123981) [Link] (2 responses)

> you would have to use fs-verity to verity the blob storage layer

I think if some blob storage layer is just an EROFS image, you could directly apply dm-verity on these layers.
And check dm-verity root digests of these layers before mounting. I think that would be in the same effect, anyway.

We can also use fs-verity to verity the blob storage layers if these layers are on a RW fs (the current composefs does.)

The PuzzleFS container filesystem

Posted Sep 26, 2023 15:02 UTC (Tue) by alexl (guest, #19068) [Link] (1 responses)

Yes, this is doable, but its kinda a weird way to store the blobs. The main goal of a system like this is to share the blobs between different images, and having the blob store be per-image is contrary to this goal.

The PuzzleFS container filesystem

Posted Sep 26, 2023 15:09 UTC (Tue) by hsiangkao (subscriber, #123981) [Link]

Yes, I know that is not composefs intended use cases.
I'm not sure if some people really rely on layering concept (such as system using raw partitions/devices without real filesystem storage), anyway.

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:35 UTC (Tue) by hsiangkao (subscriber, #123981) [Link] (3 responses)

Not quite sure if I got the point. With the current upstream overlayfs, you could actually make
- EROFS + dm-verity block device blobs as data only layers;
- a small overlayfs meta layer (with EROFS + dm-verity) to merge these data storage blobs into a merged rootfs.
Thus all layers are under dm-verity protection, so the whole image won't be tampered.

Alternatively, as an EROFS self-containerd approach, EROFS could share page cache if files with same data across images without relying on overlayfs, anyway.

The PuzzleFS container filesystem

Posted Sep 26, 2023 15:57 UTC (Tue) by bluca (subscriber, #118303) [Link] (2 responses)

How would that work in practice? Say I have an erofs image with a rootfs that contains /usr/foo/a, and other two extension erofs images that contain usr/foo/b and usr/foo/c respectively. I create two overlays, each with the base, and one of the extension, so that one has usr/foo/a+usr/foo/b and the other usr/foo/a+usr/foo/c. Is memory being deduplicated, given usr/foo/a is the same?

The PuzzleFS container filesystem

Posted Sep 26, 2023 16:08 UTC (Tue) by hsiangkao (subscriber, #123981) [Link] (1 responses)

> I have an erofs image with a rootfs that contains /usr/foo/a, and other two extension erofs images that contain usr/foo/b and usr/foo/c respectively. I create two overlays, each with the base, and one of the extension, so that one has usr/foo/a+usr/foo/b and the other usr/foo/a+usr/foo/c. Is memory being deduplicated, given usr/foo/a is the same?

In that case, memory of /usr/foo/a is deduplicated according to how overlayfs works since /usr/foo/a is on the same EROFS instance.
That already works without any extra built-in feature.

The PuzzleFS container filesystem

Posted Sep 26, 2023 16:57 UTC (Tue) by bluca (subscriber, #118303) [Link]

Ah, that's very nice, I didn't know that, thanks!