The curious case of O_DIRECTORY|O_CREAT

Posted Mar 28, 2023 19:40 UTC (Tue) by roc (subscriber, #30627)
In reply to: The curious case of O_DIRECTORY|O_CREAT by NYKevin
Parent article: The curious case of O_DIRECTORY|O_CREAT

You don't want userspace code execution to be part of the transaction, that seems really dangerous even with optimistic concurrency. You would want to be able to submit the complete transaction for validation and execution by the kernel. Actually these days you could probably cobble something together using io_uring and eBPF to do this.

to post comments

The curious case of O_DIRECTORY|O_CREAT

Posted Mar 29, 2023 14:59 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (4 responses)

> You don't want userspace code execution to be part of the transaction, that seems really dangerous even with optimistic concurrency. You would want to be able to submit the complete transaction for validation and execution by the kernel.

Isn't that exactly what SQL does with COMMIT right now?

The curious case of O_DIRECTORY|O_CREAT

Posted Mar 29, 2023 16:50 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

No SQL doesn't do it ... Oracle or PostGreSQL or ScarletDME or whatever do it.

The database batches things up until userspace/SQL/DataBASIC says "okay I'm done". At present, the kernel does not have the ability to do that.

And actually, it might be quite tricky for the kernel, in general, because with a database COMMIT, the database maintains different versions of reality. Yes I know Posix mandates some sort of reality distortion field, but when you're trying to get VFS, and btrfx, and ext?, and whatever whatever whatever all to agree, life gets rather hard ...

Cheers,
Wol

The curious case of O_DIRECTORY|O_CREAT

Posted Mar 29, 2023 18:30 UTC (Wed) by jhoblitt (subscriber, #77733) [Link] (1 responses)

> No SQL doesn't do it ... Oracle or PostGreSQL or ScarletDME or whatever do it.

AFAIK, ANSI/ISO SQL includes transactions.

The curious case of O_DIRECTORY|O_CREAT

Posted Mar 29, 2023 19:49 UTC (Wed) by Wol (subscriber, #4433) [Link]

Yes it does. But ANSI/ISO SQL doesn't *DO* transactions - it's an interpreted/jit'd language. In the words of the GP, it is "user space code" that shouldn't go anywhere near doing transactions.

COMMIT merely tells the underlying layer (and it doesn't have to be SQL, ScarletDME is NoSQL and there are a heck of a lot of NoSQL databases out there that have "commit" - without it you can't really claim to be a database at all) "this series of actions are supposed to be carried out in an atomic manner".

This is basically the big problem that everything has with transactions. It's easy for COMMIT (or my case, more likely START TRANSACTION / END TRANSACTION) to *define* *what* *is* *SUPPOSED* *to* *be* *atomic*, it's a much bigger problem for the underlying layer to actually implement it atomically.

Cheers,
Wol

The curious case of O_DIRECTORY|O_CREAT

Posted Mar 29, 2023 18:36 UTC (Wed) by jhoblitt (subscriber, #77733) [Link]

Yes and transactions can be a useful feature in a filesystem. A few years prior to the existence of AWS S3, I wrote an object store like pseudo filesystems by basically converting the kernel's ext3 headers into a sql schema. Transactions were useful in several situations, such as atomic "file" renames where the target name was an existing "file" which needed to be unlinked.