Leading items

Welcome to the LWN.net Weekly Edition for January 5, 2023

This edition contains the following feature content:

• Welcome to 2023: our predictions for the year to come.
• Adding system calls for Linux security modules: a prerequisite for security-module stacking gets closer.
• SLOB nears the end of the road: little love remains for a memory allocator aimed at tiny systems.
• The rest of the 6.2 merge window: what was upstreamed in the latter part of this merge window.
• Supporting unified kernel images for Fedora: a debate over how to secure the initial RAMdisk images used during the Linux bootstrap process.
• Not coalescing around None-aware: the ongoing debate over adding new, None-aware operators to Python.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Welcome to 2023

Yet another new year is upon us, and that can only mean one thing: the time has come for your editor to look into his crystal ball and make some predictions for what 2023 will hold. Said crystal ball is known to suffer from speculative-execution problems and parity errors, but it's the best that LWN's budget will afford. Read on for a highly unreliable look at what's to come.

The community will see a spike in AI-generated material in the coming year. Machine-learning systems can now crank out both code and text that can look convincing. It seems unavoidable that people will use them to generate patches, documentation, mailing-list polemics, forum answers, and more. It could become difficult to tell how much human-generated content a given submission contains.

Perhaps this flood of content will prove beneficial — it could increase our development pace, bring about better documentation, and provide improved help to our users. But that outcome does not seem highly likely in the near future. Instead, we're likely to see code submissions from "developers" who do not understand what they are posting; this code could contain no end of bugs and, potentially, license violations. Cut-and-paste programming has long been a problem throughout this industry. It is far from clear that automating the cutting and pasting is going to improve the situation.

AI-generated text has its own challenges. Our mailing lists and forum sites do not lack for people trying to appear authoritative on subjects they do not really understand; how many more will show up when it is easy to get a machine-learning system to produce plausible text with little effort? Even the most ardent believers in the "last post wins" approach to mailing-list discussions will get tired and shut up eventually; automated systems have no such limits. How long until we have a discussion on, say, init systems that is sustained entirely by bots?

As a community we are going to have to come up with defenses against abuses of this sort. It would be good to start in 2023.

New kernel functionality written in Rust will be proposed for inclusion into the mainline. While the initial support for Rust kernel code landed in the 6.1 kernel, it was far short of what is needed to add any interesting functionality to the kernel. As the support infrastructure is built up in coming releases, though, it will become possible to write a useful module that can be built for a mainline kernel. A number of interesting modules exist now and others are in the works; they just need the kernel to provide the APIs they depend on.

Pushing a module written in Rust for the mainline seems almost certain to spark a significant discussion. While many kernel developers are enthusiastic about the potential of Rust, there are others who are, at best, unconvinced. This latter group has gone quiet in recent times, presumably waiting to see how things play out. After all, as Linus Torvalds has said, the current Rust code is an experiment; if that experiment does not go well, the code can be taken out again.

The merging of a Rust module that people will actually use will be a tipping point, though. Once this code is merged, taking it back out would create the most obvious sort of regression; that, of course, is something that the kernel community goes far out of its way to avoid. So the merging of user-visible functionality written in Rust will mark the point where the Rust code can no longer just be torn out of the kernel; it will be a statement that the experiment has succeeded.

Anybody who is still unsure of the benefit of Rust support in the kernel will have to speak out before that happens, and some of them surely will. Reaching a consensus may take some time, to put it lightly. So, while it seems likely that this discussion will begin in 2023, it is far less likely that any user-visible functionality written in Rust will actually be merged this year.

It will be a make-or-break year for distributed social networking. The events at Twitter have highlighted the hazards of relying on centralized, proprietary platforms, and have put the spotlight on alternatives like Mastodon. This may just be the opportunity that was needed to restore some of the early vision of the net and give us all better control over our communications.

A flood of new users is also going to expose all of the weaknesses, vulnerabilities, and scalability problems in the current solutions, though. Putting up an instance running Mastodon is easy enough; managing that instance in the face of an onslaught of users, not all of whom have good intentions, is rather less so. There is going to have to be a lot of fast hacking and development of social norms if all of this is going to work.

Perhaps the outcome will be a future where we can communicate and collaborate without the need for massive corporations and erratic billionaires as intermediaries. Or maybe the open and distributed alternatives will prove unable to rise to the challenge quickly enough. People have a remarkable ability to shrug and return to abusive relationships, no matter how badly they have been burned in the past. The sad truth is that things may well play out that way this time too.

It will be the year of the immutable distribution. The classic Linux distribution follows in the footsteps of the Unix systems that came before; a suitably privileged user can change any aspect of the system at any time. For a few years now, though, we have seen movement away from that model toward a system that is, at least in part, immutable. Android is arguably the most prominent example of an immutable system; the Android core can only be changed by way of an update and reboot — and the previous version is still present in case the need arises.

Distributions like Fedora's Silverblue have been exploring the immutable space as well. The upcoming SUSE Adaptable Linux Platform (ALP) is based on an immutable core, as is the just-released, Ubuntu-based Vanilla OS system. It seems likely that others will follow, perhaps using the blueprint that was laid out at the 2022 Image-Based Linux Summit. By the end of the year, there may be a number of immutable alternatives available to play with — and to use for real work.

A distribution with an immutable core offers a higher level of security, since a malicious process or inattentive administrator is unable to corrupt the base system. It provides a stable, consistent base on which applications, often in the form of containers, can be run. Immutable systems naturally lend themselves to fallback schemes that can make recovery from a failed update easy to the point of being transparent. It's not surprising that this approach is attracting interest.

An immutable system makes a lot of sense for cloud deployments, where changes after an image is launched are unwelcome at best. They seem to work well for Android devices, where users are unable to (and generally don't want to) change the core system in the first place. It remains to be seen whether immutability will prove attractive on desktop systems where, arguably, a larger number of users want to be able to tinker with things and may not be interested in running a platform that makes life easier for vendors of proprietary software.

Let the year begin

LWN completes its first quarter-century at the end of January; what a long, strange trip it's been since we put out our first weekly edition in 1998. Thanks to you, our subscribers, we're still at it, which is a good thing, since the Linux and free-software communities are far from done. We'll still be here at the end of 2023, when the time will come to look back at these predictions and have a good laugh. In between now and then, we're looking forward to covering our community from within; to say that it will be interesting is the safest prediction of all.

Comments (89 posted)

Adding system calls for Linux security modules

The Linux security module (LSM) subsystem has long had limitations on which modules could be combined in a given running kernel. Some parts of the problem have been solved over the years—"smaller" LSMs can be combined at will with a single, more complex LSM—but combining (or "stacking") SELinux with, say, Smack or AppArmor has never been possible. Back in October, we looked at the most recent attempt to add that ability, which resulted in patches to add two new system calls for LSM. By the end of December, the number of new system calls had risen to three.

The underlying problem that Casey Schaufler is trying to solve is the handling of the multiple security contexts and how to report them to user space; that is one of the barriers to stacking two or more context-using LSMs. These contexts are a string representation of the information used by an LSM to make its access-control decisions. Schaufler's efforts to fully solve the LSM-stacking problem have now stretched over the last ten years.

His original plan for reporting the contexts was to simply have multiple null-terminated entries in the relevant /proc file for each attribute when there were multiple context-using LSMs active in the system. Multi-LSM-aware applications could be modified to read past the first null to see if there was additional context information. But in September, new LSM maintainer Paul Moore looked at the patches again with an eye toward merging them. That API, which even Schaufler was not particularly happy with, was not one that Moore was willing to add. He suggested that the time had come to add some LSM-specific system calls to make that information available.

After some grumbling, Schaufler came up with some patches along the lines of what Moore had suggested. Version 1 of the patch set added lsm_self_attr() for retrieving attributes for the current process and lsm_module_list() to list the active LSMs in the system. In version 4, which was posted December 29, lsm_self_attr() had become lsm_get_self_attr(); the patch set added lsm_set_self_attr(), which will allow setting security attributes for the current process.

The comments on this round are scant—non-existent as of this writing—which may reflect a lack of attention during the holiday season or that the code is close to ready to merge. The previous version of the patch set also added the three calls and was not particularly contentious when it was posted toward the end of November. That might suggest the latter interpretation.

Other than the addition of an interface to set the security attributes, there are not a lot of changes from the original proposal. The "reserved IDs" for LSMs has remained, though there have been multiple objections to them along the way. Essentially, all of the uses of the integer LSM IDs (to identify a specific LSM) start at 100 (LSM_ID_CAPABILITY for the capability LSM) and every positive integer below that is reserved for future use. Moore was adamant that a pool of reserved IDs remain; this round of the patch set has a comment in lsm.h that explains how those might be used:

The LSM infrastructure itself has no variable state, but that may change. One proposal would allow loadable modules, in which case an attribute such as LSM_IS_LOADABLE might identify the dynamic modules. Another potential attribute could be which security [module] is associated with network labeling using netlabel. Another possible attribute could be related to stacking behavior in a namespaced environment. While it would be possible to intermingle the LSM infrastructure attribute values with the security module provided values, keeping them separate provides a clearer distinction.

Looking at the most recent patches, the LSM listing function looks much as it did before:

    int lsm_module_list(unsigned int *ids, size_t *size, unsigned long flags);

In ids, it returns an array of LSM IDs for the active LSMs in the system in the order they were added, as long as there are sufficient entries in size. flags is reserved for future uses and must be zero. The return value is the number of IDs returned or a negative error code.

Likewise, lsm_get_self_attr() picks up from what the call that it replaced (lsm_self_attr()) did: fill in an array of struct lsm_ctx entries with all of the attributes that apply to the current process for all of the active LSMs. That structure looks much the same as it did before:

    struct lsm_ctx {
	    __u32		id;
	    __u64		flags;
	    __kernel_size_t	ctx_len;
	    __u8		ctx[];
    };

The id holds the LSM ID and flags is unused as yet; ctx_len is the size of the ctx array that holds the value. ctx_len is at least strlen(ctx)+1 in size since ctx is always null-terminated. The description of the returned array of lsm_ctx values can be found in the patch that adds lsm_get_self_attr().

The attributes that are currently defined are described in the documentation file (userspace-api/lsm.rst) that appears near the top of the first patch. They consist of a half-dozen types of attributes that are currently available in /proc files, such as LSM_ATTR_CURRENT, which is the active security context for the process (i.e. /proc/attr/current). That attribute is the only one that is shared between all three of the context-using LSMs (SELinux, Smack, and AppArmor). Two of the other attributes (LSM_ATTR_EXEC for the context when the process was executed and LSM_ATTR_PREV for the previous security context, if any) are shared between SELinux and AppArmor. The others currently only pertain to SELinux. The "Files and directories" section of the proc man page has some more information on the attributes and how they are used.

The call looks basically the same as its predecessor:

    int lsm_get_self_attr(struct lsm_ctx *ctx, size_t *size, unsigned int flags);

It will return the array in ctx, with size updated to the returned length. The previous version required flags to be zero, but now it can be used to restrict which of the attributes are returned by setting individual bits for each of the attributes of interest.

The counterpart to that is the new lsm_set_self_attr():

    int lsm_set_self_attr(struct lsm_ctx *ctx, size_t size, unsigned int flags);

It takes a pointer to a single lsm_ctx (of the given size) and tries to set the value of the (single) attribute identified in flags. The function calls into the LSM code to set the value, which allows the security modules to decide whether to allow the operation or not. The function returns zero on success or a negative error code if it fails.

Since it was initially posted, a set of kernel self-tests for the system calls have been added to the patch set. There were few real complaints about the previous version, so one might just conclude that these system calls are about ready to merge, which would be a significant step toward Schaufler's longtime goal. Perhaps before the end of 2023, we will see the quest for this phase of LSM stacking come to a close. Next up would be completing the handling the networking side of the stacking problem, which will lead to the "universal stacking" nirvana that Schaufler has been seeking—lo these many years.

Comments (5 posted)

SLOB nears the end of the road

The kernel project tries hard to avoid duplicating functionality within its code base; whenever possible, a single subsystem is made to serve all use cases. There is one notable exception to this rule, though: there are three object-level memory allocators ("slab allocators") in the kernel. The desire to reduce the count has been growing stronger over the years, and some steps have been taken in 6.2 to eliminate the least-loved allocator — SLOB — in the relatively near future.

The job of a slab allocator is to provide the kernel with (usually) small chunks of memory in an efficient way. The heavily used kmalloc() function is implemented by the slab allocator, but there is also a lower-level API specialized for the management of numerous objects of the same size. It is quite common for a kernel subsystem to need to allocate instances of a given structure, for example; all of those instances are normally the same size and can be managed in a slab.

The kernel's oldest slab allocator is typically just called SLAB (though the name is not an acronym); it has been there in one form or another since nearly the beginning. SLAB is intended to be a general-purpose allocator suitable for most workloads, and serves that purpose reasonably well. Even so, developers have occasionally wished for a different approach to object allocation; that led to the SLOB allocator, which was added in 2006 for the 2.6.16 kernel release; its purpose is to support the smallest of systems where SLAB's memory use was seen as being too high:

SLOB is a traditional K&R/UNIX allocator with a SLAB emulation layer, similar to the original Linux kmalloc allocator that SLAB replaced. It's significantly smaller code and is more memory efficient. But like all similar allocators, it scales poorly and suffers from fragmentation more than SLAB, so it's only appropriate for small systems.

The third allocator, SLUB, followed for 2.6.22 in 2007. SLUB, which was intended to eventually replace SLAB, was a reaction to the perceived complexity and scalability problems in SLAB; it had a strong focus on scalability and performance. There have been attempts to add other allocators — SLQB in 2008 and SLEB in 2010, for example — but the appetite for more allocators had mostly faded by then; Linus Torvalds made it clear in 2010 that he would not accept any more of them. So, in 2022, the set of mainline slab allocators remains SLAB, SLOB, and SLUB.

Since then, there has been occasional talk of removing at least one of the existing slab allocators, but no real movement in that direction — until Vlastimil Babka took up the issue this year. In November, he proposed deprecating SLOB as a step toward its eventual removal:

The unsurprising reasons include code maintenance burden, other features compatible with only a subset of allocators (or more effort spent on the features), blocking API improvements (more on that below), and my inability to pronounce SLAB and SLUB in a properly distinguishable way, without resorting to spelling out the letters.

The API improvement mentioned there is adding the ability to use kfree() on objects obtained from the lower-level slab functions (specifically kmem_cache_alloc()); the SLOB allocator's version of kfree() can only handle objects allocated with kmalloc(), which complicates code elsewhere in the kernel. This cost is maybe justified if it brings other benefits, but Babka suggested that, in fact, nobody is actually using SLOB. The small devices it was aimed at (less than 32MB of RAM) don't exist in large numbers anymore, and even distributions for small devices (such as OpenWrt) are not using it.

In the ensuing discussion, Paul Cercueil described his attempts to use SLOB, concluding that it simply does not work properly. Torvalds also recalled hearing about SLOB problems in recent times; Aaro Koskinen responded, though, that SLOB might just be exposing driver bugs that are hidden by the other allocators. In the end, though, SLOB had no real defenders; nobody is advocating for keeping it in the kernel.

Still, one cannot just remove an allocator from the kernel without trying to meet the needs of any existing users. So Babka posted a patch set that tried to minimize the impact on any SLOB users that may remain. It introduces a new SLUB_TINY configuration option that removes many of the scalability and debugging features in an attempt to make SLUB as small as possible. SLUB will never be as small as SLOB but, hopefully, it can be made small enough for what passes as a "small system" in the 2020s. The SLOB configuration option was then renamed to SLOB_DEPRECATED, and it gained a warning that SLOB would be removed "in a few cycles". This patch series was merged for 6.2.

Unless a compelling case for the retention of SLOB comes up over the course of the next year, the removal of that allocator seems nearly certain. Then Linux will be down to just two allocators, both of which claim to handle the general case.

Babka has made it clear that he sees two as still being too many; the removal of SLAB is sure to come up once the SLOB removal is complete. Getting rid of SLAB will not be quite so easy, though, since SLAB still has active users. Notably, Google is said to be using SLAB internally. It seems that SLUB is still worse for some use cases; as long as that situation persists, removing SLAB will be hard. This problem is complicated by a lack of general understanding of just where and why SLUB falls down.

Getting a handle on that situation may take some time, so the kernel is likely to continue to have two slab allocators for some time yet. But even the removal of SLOB will make life easier in a number of ways. It can seem like code added to the kernel is there forever, but it is, sometimes, possible to get rid of old subsystems with enough patience and effort.

Comments (18 posted)

The rest of the 6.2 merge window

The world got a special Christmas present from Linus Torvalds this year in the form of the 6.2-rc1 kernel prepatch. By the time the merge window closed, 13,687 non-merge changesets had been pulled into the mainline for the 6.2 release. This was the busiest merge window since 5.13 (which brought in 14.231 changesets) in mid-2021, and quite a bit busier than 6.1 was — but comparable to the late 5.x releases. Just under 4,000 of those changesets were pulled after the first-half summary was written; there were quite a few significant changes to be found in those late-arriving patches.

The most significant changes pulled in the latter part of the 6.2 merge window include:

Architecture-specific

• The kernel can now perform return stack buffer stuffing to mitigate the Retbleed speculative-execution vulnerability on some Intel processor generations with a much lower performance cost.
• The x86 architecture has also gained support for a control-flow integrity mechanism called FineIBT.
• There is a new qspinlock implementation for the PowerPC architecture; it should provide improved performance and fix some lockup problems seen in extreme cases.
• LoongArch has gained support for ftrace, suspend, hibernation, and stack protection.

Core kernel

• The zram device can now recompress data streams to achieve better compression rates; see this documentation commit for details.
• Shared anonymous memory areas can now be named; this capability extends the current memory-naming feature, which was previously limited to private memory.
• The new trace_trigger= command-line option can enable a tracing trigger at boot time.

Filesystems and block I/O

• There is a new set of sysfs knobs that can be used to fine-tune how much of the system's page cache can be used by pages destined to be written back to a specific device. See the documentation commits for strict_limit, max_bytes, min_bytes, max_ratio_fine, and min_ratio_fine for details.
• The F2FS filesystem has gained an "atomic replace" ioctl() operation that can write data to a file and truncate it in a single atomic operation. F2FS has also gained a block-based extent cache that can be used to determine which data is hot (in active use) or cold; this commit contains a little information.
• The ntfs3 filesystem has a few new mount options, starting with the undocumented nocase, which appears to control case-sensitive lookups. The windows_name option will prevent the creation of file names that Windows would not allow, and hide_dot_files controls whether files whose names start with "." are marked as being hidden.

Hardware support

• Industrial I/O: Analog Devices MAX11410 and AD4130 analog-to-digital converters, MediaTek MT6370 analog-to-digital converters, Kionix KX022A tri-axis digital accelerometers, Maxim MAX30208 digital temperature sensors, Analog Devices AD74115H I/O controllers, and Analog Devices ADF4377 microwave wideband synthesizers.
• Miscellaneous: Microsoft Azure network adapters, Baikal-T1 PCIe controllers, Ampere Computing SMPro error monitors, Lattice sysCONFIG SPI FPGA managers, Advantech embedded controller watchdog timers, Renesas R-Car S4-8 Ethernet SERDES PHYs, TI TPS65219 power management ICs, and Xilinx R5 remote processors.
• Also: "iommufd" is a new user-space API for the control of I/O memory-management units; see Documentation/userspace-api/iommufd.rst for details.

Miscellaneous

• As always, the perf tool has seen a lot of enhancements; see this merge message and this one (which drew an unhappy response from Torvalds) for details.
• The copyleft-next license has been added as a valid license for kernel contributions — but only when dual-licensed with a GPL-compatible license.

Security-related

• The kernel can now place an upper limit (10,000 by default) on the number of times the system can oops or warn before it just panics and reboots.
• The TIOCSTI ioctl() operation will push data into a terminal device; that data will then be read as if were input typed by the user. As one might imagine, attackers find this operation useful. It seems that almost nobody else does, though. In 6.2, the kernel has gained a configuration option and sysctl knob that can disable this functionality entirely.

Internal kernel changes

• There is a new struct encoded_page type meant to encapsulate the idea of using the lower bits of a pointer value for related information. This type, was created by Torvalds to increase type safety and prevent the accidental dereferencing of an augmented pointer without stripping out the extra bits first. There is no documentation but this commit is easy enough to read.
• The venerable container_of() macro has a new sibling called container_of_const() that preserves the const quality of the passed-in pointer. In the merge message, Greg Kroah-Hartman explains this macro this way:

  The driver for all of this have been discussions with the Rust kernel developers as to how to properly mark driver core, and kobject, objects as being "non-mutable". The changes to the kobject and driver core in this pull request are the result of that, as there are lots of paths where kobjects and device pointers are not modified at all, so marking them as "const" allows the compiler to enforce this.

  So, a nice side affect of the Rust development effort has been already to clean up the driver core code to be more obvious about object rules.

• The minimum version of binutils needed to build the kernel has been raised to 2.25.

One thing that didn't make it this time around is support for linear address masking, an Intel feature that allows storing extra data in pointer values. Torvalds complained about how the feature was implemented and refused to pull the patches. So this feature, it seems, will have to wait at least another cycle before landing in the mainline.

Meanwhile, the "extensive changelog" award must certainly go to Christian Brauner, for this patch, which features 520 lines of explanation (not including the stack trace) for a one-line fix.

Normally, the 6.2 development cycle would be expected to come to a close on February 12 or 19. Torvalds suggested that the holidays might slow down this release cycle slightly; time will tell. Meanwhile, it is time to start finding and fixing bugs — once the kernel developers finish celebrating the holidays, of course.

Comments (17 posted)

Supporting unified kernel images for Fedora

The Fedora community is currently discussing a proposal to start supporting a unified kernel image (UKI) for the distribution; these images would combine several pieces that are generally separate today (e.g. initrd, kernel, and kernel command line). There are a number of advantages to such a kernel image, at least for some kinds of systems, but there is worry from some about where the endpoint of this work lies. There is a need to ensure that Fedora can still boot non-unified, perhaps locally built, kernels and can support other use cases that unification might preclude.

A feature proposed for Fedora 38 would add "phase 1" of UKI support; it was posted on behalf of feature owner Gerd Hoffmann to the distribution's devel mailing list on December 22. Currently, a new initial RAMdisk (initrd) containing files needed early in the boot process is built on the local Fedora system whenever a new kernel or other boot-relevant component is installed. But, since the Fedora private key is not present on the local system, the newly built initrd cannot be signed with it. So the goal is to move away from locally building an initrd, at least for some kinds of installations.

A unified kernel image is an all-in-one efi binary containing kernel, initrd, cmdline and signature. The secure boot signature covers everything, specifically the initrd is included which is not the case when the initrd gets loaded as separate file from /boot.

Main motivation for this move is to make the distro more robust and more secure.

The proposal notes that switching to UKIs for all use cases is not realistic, at least anytime soon. There are too many distribution features that rely on having a host-specific initrd and kernel command line, which does not mesh well with a plan that standardizes those things in a unified image. In fact, the updated version of the proposal on the wiki makes it clear that there are no plans to ever remove support for non-UKI systems. Phase 1 targets providing a UKI for virtual machines, with a stretch goal of switching the cloud images to UKIs. The general sketch for subsequent phases was laid out in the proposal, but more details for that are being gathered on the wiki.

While the primary target is x86_64, Hoffmann clarified that arm64 (aarch64) is also a possibility. The Arm world does not generally use UEFI or secure boot but there are still advantages: "a more robust initrd build and kernel update process is nice to have". The other two architectures that Fedora supports, PowerPC and s390, are not being considered for unified images, however.

Discussion

Much of the discussion about the feature centered around the perception that the first phase is just the nose of the camel under the tent; the worry is that UKIs will take over and eliminate other boot mechanisms, which would result in a less-functional Fedora. Neal Gompa had a lengthy and strongly worded response to the proposal. He is, at best, skeptical of the thrust of the UKI support effort because of the need for installations to incorporate a wide array of third-party drivers for graphics, storage, and networking. Those drivers need to work at boot time, but cannot be incorporated into a UKI; he has also encountered numerous problems trying to coax the existing secure boot stack into handling third-party drivers (and their keys). Since Fedora features tend to make their way into Red Hat Enterprise Linux (RHEL) eventually, he sees problems on the horizon:

In the UKI model, there's no way to incorporate early boot storage, network, and graphics drivers. This is *incredibly* common for RHEL administrators because there's a general acceptance of proprietary kernel modules from vendors these days. Even ignoring those, Red Hat's kernel feature support mindset is completely incompatible with UKIs, because RHEL does default-off rather than Fedora's default-on model for kernel features. We could debate until the cows come home on whether it's right or wrong, but their current mindset essentially means tons of common hardware becomes unsupported quite regularly. The ELRepo project is popular among RHEL folks because it restores those drivers and makes it possible to use RHEL on those machines through a combination of driver disks and kernel module packages.

But Daniel P. Berrangé disagreed, noting that the system extension (sysext) mechanism provides a means to support different configurations:

Yes, system extensions are one mechanism for making the initrd content more flexible when using UKIs. There would be one base layer [defining] the 90% common case, and a number of add-ons that can cope with niche use cases. This avoids the core UKI having to be huge and ship with every possible feature present.

Gompa had mentioned system extensions in his message, but said that adding them into the mix "eliminated the value of UKIs". There are still multiple advantages to using UKIs with system extensions, Berrangé said:

The most critical is that the initrd content and cmdline is covered by the SecureBoot signature. This remains true even with system extensions, as such extensions would be signed too. They do not [necessarily] need to be signed by the OS vendor, they could use a 3rd party SecureBoot signing key, or the users' own key. That is TBD and not something we're actively considering - its not even mentioned in phase 2/3 ideas in the change proposal.

A further benefit of UKIs is supportability, since we know exactly what content is in the initrd that user is booting. With locally generated initrds there are an arbitrary number of combinations and many ways for the initrd creation process to go wrong, which are hard for Fedora maintainers to understand and debug.

Secure boot

Gompa's concerns run deeper, though. He is clearly frustrated with the state of Linux support for secure boot, UEFI, and more. Berrangé had mentioned confidential computing in the cloud as one of the main use cases driving the proposal. Today on Azure, Ubuntu virtual machines can be booted with root filesystems that are encrypted such that they can only be decrypted by the specific instance, but Gompa was not particularly impressed: "Linux's model for supporting confidential computing is not user-friendly, so I expect low adoption and resistance once the flaws become apparent to would-be users."

Secure boot support has been a part of the Linux world for a long time at this point and Simo Sorce noted that it is the boot security mechanism available on much of the hardware sold today. He said that Gompa was being "unnecessarily negative" because the proposed feature is meant to increase the user-friendliness level for secure boot and related features.

Gompa complained further about the lack of friendlier support for secure boot, but Lennart Poettering said that Linux itself is the source of many of the problems. The proposal being made is a step toward making things better, though there are still problems to be solved; "Linux never solved the initrd hole so far, but that's not really the fault of SecureBoot, it's the fault of Linux". Other operating systems are delivering a trusted-boot experience that is not possible for Linux, "this proposal is an attempt to finally do something about this".

The difference between Windows and Linux boot security is stark, Luca Boccassi said.

You meant to say "light years ahead" here - it is not even funny anymore how far behind Linux is to Windows w.r.t. security, especially in the boot process. We have been playing catch-up for 10 years and are nowhere near done.

Boccassi and Poettering were both part of the Image-Based Linux Summit, where a lot of effort was made to help close that gap. There is progress, Boccassi said, but perhaps things are still moving too slowly:

Now we need the wide adoption, and this proposal is one timid step forward in that direction. The fact that in 2022 there is still no mainstream distro that has closed the glaring security gaping hole of writable, untrusted initrd (yes some distros have non-default specialized flavours for this, but it's niche) should be a source of embarrassment for anybody who works on OS development. It is a difficult problem, but by no means impossible, and it really ought to have been fixed at least for the generic use case by now. We need to get this sorted at long last.

There were others posting about concerns in trying to boot systems without access to the initrd and kernel command line, such as Björn Persson and Dennis Gilmore; there are certainly many reasons a Fedora user might want to be able to continue booting the way it is done currently. But, as Sorce put it, "nobody is taking away the initrd way"; that option will remain even after all of the phases of the UKI project run their course.

Overall, it is a feature that should only affect those who are interested in using it; there is still a huge amount to do before there can be any consideration of bare-metal Fedora installs using a UKI, for example. But, for some use cases, the secure boot problems need to be resolved and, at least so far, other solutions do not seem to be on offer. The Fedora Engineering Steering Committee (FESCo) ticket for the feature currently shows a half-dozen "+1" votes, including Gompa ("I feel like we are going to regret this, but... "). Thus it seems likely that Fedora 38 will see the initial support for UKIs when it arrives in April.

Comments (23 posted)

Not coalescing around None-aware

The wish for a "None-aware" operator (or operators) is longstanding within the Python community. While there is fairly widespread interest in more easily handling situations where a value needs to be tested for being None before being further processed, there is much less agreement on how to "spell" such an operator (or construct) and on whether the language truly needs it. But the idea never seems to go away, with long discussions erupting every year or two—and no resolution really in sight.

PEP 505

We looked at the idea back in 2018, but PEP 505 ("None-aware operators") goes back to 2015; it seems likely that the idea had been discussed well before that as well. At the time of our article, the Python community was still working on figuring out how it would be governed after Guido van Rossum stepped down from his benevolent dictator role. PEPs were not being accepted or rejected because of that and PEP 505 was deferred.

In Python, it is not uncommon to see code like the following (taken from the more extensive examples in our earlier article):

    if x is None:
        x = 42

The PEP proposes the "??" None-aware (also known as null-coalescing) operator. It returns the left-hand side if it is not None, otherwise the right-hand side is evaluated and returned. So that code above could be replaced with:

    x = x ?? 42

There is also an "augmented assignment" version ("??=") and two operators that apply the None-awareness to indexing ("?[]") and to attribute access ("?.") proposed as well:

    x ??= 42   # same as above
    x?['foo']  # retrieve dict entry 'foo' if x is not None
    x?.foo     # retrieve attribute foo if x is not None

In the last two examples above, if x is None, then the expression evaluates to None as well.

The PEP has come up again a few times since the Python governance upheaval put it on the back burner, including in a Python discussion-forum thread back in April 2021. Kumar Aditya asked about adding a null-coalescing operator; Steven D'Aprano pointed to the PEP and some of the previous discussions. Along the way, PEP co-author Steve Dower said that his plan for the PEP is to drop it "because I now think the additional syntax would not make Python cleaner or a better language".

The PEP came up again in October 2021 when Doug Swarin raised it on the python-dev mailing list. As with Aditya and others, Swarin pointed to the existence of similar operators in other languages (such as JavaScript) and said that it would "result in a significant reduction in boilerplate code" testing for None. He suggested that the indexing ("maybe subscript") and attribute-access ("maybe dot") operators might actually lead to a decrease in code readability, so perhaps they could be removed from the PEP, though he was not personally in favor of doing so. He had also created a pure-Python implementation that he made available for testing.

Van Rossum, who has generally had a favorable inclination toward the PEP all along, thanked Swarin and said that he hoped more discussion would "help convince the Steering Council to accept it". The conversation proceeded along familiar lines, with some strongly advocating for at least the addition of ?? and ??=, while others were expressing concerns about the readability. The specter of "Perl-ification" was present as well; to some, adding more operators of this sort just makes the language look like Perl (or, worse still, line noise).

Revival

On December 11, "yingshaoxo" revived the forum thread that started 18 months earlier and the discussion was off and running again. Yingshaoxo advocated adopting the feature, while noting that multiple languages (JavaScript, Flutter, Kotlin, and C#) already have it. Cameron Simpson noted that there is an existing way to handle the example yingshaoxo posted, but that it does not work in all cases:

    # proposed new syntax
    new_variable = a_variable ?? 'Hello Python'

    # "same" thing with existing syntax
    new_variable = a_variable or 'Hello Python'

That relies on a_variable evaluating to a false value, which is the case for None, but it is also the case for the empty string, 0, and a few other values. D'Aprano pointed out that the same argument was made to resist adding a ternary operator to the language, until PEP 308 ("Conditional Expressions") was abruptly adopted because of the problems that can occur with expressions that unexpectedly evaluate to false:

For many, many years we rejected every request for a ternary if operator with the argument that you can just write

    condition and if_true_value or if_false_value

E.g. len(mylist) > 0 and mylist[0] or "default". Can you see the bug? (Hint: what if the value of mylist[0] is, say, the empty string?).

What followed was a long discussion on "readability". "Fancidev" asserted that the conditional version was more readable:

"y if x is None else x" is more readable than "x ?? y".

Since code is written once and read 100 times, it's probably worth the extra typing.

As might be guessed, others disagreed. Tzu-ping Chung said that the conditional form "requires more mental attention", but did recognize that there is a potential hurdle in needing to learn what ?? means. D'Aprano said that those coming to Python from other languages often complain that the conditional construct in Python is unreadable. "So maybe readability depends on what you are used to."

Fancidev pointed out that ?? can only be applied to a subset of the situations where the more general conditional expression can be used; it requires learning about the operator but can then only be applied to None tests. "Vladimir" agreed, but linked to a section of PEP 505 that argues for ?? because it is more concise. The PEP has an example of initializing default values:

    data = [] if data is None else data

    # or

    data = data if data is not None else []

Either of those will work, but the first puts the operands in an "unintuitive order", while the second is a bit longer and repetitive (i.e. "data if data"). Using the proposed operator, it would simply be:

    data = data ?? []

To Fancidev, those examples just reinforce their belief that the conditional version is more readable, but Chris Angelico argued that "readability" is totally subjective, at least as it is used in discussions like these:

I personally suspect that some people consider something "readable" on the basis that "I can understand what it does based on my pre-existing knowledge of what Python can already do" (meaning that new syntax is ALWAYS less 'readable' than a verbose form that already exists), and other people consider something "readable" on the basis that it is compact and expresses a thought that can be fit into a larger "sentence" or "paragraph" (meaning that a new syntax is almost always more 'readable' than the more-verbose form that already exists).

Coalesce

Marc-André Lemburg took a different tack on the idea. He pointed to the PostgreSQL COALESCE() function as a possible model for "an explicit and elegant way" to solve many of the problems that PEP 505 describes. He suggested adding a Python builtin along the following lines:

coalesce(*args, /, check=None, logic=False)

Return the first value from args which is not the check object (defaults to None). If check is a callable, return the first value from args which check(value) is False. logic may be set to True, to swap the test logic.

The "/" in the argument list separates the positional-only parameters (args in this case) from those that can be specified by position or keyword (check and logic). He goes on to list a few operators as possibilities for the check values, such as a new list.isempty() or the existing math.isnan().

But there is a problem with that approach, as several people pointed out: there is no ability to short-circuit the evaluation of some of the arguments. Using a conditional expression (or the proposed ?? operator) would not evaluate the "else" argument if the value is not None. As Fancidev put it: "it does not lazily evaluate the arguments, which I imagine will be a necessary requirement". Peter Suter added some other ways where the operator approach is superior, including:

Chaining is much clearer and less error prone with operators:

(override ?? fallback).name ?? default
coalesce(coalesce(override, fallback).name, default)

Meanwhile, the "debate" over readability continued, much of it between Angelico and Vladimir, seemingly without changing any minds. In fact, early on in that part of the discussion, Angelico hit the nail on the head. In a message linked above, he decried the idea of "readability" because: "Everyone has their own definition, never backed by any sort of actual studies, and nobody ever changes their mind based on other people's examples." Eventually, after a good bit of back and forth on readability, the thread was locked on December 17.

As should be clear, though, it is a topic that comes up often; until the PEP is either accepted or rejected, it will undoubtedly come up again. Even if the PEP is rejected, it will not be a huge surprise to see the idea pop up again; it has obvious utility and other languages have something similar, which will make it that much more desirable. The arguments against the simplest form (just ?? and ??=) seem to mostly boil down to being a barrier to learning or understanding the language. But, as D'Aprano put it, Python has features for both casual and more serious programmers:

The beauty of Python is that it is accessible to casual programmers. You don't have to use null-coalescing operators any more than you have to write classes, or use closures, or use threads.

But we didn't let those casual programmers stand in the way of Python getting classes, closures, threads, async, regexes etc. Let the casual programmers continue to write using the basic features, and the power users use the power features.

It is hard to predict what will happen from here, especially given that the PEP's authors are no longer pushing it; it seems likely that some other core developer(s) would need to sponsor it for the newly elected steering council to even consider it. If that happens, perhaps an alternate spelling can be found to reduce the unhappiness with the "ugliness" of ??. A reduction in scope (eliminating ?. and ?[]) might also help the prospects of the PEP; down the road those additions could be considered again. One way or another, getting PEP 505 out of limbo would help clarify things quite a bit.

Comments (87 posted)