Hopefully the auto-discovery works as well:

<link rel="alternate" type="application/rss+xml" title="davidjenei.com" href="/feed.xml">

Still using my handcrafted, Makefile-based static site generator. The feed is valid for now, though I suspect there is still some room for improvement.

More posts soon.

Schopenhauer says when we read we effectively follow someone else’s thought process. But if we stay in permanent read-only mode, we gradually lose our own capacity for thinking.

Fair point, but not everyone has hours to spare detailing new ideas every day. Writing doesn’t have to be ambitious and time consuming though.

My goal here is recording and clarifying my own thinking. Some say that the best way to deeply understand something is to write about it.

I recently got introduced to some indieweb principles which motivated me to create a place for myself on the internet. Gardening your own corner of the internet is fun, and I should definitely write more about this to encourage others.

As for now, I’m not starting any blog challenge but I try to keep practicing writing regularly. I hope it turns out to be useful to anyone else.

References:

• HN: Why it’s great to write blog
• HN: Why don’t you have a blog

Music

Tesseract - Legion

Djent meets Muse. Exceptional vocal performance by Daniel Tompkins.

TV

State of the Union

Before appointments with their (marriage) therapist, Tom and Louise meet at a pub to try to get their stories consistently coherent.

Fast-paced comedy hits hard with dense 10-minute episodes, even if you are unmarried.

Software engineering

-10x engineer

Give zero consideration to how your system design will evolve over time. Alternatively, drive your team obsess over architecture decisions so that they don’t have time to test their hypotheses.

+1: Build software that is impossible to change. A reminder from Philosophy of software design: Change amplification is a symptom of complexity which is that a seemingly simple change requires code modifications in many different places.

Python isinstance() function

How to avoid using isinstance().

Pkl: configuration as program

Configuration file formats lack the expressiveness of a programming language. This means we don’t have mechanisms to avoid repetetions or validate data easily.

By generating this configuration with Pkl, you can reduce verbosity and increase maintainability through reuse, templating, and abstraction.

Embedded

SIL-4 RTOS ThreadX open sourced under MIT licence

Getting started with ThreadX

Now

Work in progress: Promoting software artifacts, organising integration tests and environments.

The official Azure IoT Edge documentation has instructions for installing the runtime with package managers on Linux. But on some constrained hardware, you may want to consider building a minimal Linux without package managers or the prebuilt package is simply not suitable for your system. In this post I would like to show how to compile a Linux image from scratch with Azure IoT Edge runtime. I will also test the service and deploy an example application to the board in a Linux container.

For building Linux, I will use the Yocto framework, so first make sure that you have the required packages for your system.

Toradex uses the repo utility to manage multiple git repositories for their BSP. It is available for many distros, so we can install simply install with our package manager

david.jenei@yocto-env:~$ sudo apt install repo

Now we can set up the build environment. Follow this Toradex tutorial and check out the bsp layers.

david.jenei@yocto-env:~/oe-core$ repo init -u http://git.toradex.com/toradex-bsp-platform.git -b LinuxImage3.0 -m default.xml
david.jenei@yocto-env:~/oe-core$ repo sync

Open local.conf and set your machine. In this demonstration I am working with a Colibri i.MX6 board:

MACHINE ?= "colibri-imx6"

Append this line to the config to accept Freescale EULA, so the build will complete on your first try:

ACCEPT_FSL_EULA = "1"

Build the default console image with bitbake:

david.jenei@yocto-env:~/oe-core/build$ bitbake -k console-tdx-image

When the build completes, you can find the binaries in the deploy directory:

david.jenei@yocto-env:~/oe-core/build$ ls deploy/images/colibri-imx6/

I am going to assume here, that you already have bootloader installed on your board (otherwise you have to enter recovery mode and use imx_loader and execute u-boot.imx).

Connect to the debug serial and get a u-boot shell. Also connect to the USB OTG port, and attach the eMMC boot partition to your PC as an USB mass storage.

Colibri iMX6 # ums 0 mmc 0.1
UMS: LUN 0, dev 0, hwpart 1, sector 0x0, count 0x1000
USB EHCI 1.00

Write the SPL and the u-boot.img to the eMMC with dd:

Make sure that you select the right output device here (/dev/disk2, /dev/sda etc. depending on your platform).

sudo dd if=./SPL-colibri-imx6-2019.07+gitAUTOINC+e1cbe8c74e-r0-spl-2019.07+gitAUTOINC+e1cbe8c74e-r0 of=/dev/disk2 bs=1k seek=1 conv=sync
sudo dd if=./u-boot-spl-2019.07+gitAUTOINC+e1cbe8c74e-r0.img of=/dev/disk2 bs=1k seek=69 conv=sync

Reset the board, and check the build time stamp:

Colibri iMX6 # reset
resetting ...

Industrial temperature grade DDR3 timings, 64bit bus width.
Trying to boot from MMC1

U-Boot 2019.07-3.0.3+g694e2136ee (Mar 05 2020 - 11:14:09 +0000)

Now that we have our fresh bootloader installed on the board, we can try to compile Azure IoT Edge. Edit local.conf once again, and add the following:

DISTRO_FEATURES_append = " virtualization"
IMAGE_INSTALL_append = " iotedge-daemon iotedge-cli cgroup-lite rng-tools procps"

We enable virtualization layer as a distro feature and install a few packages. With cgroup-lite we can mount cgroups on startup, we need rng-tools.

Run bitbake again and compile our eMMC user partition image now:

david.jenei@yocto-env:~/oe-core/build$ bitbake -k console-tdx-image

Enter UMS mode again, and write the contents of the image.

Colibri iMX6 # ums 0 mmc 0

End of chapter #1. To be continued.

If you wish to run all applications in Linux containers, you will probably aim for a lightweight distribution instead of a full featured one. Minimal container operating systems like CoreOS, RancherOS and others were created for this purpose and you can use them on regular VMs.

For IoT devices, there is Ubuntu Core with off-the-shelf Raspberry and Dragonboard support. But if your hardware is not supported or you’re not planning to use the snap package format of Ubuntu, you can always compile your own distribution with Buildroot or Yocto.

In this tutorial, I will show how to build a custom Linux OS with container support for an ARM board and how to deploy some simple apps in containers.

For managing containers, I will use Podman instead of Docker as an experiment. I found during my early tests that starting containers with Docker took ages on my board. But instead of playing around with different container runtimes, I chose to test Podman, so later I can move to Kubernetes or run rootless containers.

First, set up a basic Yocto environment and clone the poky reference distribution repository.

git clone -b thud git://git.yoctoproject.org/poky

Configurations in the Yocto build system are stored in layers. The first layer we need is meta-sunxi, which is the BSP layer for my Allwinner based Orange Pi board, so let’s download this layer.

git clone https://github.com/linux-sunxi/meta-sunxi.git

We also need meta-openembedded, meta-virtualization, meta-security layers to compile Podman and the container runtime.

git clone -b thud git://git.openembedded.org/meta-openembedded && \
git clone -b thud git://git.yoctoproject.org/meta-virtualization && \
git clone -b thud git://git.yoctoproject.org/meta-security

Before adding these layers to the build system, we have to setup the build directory. Run the poky environment setup script which will automatically generate your /build folder. Note that this command will change your working directory.

source poky/oe-init-build-env

Now we can set up the configuration files in the build directory. Enable the following layers from the previously cloned repos with the bitbake-layers command, so we don’t have to edit bblayer.conf by hand.

bitbake-layers add-layer ../meta-sunxi/ \
    ../meta-openembedded/meta-oe/ \
    ../meta-openembedded/meta-filesystems/ \
    ../meta-openembedded/meta-python/ \
    ../meta-openembedded/meta-networking/ \
    ../meta-openembedded/meta-perl/ \
    ../meta-virtualization/ \
    ../meta-security/

For the time being, I am going to add every customisation to the local.conf file. This file stores the local or user specific configuration. Later we can create a new configuration layer with recipes for this image to store our changes permanently.

First we set the target device in local.conf. I checked the exact machine name for my board in the BSP layer (meta-sunxi/conf/machine/) and modified the MACHINE parameter accordingly.

# This sets the default machine to be qemux86-64 if no other machine is selected:
MACHINE ??= "orange-pi-one"

Linux container isolation is built on kernel features like cgroups and namespaces but the kernel configuration for my board did not enable these by default.

Instead of editing the kernel configuration in the BSP layer, settings in meta-virtualization can extend our configuration to enable kernel features to support containers. The meta-virtualization layer implements this with defining a new DISTRO_FEATURE. Distro features can enable functionality across multiple packages. So add this to our local.conf:

DISTRO_FEATURES_append = " virtualization"

Finally select the packages we want to build into the image.

IMAGE_INSTALL_append = "  podman podman-compose crun cgroup-lite rng-tools \
                          procps ca-certificates python3-setuptools \
                          python3-pyyaml python3-json"

Here is a little explanation for this part:

• podman: Docker compatible container manager tool for libpod. We will use this command to manage containers just as with Docker.
• podman-compose: docker-compose implementation with podman backend. We can run multi container applications and store container configuration in yaml files with this tool.
• crun: OCI Container Runtime written in C. Originally it was written for supporting cgroupV2. This runtime is a bit more leightweight than runc.
• cgroup-lite: mounts cgroups on startup.
• rng-tools: on my 4.19 kernel hardware rng is not yet mainlined for the Allwinner H3 processor, so I need jitter entropy to keep my startup time reasonable and don’t wait for /dev/random too long.
• ca-certificates: podman wants to check the server certificates when pulling new images
• some python3 packages: podman-compose depends on these, but were not included as dependencies the recipe.
• e2fsprogs-mke2fs: we also need mkfs to create a data partition for the containers

Now we need to specify the image recipe we want to build. As a starting point I used the core-image-minimal as the smallest default image. Use the bitbake command to start the build.

bitbake core-image-minimal

This image configuration is the reason why we added all these seemingly random packages to the build. It is a minimal default configuration, so we had to add these otherwise essential tools like rngd to the image.

After a few hours hopefully the build completes, and you can find the sd card image in build/tmp/deploy/images

ls -ailh build/tmp/deploy/images/orange-pi-one/core-image-minimal-orange-pi-one-20200310152758.rootfs.sunxi-sdimg

Write this image to an SD card and boot the board with the new image. Get a terminal using a serial port or SSH.

Before starting a container with podman, we have to modify libpod.conf.

root@orange-pi-one:~# vi /etc/containers/libpod.conf

Set crun as a container runtime and use cgroupfs instead of systemd to manage cgroups.

cgroup_manager = "cgroupfs"
runtime = "/usr/bin/crun"

Now we can try running our first container.

root@orange-pi-one:~# podman run hello-world
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob b6206e5d545d done
Copying config cfdb1bf11e done
Writing manifest to image destination
Storing signatures

Hello from Docker!
...

With this build, your root partition has very little free space available. If you would like to store your containers and container images on another partition, edit storage.conf.

root@orange-pi-one:~# vi /etc/containers/storage.conf

Point runroot and graphroot locations to your data partition.

# Temporary storage location
#runroot = "/var/run/containers/storage"
runroot = "/mnt/data/storage"

# Primary Read/Write location of container storage
#graphroot = "/var/lib/containers/storage"
graphroot = "/mnt/data/storage

(Now you can create a new partiton with fdisk and mkfs, then edit your fstab to mount your partition on startup. Or you can use wic to create an sd image with an empty data partition)

Test podman-compose with a simple configuration. Create a new compose file.

vi docker-compose.yml

Define an example service.

version: '2.0'
services:
  hello-world:
     image: hello-world

And start this configuration.

root@orange-pi-one:~/test# podman-compose up
podman pod create --name=test --share net
4dc74923c2314d0ef0379f90e0246fab5d41896ae815ee33287b5e6360c40fac
0
podman create --name=test_hello-world_1 --pod=test -l io.podman.compose.config-hash=123 -l io.podman.compose.project=test -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=hello-world --add-host hello-world:127.0.0.1 --add-host test_hello-world_1:127.0.0.1 hello-world
ea9f6fd6fce3af419d1306b94e9de21f49bb323a870ed053abf4e5a8d9ae1b0c
0
podman start -a test_hello-world_1

Hello from Docker!

In this example, we used Yocto to build a small Linux image from scratch and installed essential packages to get started with containers on an ARM devices. With the lightweight container runtime and the minimal configuration the image still has a significant size (~250MB) but it is a good starting point for testing.

The next steps would be saving this configuration in a separate Yocto layer, but this is a topic for another tutorial.

• Locking Engineering Principles - Daniel Vetter
• Apple Silicion boot flow - Asahi Linux wiki
• The quantum state of Linux kernel garbage collection CVE-2021-0920 - Google project zero
• Keeping a project bisectable - André Almeida
• Hunting bugs with git biscect - Memfault

Hardware

• $52 7.5-inch E-paper display connects to ESP32 boards - CNX

Tech

• Where Are The Autonomous Cars? - Semiconductor Engineering
• IoT Signals: Manufacturing Spotlight - Microsoft

Book

• The Cathedral and the Bazaar - Eric S. Raymond

The Manufacturing Delusion

• Less is more agile - Gerald Benischke
• Your Makefiles are worng - Jacob Davis-Hansson

Hardware

• SFP GPON module firmware customisation - Github
• VW Golf Steering ECU reversing - Willem Melching

excellent writeup, very detailed.

• Hyundai Ioniq firmware update - programmingwithstyle

this monthly is a car hacking special edition

• TPM Sniffing attacks - Jos Wetzels

Book

• Thinking, Fast and Slow - Daniel Kahneman

• The Era of Visual Studio Code - Roben Kleene
• Phillip on EmbeddedFM

Hardware

• Welcome to libcamera v0.0.1
• Disabling Intel management engine
• Enhanced Cybersecurity and FIPS 140-2 Compliance using NXP i.MX 8X

There are plenty of tutorials out there how to do this on a Linux machine. But if you prefer running LXC containers as workstations rather than regular VMs, you may need to change a few things like tweaking the uid/gid mapping.

This short tutorial will show you how to join a newly created LXC container to an Active Directory and how to configure SSH to login with your domain account.

I will assume that you already:

• have a running Active Directory in your network
• have an administrator account for the AD server
• installed LXC on your Linux host

Let’s start with launcing a new lxc container:

lxc launch ubuntu:18.04 ad-join-test
Creating ad-join-test
Starting ad-join-test

Get a shell inside:

lxc exec ad-join-test -- /bin/bash
root@ad-join-test:~#

On Ubuntu or similar distros, we can use System Security Services Daemon (sssd) for connecting to remote authentication providers. The documentation recommends using an automated tool to configure it and here I am going to use the realm command-line tool for quick enrollment.

But first we need to install realmd and a few more packages:

root@ad-join-test:~# apt update && apt install realmd sssd-tools
sssd libnss-sss libpam-sss adcli packagekit

Now we can try to discover our realm:

root@ad-join-test:~# realm discover eilabs.local

In case of no answer here, make sure that you can resolve the name of domain controller:

root@ad-join-test:~# nslookup -type=srv _ldap._tcp.eilabs.local

I couldn’t resolve the domain name of my domain controller in the first place, because we used .local as top level domain, which is nowadays intended to use in multicast DNS, so I had to change the systemd-resolver configuration.

If realm can discover the ldap service, use your administrator account to join. Realm is a silent program, so I am going to use –verbose to see the details of the process:

root@ad-join-test:~# realm join eilabs.local --user=david.jenei --verbose

Now try to check connection to the domain controller and look up your user passwd entry:

root@ad-join-test:~# getent passwd david.jenei@eilabs.local

If you try to login now, you see that the system is using the pam_sss module now as it should, but there is this error in the auth.log.

pam_sss(login:auth): received for user david.jenei@eilabs.local: 4 (System error)

Changing the debug log level in the sssd reveals that there is problem with the uid. If you would like see the error message, put the debug directive under your domain’s section in the sssd configuration (/etc/sssd/sssd.conf) and restart sssd.

Here you can see, that my uid is way out of range, and the default id mapping in sssd can’t map those into the allocated uid map:

root@ad-join-test:~# cat /proc/self/uid_map
         0     100000      65536
root@ad-join-test:~# getent passwd david.jenei@eilabs.local
david.jenei@eilabs.local:*:1149001181:1149000513:Jenei,
    David:/home/david.jenei@eilabs.local:/bin/bash

Add these lines to the sssd configuration for your domain to limit idmap range and to map uid/gid into the available space.

ldap_id_mapping = True
ldap_idmap_range_min = 10000
ldap_idmap_range_max = 50000
ldap_idmap_range_size = 1000

I also changed the backup homedir line to a new home directory format and enabled login without fully qualified names:

use_fully_qualified_names = False
override_homedir = /home/%d/%u

If you restart sssd after changing the configuration, it will fail, because we also need to manually clear the cache:

root@ad-join-test:~# rm -rf /var/lib/sss/db/*

Restart sssd service and login with your account:

root@ad-join-test:~# systemctl restart sssd
root@ad-join-test:~# login
ad-join-test login: david.jenei
Password:
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-88-generic x86_64)

Run pam-auth-update and enable home directory creation on login. Next time you login, your home directory will be created automatically.

root@ad-join-test:~# pam-auth-update

Finally edit /etc/ssh/sshd_config and enable password login:

PasswordAuthentication yes

Restart the ssh daemon and now you can use SSH with your AD user name to login to your container.

root@ad-join-test:~# systemctl restart sshd
root@ad-join-test:~# ssh david.jenei@localhost

For more information, you can check out the details of uid mapping in LXD here.