Docker is a very convenient abstraction layer that allows ease and consistency of deployment of software. However, it requires that the image platform matches the platform on which we want to run the software. Luckily, docker has multi-platform support which allows to both run images from different platform and, via buildx, create images for variety of platforms. There is a small downside to it - it uses QEMU to, as name may suggest, emulate other platforms, which entails one downside - performance penalty, both when running the image as well as when creating it.
During recent preparation of Tigase XMPP Server 8.3 release I faced an issue where, due to introduction of jlink into our build pipeline to make the images smaller and more lean, additional processing made creation of multi-platform images virtually impossible.
Fortunately, buildx tool is very versatile an allows using multiple builders to create images and in addition, those builders can be remote so it’s possible to take advantage of computing instances from cloud providers to build images for platforms not native to the machine on which we run the build making the build speed native-like.
Preparing remote environment
There is nothing all that special when it comes to remote machine preparation - it has to have docker installed (follow Install on Linux guide or one dedicated to partiular distribution used). One caveat is to make sure that it’s possible to use docker without sudo which is easily accheved by adding user to docker group:
$sudo gpasswd -a $USER docker
(restart of the shell session required afterwards).
Machine has to be accessible via ssh. Another caveat - because it’s not possible to specify key used it has to either be one available via SSH Agent or one of the following files: id_rsa, id_ed25519, id_ecdsa, id_dsa or identity under ~/.ssh.
After everything is set up it’s possible to check if everything is correct byt executing info command:
docker -H ssh://<usernane>@<hostname> info
which should give output similar to the one below:
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.9.1)
compose: Docker Compose (Docker Inc., v2.12.2)
dev: Docker Dev Environments (Docker Inc., v0.0.3)
extension: Manages Docker extensions (Docker Inc., v0.2.13)
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
scan: Docker Scan (Docker Inc., v0.21.0)
Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 2
Server Version: 20.10.22
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.11.0-1022-aws
Operating System: Ubuntu 20.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.901GiB
Name: ip-172-31-48-98
ID: X6BF:XU3T:QUBB:M5NY:IGEV:UAQL:6PJ5:GTGO:EBDX:73AS:UD5X:IY5A
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Preparing local buildx
The catch here is to create a builder that has two nodes: one local for one architecture and then another one for another architecture.
Let’s add new local builder for arm64 platform (I’m running the build on MacBook with M1 chipset):
docker buildx create --node local --name local-remote-builder --driver docker-container --platform linux/arm64
Checking available builders with docker buildx ls should give following output
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
local-remote-builder docker-container
local unix:///var/run/docker.sock inactive linux/arm64*
Let’s add remote machine to builds for x86 architecture. The caveat is to use --append parameter to add remote node to just created builder:
docker buildx create --append --name local-remote-builder --node remote --driver docker-container --platform linux/amd64 ssh://<usernane>@<hostname>
Available nodes at this time should include our remote builder:
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
local-remote-builder docker-container
local unix:///var/run/docker.sock inactive linux/arm64*
remote ssh://<usernane>@<hostname> inactive linux/amd64*
Builders are still inactive so it’s essential to make sure they are properly booted before execution with
docker buildx inspect --bootstrap --builder local-remote-builder
command, yielding following ouput if everything went correctly
[+] Building 12.2s (2/2) FINISHED
=> [local internal] booting buildkit 2.9s
=> => pulling image moby/buildkit:buildx-stable-1 2.3s
=> => creating container buildx_buildkit_local 0.5s
=> [remote internal] booting buildkit 7.7s
=> => pulling image moby/buildkit:buildx-stable-1 1.1s
=> => creating container buildx_buildkit_remote 6.4s
Name: local-remote-builder
Driver: docker-container
Nodes:
Name: local
Endpoint: unix:///var/run/docker.sock
Status: running
Buildkit: v0.10.5
Platforms: linux/arm64*, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
Name: remote
Endpoint: ssh://<usernane>@<hostname>
Status: running
Buildkit: v0.10.5
Platforms: linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/386
The only remaining thing is to set newly created and configured builder to be used by default via
docker buildx use local-remote-builder
and with that our builder list should look like this (nodes listed as running and builder annotated with asterisk indicating it’s the default one):
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
local-remote-builder * docker-container
local unix:///var/run/docker.sock running v0.10.5 linux/arm64*, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
remote ssh://<usernane>@<hostname> running v0.10.5 linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/386
Building Tigase XMPP Server image
With everything in place, building latest version of Tigase XMPP Server 8.3 is a matter of executing buildx build command with desired target platforms:
docker buildx build --platform linux/amd64,linux/arm64 -t tigase/tigase-xmpp-server:${VERSION} -f ${VERSION}/Dockerfile --no-cache ${VERSION}/
giving us image in less than a minute.
]]>
