Docker and the OCI container ecosystem

Posted Jul 26, 2022 20:31 UTC (Tue) by Cyberax (✭ supporter ✭, #52523)
Parent article: Docker and the OCI container ecosystem

One important part of containers is _building_ the images.

I personally am completely disgusted by Dockerfiles, but it doesn't look like there are any mature alternatives out there. Has this changed recently?

to post comments

Docker and the OCI container ecosystem

Posted Jul 26, 2022 20:49 UTC (Tue) by jordan (subscriber, #110573) [Link] (6 responses)

Lately I've been getting into Earthly but if you are disgusted by Dockerfiles you probably won't like it; it's somewhat like a cross between a Dockerfile and a Makefile.

Buildah offers a more shell-driven workflow - it's kind of like an "exploded" Dockerfile. Instead of writing your build steps in a Dockerfile, each step is a separate buildah command, which you can run interactively, or via a shell script, or via whatever else.

I've played a bit with ansible-bender, which builds on top of Buildah and seems like an entirely more reasonable way to build a container, but it needs to start with a base image that has a Python capable of running Ansible.

Buildpacks is higher-level and works kind of like Heroku; it detects what language your application is written in, and then picks appropriate builder and base images. It can rebase application images on top of newer base images without rebuilding the whole application.

Docker and the OCI container ecosystem

Posted Jul 26, 2022 21:03 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

Thanks for Earthly!

I actually like it a lot from reading the docs. It seems like a small, but practical improvement over Docker. At the very least they fixed the moronic behavior of Docker's COPY command. For those who don't know, "COPY src1 src2 target" in Docker means "cp src1/* src2/* target", making it impossible to copy multiple paths as one layer.

Yeah, their examples still have the "RUN apk --upgrade" nonsense that makes builds unreproducible. But I don't think it's possible to work around this without way more complex infrastructure.

The last time I looked, buildah was not working well on macOS and Windows. Has this changed?

> It can rebase application images on top of newer base images without rebuilding the whole application.

I have my own bag of tricks for that. For example, I copy go.mod/go.sum or Conanfile in a separate layer before the application and make sure all deps are built before copying in the source code.

Docker and the OCI container ecosystem

Posted Jul 26, 2022 21:11 UTC (Tue) by jordan (subscriber, #110573) [Link] (3 responses)

> Yeah, their examples still have the "RUN apk --upgrade" nonsense that makes builds unreproducible. But I don't think it's possible to work around this without way more complex infrastructure.

I've seen a project that implements package version lockfiles with apt, but the name escapes me at the moment. At a previous employer, we had internal Debian "mirrors" maintained with aptly that held on to every version of the package they'd ever seen, and explicitly specified the version of each package in a manifest file. As you say, more complex infrastructure.

> The last time I looked, buildah was not working well on macOS and Windows. Has this changed?

I don't think it has built in "remote client to a managed VM" stuff like Podman and Docker have gone out of their way to add to their stuff. I bet you could run it in a privileged container on a Podman/Docker on Windows/macOS setup, though.

Docker and the OCI container ecosystem

Posted Jul 26, 2022 21:21 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> At a previous employer, we had internal Debian "mirrors" maintained with aptly that held on to every version of the package they'd ever seen, and explicitly specified the version of each package in a manifest file. As you say, more complex infrastructure.

There is a Debian snapshot service ( https://snapshot.debian.org/ ) that preserves all the Debian history. So you can just refer to timestamped sources in your APT config. E.g.: https://snapshot.debian.org/archive/debian/20220720T151841Z/

I'm not sure this service can survive if more people start using it, though. It's pretty slow as it is :( We pipe it through our own proxy that basically stores every artifact, to make sure we don't bring it down.

Docker and the OCI container ecosystem

Posted Jul 26, 2022 21:24 UTC (Tue) by jordan (subscriber, #110573) [Link] (1 responses)

Sure, but using Debian snapshots would mean that you'd have to take all the updates in the snapshot that you moved to at once, and that you'd have to take updates in the order they were supplied upstream. Having a packrat mirror that holds on to all the versions gives you more flexibility in deciding what you want to update and when.

Docker and the OCI container ecosystem

Posted Jul 26, 2022 21:34 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> Sure, but using Debian snapshots would mean that you'd have to take all the updates in the snapshot that you moved to at once

Yeah, but this is usually OK. It also makes it easier to audit dockerfiles to check if they cover all CVEs in the base Debian image.

We also have a script that checks if an image contains packages that are different between two snapshots, this helps to automate "empty" version bumps. Not perfect, but it helps.

We also tried Nix that gives strong reproducibility gurantees, but it wastes way too much time on rebuilding everything.

Docker and the OCI container ecosystem

Posted Jul 27, 2022 9:00 UTC (Wed) by cortana (subscriber, #24596) [Link]

Buildpacks is higher-level and works kind of like Heroku; it detects what language your application is written in, and then picks appropriate builder and base images. It can rebase application images on top of newer base images without rebuilding the whole application.

For another take on this approach, see Red Hat's source-to-image which invokes an assemble script at a well known location in the base container image; this script embeds all the build logic, so that the developer doesn't need to write a Dockerfile, only comply with the conventions of the base image.

Docker and the OCI container ecosystem

Posted Jul 26, 2022 23:17 UTC (Tue) by anguslees (subscriber, #7131) [Link] (2 responses)

Nix, bazel, and yocto/openembeded are some mature systems that can build container images without actually using docker or Dockerfiles - they just produce the layer tar files (OCI image) directly.

I agree there should be more tools like this - it's not that hard. The thing you lose with these approaches is any easy "bootstrap" step to set up the build system in the first place. If you put the build system in a docker container, then you get back to something like a multistage Dockerfile anyway (for simple projects - complex projects can still benefit from these more advanced systems).

Also be aware of cross compiling. In the FOSS community, we would like to support our non-intel brethren and this requires building images for platforms you don't have by default. Some systems (yocto) are good at this, others (bazel) are still maturing. Docker buildx makes this very easy through transparent emulation, again if you're willing to settle for Dockerfiles.

For me, the combination of cross compiling and easy bootstrap means I use docker buildx for all my simpler projects, and just put up with the awkward Dockerfile syntax :-(

Docker and the OCI container ecosystem

Posted Jul 26, 2022 23:39 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> Nix, bazel, and yocto/openembeded are some mature systems that can build container images without actually using docker or Dockerfiles - they just produce the layer tar files (OCI image) directly.

Do they support layering? This is one feature of Docker that makes it bearable, especially for iterative development.

Docker and the OCI container ecosystem

Posted Jul 31, 2022 19:07 UTC (Sun) by robert_s (subscriber, #42402) [Link]

> Do they support layering?

Answering for Nix and its dockerTools: short answer yes (https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-docker...), though half of the benefits of using "layers" in docker-land are already solved by Nix's build-caching system. So the advantages gained by "layers" are mostly around image size.

Docker and the OCI container ecosystem

Posted Jul 27, 2022 18:21 UTC (Wed) by rjones (subscriber, #159862) [Link] (1 responses)

Docker files are gross, but there are ways to mitigate the grossness.

The big one I do is to simply have a shell script that does all the work of setting up the docker image. The dockerfile only copies in the script and required assets and the script does all the work of actually setting it up. Of course you can do the same thing with makefiles or scons or whatever build tool you like using. This reduces things to a single COPY and single RUN command.

Along with that it's useful to do multistage builds.

https://docs.docker.com/develop/develop-images/multistage...

This is especially nice if you want to do statically compiled binaries. You end up with a build container and a runtime container. You only then need to distribute the runtime container. This results in a very clean OCI image with all the build-time and setup portions stripped out. It is usually very easy to keep things down well under a 100MB in size.

As far as getting rid of dockerfile itself... It is probably not worth it at this stage. Everybody and their mom supports it and thus it makes it easy to integrate containers into CI/CD workflows, among other things. Sefl-hosting build farms in kubernetes is extremely easy with kaniko. You can use buildah and other associated tools related to podman. Gitlab, github, and most everything else supports building and testing against OCI images.

Docker and the OCI container ecosystem

Posted Jul 27, 2022 20:07 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

I agree. I don't really like Dockerfiles either, but the `COPY` and `RUN` with shell scripts really helps. It also keeps artifacts of what happened in the image itself (no point in deleting since it's in the previous layer anyways).

Other than that, I've found `buildah` to be useful for building minimal images where I can do some logic outside of the container and tell it afterwards. Much better than manual flattening after the fact IMO.

Docker and the OCI container ecosystem

Posted Jul 28, 2022 5:28 UTC (Thu) by bartoc (guest, #124262) [Link] (1 responses)

I don't find dockerfiles all that horrid as long as everyone on the team writing them actually understands how the layering works. They are reasonably simple and good enough.

I do prefer buildah's approach though.

Actually I would prefer a scheme that used a content addressable datastore instead of the layering thing, something like ostree or casync (or git, really that would be most convenient, but git has some missing features when dealing with large binary files)

Docker and the OCI container ecosystem

Posted Aug 5, 2022 6:01 UTC (Fri) by hsiangkao (subscriber, #123981) [Link]

you could refer to chunk-based content-addressable Nydus image service [1] and erofs filesystem fscache support for container images [2].

[1] https://github.com/dragonflyoss/image-service/
[2] https://d7y.io/blog/2022/06/06/evolution-of-nydus/

Docker and the OCI container ecosystem

Posted Jul 28, 2022 5:49 UTC (Thu) by pabs (subscriber, #43278) [Link] (1 responses)

Where does your disgust stem from? Mine is that they are mixed-language files. I dislike Makefiles and even printf/regex/SQL for this reason too.

Docker and the OCI container ecosystem

Posted Jul 28, 2022 5:54 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Ohhh.... So many reasons.

1. Dumb syntax. You have \
to write \
very long \
lines that blow up logs because they are actually ran as one long line.

2. The way the layering system works. All of it is kinda crappy.

3. COPY command. There's no way to copy multiple source directories in one layer.

And finally, Dockerfiles pretend to be purely functional and reproducible, but most projects have lines like "RUN apt-get blah" that immediately blow that up.

A true content-addressable system with a better build language than Dockerfiles would be great. Earthly seems to be a good incremental improvement on Dockerfiles.

Docker and the OCI container ecosystem

Posted Aug 19, 2022 7:36 UTC (Fri) by daenzer (subscriber, #7050) [Link]

FWIW, the freedesktop.org GitLab CI templates (https://gitlab.freedesktop.org/freedesktop/ci-templates) use buildah, seems pretty nice.

The templates currently offer two modes of operation for generating images: A simple one where one can just specify the base system packages to be installed, and a generic one which allows running a script to set up the filesystem arbitrarily.

The templates produce a single layer, but it can be based on any other image (by default a standard image of one of the supported base distros), which is included as separate layers. This allows efficient sharing of common base contents.

Docker and the OCI container ecosystem

Posted Sep 3, 2022 7:30 UTC (Sat) by jond (subscriber, #37669) [Link] (1 responses)

I use and work on a pre-processor tool cekit - https://cekit.io - which abstracts (somewhat) over dockerfiles. It’s more declarative and lets you break the recipe up into separate modules that can be shared amongst different container sources (like mix-ins, contrast to container image single inheritance )

Docker and the OCI container ecosystem

Posted Sep 3, 2022 7:54 UTC (Sat) by jond (subscriber, #37669) [Link]

Example image sources for cekit: https://GitHub.com/jboss-container-images/openjdk