Building Windows multi-arch container images on Linux

When running Windows containers the container host version must match the image version. For container image publishers this often introduces the desire to provide multi-arch Windows images for all different available Windows versions for their users.

I see many publishers struggle to accomplish this because many hosted CI services often only support 1 or 2 required Windows versions and also can’t run Hyper-V isolated containers.

One solution that many don’t seem to know is to build multi-arch Windows images on Linux instead. This works for a large portion of applications.

TL;DR

You can cross-build multi-arch Windows images on Linux using BuildKit as long as you don’t need to execute Windows commands on the Windows image (no RUN instructions in the Windows stage of the Dockerfile). All other instructions can be used like normally.

Most Dockerfiles for Windows images can be refactored to benefit from cross-building. The idea is to move the plumbing that requires RUN instructions (like cross-compiling, downloading binaries/libs/dependencies, creating directory structures/configs etc.) to a Linux build stage and copy the results over to the Windows image. Examples:

(a) Cross-compile .NET app on Linux and copy to Windows image:

ARG WINBASE
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:6.0-alpine AS build
WORKDIR /src
COPY . .
RUN dotnet publish -c release -o app -r win10-x64 --self-contained true /p:PublishTrimmed=true /p:PublishReadyToRun=true /p:PublishSingleFile=true

FROM ${WINBASE}
ENTRYPOINT [ "app.exe" ]
COPY --from=build /src/app/dotnet.exe app.exe

(b) traefik: Download pre-compiled binary, prepare on Linux and copy to Windows image:

ARG WINBASE
FROM --platform=$BUILDPLATFORM curlimages/curl:7.80.0 AS build
WORKDIR /src
RUN curl -Lo traefik.zip https://github.com/traefik/traefik/releases/download/v2.5.4/traefik_v2.5.4_windows_amd64.zip ; \
    unzip traefik.zip

FROM ${WINBASE}
ENTRYPOINT [ "traefik.exe" ]
COPY --from=build /src/traefik.exe traefik.exe

(c) Pull external dependencies from single-arch images and copy to Windows image:

ARG WINBASE
FROM ${WINBASE}
ENTRYPOINT [ "wins.exe", "-v" ]
COPY --from=sigwindowstools/kube-proxy:v1.22.4-1809 /utils .

More examples like for cross-building multi-arch Windows images for Go and Rust applications can be found on GitHub. At the end of this post you can find tips on how to refactor to enable cross-building for your Windows images.

You can build these Dockerfiles on Linux with Buildkit for all six different Windows versions, create a manifest list and annotate the Windows version for each image in the list with the following script:

docker buildx create --name img-builder --use --driver docker-container --driver-opt image=moby/buildkit:v0.9.3

TARGETIMAGE="lippertmarkus/test:1.0"
BASE="mcr.microsoft.com/windows/nanoserver"
OSVERSIONS=("1809" "1903" "1909" "2004" "20H2" "ltsc2022")
MANIFESTLIST=""

for VERSION in ${OSVERSIONS[*]}
do 
    docker buildx build --platform windows/amd64 --push --pull --build-arg WINBASE=${BASE}:${VERSION} -t "$TARGETIMAGE-${VERSION}" .
    MANIFESTLIST+="${TARGETIMAGE}-${VERSION} "
done

docker manifest rm $TARGETIMAGE > /dev/null 2>&1
docker manifest create $TARGETIMAGE $MANIFESTLIST

for VERSION in ${OSVERSIONS[*]}
do 
  docker manifest rm ${BASE}:${VERSION} > /dev/null 2>&1
  full_version=`docker manifest inspect ${BASE}:${VERSION} | grep "os.version" | head -n 1 | awk '{print $$2}' | sed 's@.*:@@' | sed 's/"//g'`  || true; 
  docker manifest annotate --os-version ${full_version} --os windows --arch amd64 ${TARGETIMAGE} "${TARGETIMAGE}-${VERSION}"
done

docker manifest push $TARGETIMAGE

In a matter of seconds your favorite applications are available as multi-arch Windows images for every existing Windows version:

If you want to know more details on building Windows and Linux multi-arch images and see how to combine them to a single manifest as well as use a single Dockerfile for Linux and Windows images, read on! You’ll also get tips on preventing RUN in your target platform stages to enable cross-building.

Intro to multi-arch images

There are three main ways to build multi-arch images in general:

1. Cross-building images through cross-compiling/downloading the binaries/scripts for each architecture on the build platform in build stages and copy them into the target platform stage. I’ll refer to this as cross-building in the following but like I mentioned, depending on the application this could mean cross-compilation, downloading pre-compiled binaries, setting up scripts/libs/directories/configs etc. in the build stages (see Windows examples in TL;DR).
2. Natively build the images for each architecture using machine emulation like QEMU on Linux or Hyper-V on Windows.
3. Natively build the images for each architecture directly on that target architectures.

The third path isn’t widely used as it requires separate machines for each architecture. This is costly and often not available in CI environments but it’s also rarely needed, as the first and second paths support building images for almost all relevant architectures.

Both the first and second ways can also be used to build Windows and Linux multi-arch images. Whenever possible, you should cross-build images for the target platform on your native build platform. This is the most efficient way to build multi-arch images. If you need to RUN commands within the build stage for the target platform because your scenario/app/language/framework doesn’t support cross-building or you need to setup something that’s not possible without running commands you can use machine emulation to emulate the target platform on your native build platform. Cross-building and natively building images with machine emulation on Windows and Linux will be shown in the next sections.

BuildKit

When building on Linux BuildKit can be used to simplify the process for all paths mentioned. BuildKit is a toolkit for building container images in an efficient way. It’s included in Docker and can be used via the docker buildx subcommand. With buildx you can add multiple local and remote builders. You can use your local Docker engine, a buildkit daemon running inside a container or in Kubernetes as builders. Let’s add our own:

docker buildx create --name img-builder --use --driver docker-container --driver-opt image=moby/buildkit:v0.9.3

In the next sections we now investigate how creating multi-arch images with cross-building and machine emulation works for Windows and Linux with a simple Hello World Go app and the following Dockerfile as an example:

ARG WINBASE=scratch
FROM --platform=$BUILDPLATFORM golang:alpine AS build
WORKDIR /src
COPY . .
RUN go get -d -v ./...
RUN if [ "$TARGETARCH" = "arm" ]; then export GOARM="${TARGETVARIANT//v}"; fi; \
    GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o app -v ./... ; \
    chmod +x app

FROM ${WINBASE} AS windows
ENTRYPOINT [ "app.exe" ]
COPY --from=build /src/app app.exe

FROM scratch AS linux
ENTRYPOINT [ "/app" ]
COPY --from=build /src/app /app

Note that the example used shows how this would work for an application which can be cross-compiled to the target platform. Like I mentioned before, you could instead download/copy pre-compiled binaries for the target platform or set up other configs/scripts/libs/directories in the build stage (see Windows examples in TL;DR). The following informations apply for all cases.

Quick recap: Multi-arch Linux images

Creating multi-arch Linux images with Buildkit using cross-building and/or QEMU is widely used and well described. To highlight the differences to Windows I want to quickly recap the process anyway.

Cross-building images

When inspecting the builder we created before we can see that only a few Linux platforms are supported:

docker buildx inspect --bootstrap
# ...
# Platforms: linux/amd64, linux/386

Note that the output for you can be different depending on your build platform. Also if you use Docker Desktop, QEMU machine emulators are already installed and more platforms are listed here.

The listed platforms only show on which target platforms you can execute commands, e.g. via RUN in your Dockerfile. If you don’t need to RUN commands on your target platform (i.e. in the linux stage of the Dockerfile above) you can build our example Dockerfile for other platforms anyway:

docker buildx build --platform linux/amd64,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/mips64le,linux/mips64,linux/arm/v7,linux/arm/v6 --push --pull --target linux -t lippertmarkus/test:1.0 .

Using QEMU machine emulation

If you now try to run a command on the target platform in the linux stage of the Dockerfile e.g. by adding RUN ["/app"] at the end of the example Dockerfile you’ll get an error:

> [linux 2/2] RUN ["/app"]:
#15 0.181 .buildkit_qemu_emulator: /app: Invalid ELF image for this architecture

If you need to run commands on the target platform e.g. because your scenario/app/language/framework doesn’t support cross-building, you can install QEMU emulators for your target architecture on the build platform:

docker run --privileged --rm tonistiigi/binfmt --install all

When inspecting the builder again, we now see new plattforms supported:

docker buildx inspect --bootstrap
# ...
# Platforms: linux/amd64, linux/386, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Running a command on the target architecture in the linux stage of the example Dockerfile during the build now also works:

#17 [linux 2/2] RUN ["/app"]
#17 0.150 Hello World

That’s how it works for Linux multi-arch images. How does the process differ for Windows multi-arch container images?

Multi-arch Windows images

To create multi-arch Windows images, I think you should - like for Linux - always try to cross-build Windows images on Linux whenever possible.

Another option would be to use Hyper-V isolated containers to build the multi-arch images on Windows natively. As this isn’t as efficient as cross-building you should only use it when absolutely necessary (if you need to RUN commands on the target platform during build). Using Hyper-V isolated container also has a few more drawbacks I will describe later in the according section.

Cross-building images for Windows on Linux

The same restrictions described for cross-building for Linux multi-arch images also apply to building Windows multi-arch images on Linux: If you don’t need to run commands on Windows (i.e. in the windows stage of the Dockerfile above) you can build your Windows multi-arch image on Linux similarly to how you build Linux multi-arch images.

For Windows instead of specifying multiple platforms you need to use different base images for each version of Windows. The base image in our example is passed via a build argument WINBASE and each image is pushed independently:

TARGETIMAGE="lippertmarkus/test:1.0"
BASE="mcr.microsoft.com/windows/nanoserver"
OSVERSIONS=("1809" "1903" "1909" "2004" "20H2" "ltsc2022")
MANIFESTLIST=""

for VERSION in ${OSVERSIONS[*]}
do 
    docker buildx build --platform windows/amd64 --push --pull --build-arg WINBASE=${BASE}:${VERSION} --target windows -t "$TARGETIMAGE-${VERSION}" .
    MANIFESTLIST+="${TARGETIMAGE}-${VERSION} "
done

To combine all images to a single manifest list we can use the docker manifest command. For Windows you also need to manually annotate the os.version so that users automatically get the correct version of the image depending on the Windows host OS version:

docker manifest rm $TARGETIMAGE > /dev/null 2>&1
docker manifest create $TARGETIMAGE $MANIFESTLIST

for VERSION in ${OSVERSIONS[*]}
do 
  docker manifest rm ${BASE}:${VERSION} > /dev/null 2>&1
  full_version=`docker manifest inspect ${BASE}:${VERSION} | grep "os.version" | head -n 1 | awk '{print $$2}' | sed 's@.*:@@' | sed 's/"//g'`  || true; 
  docker manifest annotate --os-version ${full_version} --os windows --arch amd64 ${TARGETIMAGE} "${TARGETIMAGE}-${VERSION}"
done

docker manifest push $TARGETIMAGE

The result on Docker Hub looks like this:

More examples for cross-building are described in the TL;DR section and on GitHub.

Using Hyper-V machine emulation on Windows

First off: You can’t use QEMU on Linux to run Windows binaries like you can do for other Linux architectures.

If you need to run commands in the target platform build stage for building multi-arch Windows images, you need to instead use Hyper-V isolation and get rid of the Linux build stage (may replace it with a Windows stage). Hyper-V isolation works different to the QEMU machine emulation on Linux which only emulates running single binaries. Hyper-V isolation instead runs the whole Windows container image in an optimized Hyper-V virtual machine and therefore emulates a machine with a different Windows version. This is known as Hyper-V isolated containers.

Hyper-V isolation supports running Windows containers with Windows versions lower or equal to the container host and therefore also enables building Windows container images for different Windows versions.

One example when you would use Hyper-V for building images could be running custom installers like for python:

ARG WINBASE
FROM ${WINBASE}
ADD https://www.python.org/ftp/python/3.11.0/python-3.11.0a2-amd64.exe python.exe
RUN ["python.exe", "/quiet"]

It’s important to understand that there are not many cases where Hyper-V isolation is really needed. To enable cross-building for this specific case here you could try extracting files from the installer in a Linux build stage. Another way would be to build this image natively on Windows a single time as a single-arch image and use it in your cross-build to copy C:\python to the target Windows stage similarly to the example (c) in the TL;DR section. I describe some tips on how to avoid using Hyper-V isolation later in this post.

As BuildKit doesn’t work on Windows you would need to use the default docker build command instead with Hyper-V isolation enabled to create the Windows container images:

docker build --pull --isolation hyperv --build-arg WINBASE=${BASE}:${VERSION} -t "${TARGETIMAGE}-${VERSION}" .
docker push "${TARGETIMAGE}-${VERSION}"

You can also run that in a loop and create a manifest list like we did in the last section.

Using Hyper-V isolated containers to create multi-arch Windows images however has a few drawbacks:

• Building multi-arch images natively on Windows with Hyper-V isolated containers is much slower than cross-building because the RUN commands need to be executed for every Windows version.
• Hyper-V isolation is not always available in CI environments, e.g. GitHub Actions and Azure Pipeline Hosted Agents don’t support it.
• Hyper-V isolation requires nested virtualization which is only available for certain virtual machine sizes in public clouds like Azure.
• You must keep the container host up to date because you can only build images for image versions lower or equal to the container host.
• Hyper-V isolation adds overhead as each container or RUN-step during image build is run in a separate VM.

Note that you could also “cross-build for Windows on Windows” with Hyper-V isolation by using a build stage with a fixed Windows image version and copying the binary to images with different target Windows versions like we did on Linux. This would be almost as fast as on Linux as the build stage would be running only once. But if you only need to copy files to a target platform image anyway you should do it on Linux because of the drawbacks of Hyper-V isolation mentioned above.

Combining Windows and Linux multi-arch images

To get a single manifest list with all Windows versions and Linux platforms we can combine our snippets from above and extend them with commands to get images from the Linux manifest list, append and annotate Windows images and overwrite the manifest list in the registry:

TARGETIMAGE="lippertmarkus/test:1.0"
BASE="mcr.microsoft.com/windows/nanoserver"
OSVERSIONS=("1809" "1903" "1909" "2004" "20H2" "ltsc2022")
MANIFESTLIST=""

# build for Linux
docker buildx build --platform linux/amd64,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/mips64le,linux/mips64,linux/arm/v7,linux/arm/v6 --push --pull --target linux -t $TARGETIMAGE .

# build for Windows
for VERSION in ${OSVERSIONS[*]}
do 
    docker buildx build --platform windows/amd64 --push --pull --build-arg WINBASE=${BASE}:${VERSION} --target windows -t "${TARGETIMAGE}-${VERSION}" .
    MANIFESTLIST+="${TARGETIMAGE}-${VERSION} "
done

# Get images from Linux manifest list, append and annotate Windows images and overwrite in registry
docker manifest rm $TARGETIMAGE > /dev/null 2>&1
lin_images=$(docker manifest inspect $TARGETIMAGE | jq -r '.manifests[].digest')

docker manifest create $TARGETIMAGE $MANIFESTLIST ${lin_images//sha256:/${TARGETIMAGE%%:*}@sha256:}

for VERSION in ${OSVERSIONS[*]}
do 
  docker manifest rm ${BASE}:${VERSION} > /dev/null 2>&1
  full_version=`docker manifest inspect ${BASE}:${VERSION} | grep "os.version" | head -n 1 | awk '{print $$2}' | sed 's@.*:@@' | sed 's/"//g'`  || true; 
  docker manifest annotate --os-version ${full_version} --os windows --arch amd64 ${TARGETIMAGE} "${TARGETIMAGE}-${VERSION}"
done

docker manifest push $TARGETIMAGE

Now almost anyone should be able to use our application:

Conclusion

It’s recommendable to cross-build multi-arch images for Linux and Windows on Linux when possible. Using QEMU when building Linux multi-arch images to execute the RUN commands in target platform stages is slow but depending on the effort may acceptable.

For Windows however using Hyper-V isolation to build multi-arch images comes with many drawbacks described above and isn’t always possible depending on your infrastructure or CI system. Therefore if you want to provide multi-arch Windows images you should consider refactoring your Dockerfiles to support cross-building on Linux.

Tips to avoid RUN commands in the target platform stages to enable/optimize cross-building:

• For platforms/frameworks with support for cross-compiling use a build stage running on the build platform to build binaries for your target platform stage and copy them there. This is shown in example (a) in the TL;DR section.
• You don’t always need to do the build within a build stage. If you already have an environment set up to generate binaries for different target platforms you can just COPY them to the target platform stage from the local file system.
• For images based on hosted pre-compiled binaries, use a build stage running on the build platform to download and extract them and copy them to the target platform stage. This is shown in example (b) in the TL;DR section.
• Installation scripts or installers often do nothing else than downloading some files. They often have parameters to set the platform for the files to download so you can run them in the build stage and just copy the needed paths to the target platform stage.
• Linux: There are cross-compilation helpers like tonistiigi/xx that help you set up the right environment for cross-compiling, installing dependencies etc. in your build stage running on the build platform.
• Linux: For minimal Linux images using scratch often doesn’t work because it’s missing CA certificates or non-root users. While you could copy them to a target platform stage yourself, you could also use distroless images
• Windows: If you use MSI installers which only extract files to a certain location, you can try using msitools to extract the files and copy them to the target platform stages.
• Windows: When you have no alternatives and need to run Windows programs, installers or things like Chocolatey you can also natively build a single-arch image where you run those things once. In your cross-builds on Linux you can copy the resulting directories from that image to the Windows target stage. This also makes sure that your dependencies are locked to a known-working version. This is shown in the example (c) in the TL;DR section.

Further reading

• Repository with presented examples and scripts on GitHub
• Peri Thompson’s blog post about the same topic as big inspiration for this blog post
• SIG Windows using cross-building for easily creating flannel and kube-proxy Windows images on Linux
• Issue for BuildKit Windows Support on GitHub
• Google ko supports building Go containers for Windows by applying a tar with the binary to a base image layer
• crane support for adding tars to Windows base image in progress