Kubernetes v1.22+ together with containerd v1.6+ introduced HostProcess containers for Windows nodes as an alpha feature. Like priviledged Linux containers this allows for many management scenarios like driver, networking and storage provisioning where host access is required. Running proxy tools like
csi-proxy on the host is now no longer needed.
I’ll describe how you can set up a cluster with HostProcess container support by yourself before going into the technical aspects. If you don’t want to try it out yourself, feel free to directly jump to the introduction of HostProcess containers.
Setting up a cluster with HostProcess Container support
I created a
Vagrantfile for deploying a two-node cluster with a Linux controlplane and a Windows Server 2022 worker node with Windows HostProcess containers enabled. Calico is used for networking. The Container Network Interface (CNI) configuration, Calico itself and
kube-proxy are deployed via HostProcess pods to the Windows nodes, which makes the setup very simple.
Have a look at the repository for a detailed description on how to set up the cluster with Vagrant. To summarize, you need to:
- Make sure Vagrant and Hyper-V are installed and you have created a virtual external switch connected to your main network with DHCP.
- Verify that the
setup-scripts/kubeadm-config.ymldo not overlap with your main network or set those as well as
vagrant upto create the cluster and select your virtual external switch and enter your admin user credentials while provisioning.
- After provisioning execute
$env:KUBECONFIG=(Get-Item .\share\kubeconfig).FullNameto access your newly created cluster and wait for everything to settle.
Alternatively you could also use the Kubernetes Cluster API with the recently added Cluster Template to create a cluster with Windows nodes, containerd and HostProcess Container support on Azure. If you go that route, make sure to manually deploy the addons for Calico and kube-proxy on Windows after provisioning.
Intro to Windows HostProcess Containers
Windows HostProcess Containers run directly on the host and have neither filesystem nor networking or process isolation. They are sharing the host’s filesystem, process and networking space. Under the hood HostProcess containers are using Windows Job Objects instead of the server silos used by normal Windows containers. See this diagram for an overview of the overall architecture differences. Because of this architecture there are further special considerations when it comes to running HostProcess containers:
- The version of the base image does not need to match the host OS version because HostProcess containers directly use the host’s kernel.
- Currently there’s no TTY support when exec into HostProcess containers
- HostProcess pods currently can only consist of HostProcess containers. Standard Windows Server containers are not supported in the same pod.
- Resource limits for disk, memory or CPU count work the same as for normal containers.
- Mounting named pipes is not supported. They can instead be accessed directly via their path on the host.
The Pod spec for a HostProcess containers needs a few configurations to be set. A minimal example would look like:
apiVersion: v1 kind: Pod metadata: name: test spec: containers: - name: test image: mcr.microsoft.com/windows/servercore:ltsc2022 command: [powershell, -command] args: [Get-ChildItem, C:\] securityContext: windowsOptions: hostProcess: true runAsUserName: "NT AUTHORITY\\SYSTEM" hostNetwork: true nodeSelector: "kubernetes.io/os": windows
Note that you can run the Pod as
Local Service or
Network Service user depending on which degree of privileges you need.
Now let’s have a look at a few things which are different to normal pods.
Experimenting with Windows HostProcess Containers
First of all you can spot that the pod is using the host’s filesystem at it’s root:
# args: [Get-ChildItem, C:\] ... Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 11/4/2021 7:31 PM C d----- 11/3/2021 6:32 PM CalicoWindows d----- 10/5/2021 2:17 PM chef d----- 11/3/2021 6:28 PM etc d----- 11/3/2021 6:28 PM k ...
However, each HostProcess pod still gets its own volume with the image content as a clean workspace. The path to this volume gets set in
$CONTAINER_SANDBOX_MOUNT_POINT. There you’ll find the directory structure you’re used to:
# args: [Get-ChildItem, $env:CONTAINER_SANDBOX_MOUNT_POINT] Directory: C:\C\d6a326de490853457f55c05f10a9c4f7ccbb88c6356c7480c896589e1afe78bc Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 11/4/2021 7:36 PM dev d-r--- 10/7/2021 6:35 PM Program Files d----- 10/7/2021 11:23 AM Program Files (x86) d-r--- 10/7/2021 11:40 AM Users d----- 11/4/2021 7:36 PM var d----- 10/7/2021 11:40 AM Windows -a---- 5/8/2021 10:26 AM 5510 License.txt
Because of that, there are no special considerations when creating images for HostProcess containers. Volumes you may specified in the Pod spec are also mounted relative to the
$CONTAINER_SANDBOX_MOUNT_POINT. The service account tokens can be found within
Due to the shared networking space the Pod obviously also uses the IP address of the host and every service running inside your HostProcess pod would be reachable via this IP:
# args: [ipconfig] Ethernet adapter vEthernet (Ethernet): ... IPv4 Address. . . . . . . . . . . : 10.1.0.137
The shared host networking space also means that cluster DNS is not working for HostProcess containers:
# args: [curl.exe, http://my-nginx] ... Could not resolve host: my-nginx
Lastly you can also tell that HostProcess containers share the process space with the host by looking at the process list:
# args: [Get-Process] Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName ------- ------ ----- ----- ------ -- -- ----------- 278 17 30008 41572 5.13 9128 0 calico-node ...
That’s basically it. If you want to have a look at Calico, Flannel or
kube-proxy as real-world examples for HostProcess pods, you’ll find them in the HostProcess examples of SIG Windows.
Conclusion and Future
Running Windows HostProcess pods is not much different from running normal Windows pods. The behaviour of HostProcess Containers is mostly like you would expect when sharing the host’s filesystem, process and networking space.
Nevertheless Windows HostProcess pods will greatly simplify management and administration of your mixed Kubernetes Clusters. Please keep in mind that HostProcess Containers are still in alpha and may change in the future. With Kubernetes v1.23 HostProcess Containers will move to beta. Things planned for beta or GA include:
- Support for running as any user that’s available on the host
- Filesystem layout may change again to present the filesystem similar to normal Windows pods to easier access files in the container filesystem from Scripts or client libraries
- Support for mounting Unix domain sockets
- Running Container Storage Interface (CSI)-proxy as a daemon set
- Support for TTY when exec into a container