Kubernetes has established itself as the container orchestration platform of choice across a number of organisations, and is well supported on a number of different cloud providers. Everywhere from AWS and Google Cloud Platform to Linode and Digital Ocean has their own Kubernetes offering.

Kubernetes began as a Linux-only project. However a number of improvements within the Windows operating system (including native Container support) mean that it is now possible to run Kubernetes workloads on either Linux or Windows. Windows is currently not supported as the control plane (i.e. what controls the container orchestration), but can run as a Kubernetes worker.

In a short series of posts, I am going to cover how to build a Kubernetes cluster that includes both Linux and Windows workers. This series will cover: -

  • Initial image creation (this post)
  • Deploying the images and creating a cluster with Terraform, KVM and Cloud-Init
  • Deploying applications to the cluster and making them externally available

Environment

Rather than running this on a specific cloud provider (and getting into specifics about each provider’s managed Kubernetes implementation), this will be deployed on the KVM virtualisation environment on Linux. This means that while we may need to do a little more work up front, many of the lessons learned will be applicable to all Kubernetes implementations.

We will use Packer and Ansible to generate the base images for both Linux and Windows. Packer will create the images, whereas Ansible will manage the configuration and dependencies. This will decrease the time it takes to spin up a new node into the cluster, as all the necessary features and applications are installed. They will only require the cluster-specific details to be able to join.

We will also make use of Terraform (with the Terraform Libvirt provider and Cloud-Init (native Cloud-Init for Linux, Cloudbase-Init for Windows) to deploy the control plane nodes and the workers.

To provide a highly-available control plane we will use OpenBSD’s relayd(8) to provide basic loadbalancing for the Kubernetes API. The reason for using OpenBSD is that I always have an OpenBSD virtual machine running in my lab to provide DNS, DHCP and basic firewalling, so adding basic loadbalancing functionality here makes sense. In your own scenarios you can use any loadbalancer of your choice (NGINX, HAProxy, Cloud load balancers etc).

Why Packer?

Packer can build images across cloud providers, virtualisation environments, Docker, Vagrant and more. This means that the Packer configuration we will use will be easily transferable to your provider of choice. If your preferred platform is VMWare, then you can adapt the configuration to see fit. If you choose AWS or Azure, then you can remove most of the configuration and still build a similar cluster.

You can look at the list of Packer builders here.

Why Ansible?

Ansible was built from day one as an agentless configuration management system. Given that the Packer images are going to cloned, we do not want to register the images with a configuration management system only to then require unregistering afterwards.

Also, Ansible has very good support for managing Windows as well as Linux, which works for our scenario.

Other configuration management tools (Chef, Puppet and Saltstack) do support agentless operation now, so if you are more comfortable with them then you can use them instead with Packer.

Why Cloud-Init and Cloudbase-Init?

Rather than creating each individual node’s image separately in Packer, we can create two images (a Linux image and a Windows image) that can be customized at first boot. The images will be prepared with all the common dependencies (i.e. base package installation, container support, Kubernetes binaries), and then any cluster-specific details can be supplied on the next boot.

This means that the images are reusable across many clusters, rather than being specific to one cluster.

Packer

I have covered Packer previously as well as how to use Ansible to customize the images that Packer builds.

With this being the case, we will cover the specifics for how to use it with KVM, as well as some of the quirks with building Windows images for KVM using Packer.

QEMU Builder

To build images for KVM, we use the Packer QEMU Builder. This will generate a qcow2-based image that KVM will use as the root disk of the Kubernetes virtual machines.

Unlike the builders for AWS, Google Cloud, Azure, DigitalOcean or otherwise, we do not use a pre-installed image that you can then customize. Instead we treat the installation as if it was a headless install of the machine from an ISO. This requires the use of preseed files (Debian Preseed, CentOS/RHEL Kickstart or Ubuntu Autoinstall) to provide answers to the questions in the ISO installers.

On Windows, this requires using Autounattend files. Also because Windows does not have virtio drivers by default, we need to provide drivers at install time so that it can access the storage controller.

Linux Build

We build the Linux images using Debian Buster. Most of the Packer configuration file was sourced from this repository by Goffinet, with some changes based upon my environment.

{
  "builders": [
    {
      "name": "debian10",
      "type": "qemu",
      "format": "qcow2",
      "accelerator": "kvm",
      "qemu_binary": "/usr/bin/qemu-system-x86_64",
      "net_device": "virtio-net",
      "disk_interface": "virtio",
      "disk_cache": "none",
      "cpus:": "2",
      "memory": "2048",
      "ssh_wait_timeout": "30m",
      "http_directory": "http",
      "ssh_username": "root",
      "ssh_password": "testtest",
      "iso_urls": "https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.6.0-amd64-netinst.iso",
      "iso_checksum": "2af8f43d4a7ab852151a7f630ba596572213e17d3579400b5648eba4cc974ed0",
      "boot_wait": "15s",
      "boot_command": [
                "<esc><wait>",
                "auto <wait>",
                "console-keymaps-at/keymap=us <wait>",
                "console-setup/ask_detect=false <wait>",
                "debconf/frontend=noninteractive <wait>",
                "debian-installer=en_US <wait>",
                "fb=false <wait>",
                "install <wait>",
                "kbd-chooser/method=us <wait>",
                "keyboard-configuration/xkb-keymap=us <wait>",
                "locale=en_US <wait>",
                "netcfg/get_hostname=debian10 <wait>",
                "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/deb10-kube/debian-preseed.cfg <wait>",
                "<enter><wait>"
      ],
      "disk_size": "40000",
      "disk_discard": "unmap",
      "disk_compression": true,
      "headless": "true",
      "vnc_bind_address": "0.0.0.0",
      "shutdown_command": "echo 'testtest' | sudo -S shutdown -P now",
      "output_directory": "artifacts/qemu/debian10-kube"
    }
  ],
  "provisioners": [
    {
         "type": "ansible",
         "extra_arguments": [
                "--extra-vars",
                "ansible_python_interpreter=/usr/bin/python"
              ],
         "playbook_file": "ansible/deb10-kubernetes/main.yml",
         "user": "root"
    }
  ]
}

Some points to note in this are: -

  • We set the type to qemu (which informs Packer which builder to use)
  • We set the format to qcow2 (the native disk format for KVM)
  • We set the accelerator to KVM so that the machine can make native use of the (much quicker) KVM acceleration than the native QEMU emulation for building our image
  • The qemu_binary location may differ depending upon your distribution (check the location with which qemu-system-x86_64)
  • We use virtio based network and disks as they are much quicker than any other emulated interface
  • We supply the ISO and checksum so that Packer can source the ISO from Debian directly, rather than relying on a local copy
  • We supply the vnc_bind_address as 0.0.0.0 so that we can monitor the build even if running on a headless server

The major part that automates the installation is the boot_command section. The list of commands provide keyboard input to the installer (as if someone was typing these commands) so that it knows to use the preseed file, as well as setting keymaps, the hostname and more.

When supplying the http_directory field, Packer will run a HTTP server that exposes the directories (and contents) provided. This means that you can serve a preseed.cfg file over HTTP (in our case, this is called debian-preseed.cfg, n the http directory) rather than having to supply it on a disk image.

The contents of the preseed file are: -

d-i pkgsel/install-language-support boolean false

# locale
d-i debian-installer/locale string en_GB.UTF-8

# keyboard
d-i keyboard-configuration/xkb-keymap   select  us

# timezone
d-i time/zone string Europe/London

# Controls whether to use NTP to set the clock during the install
d-i clock-setup/ntp boolean true

# apt
d-i mirror/country string manual
d-i mirror/http/hostname string deb.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string
d-i apt-setup/use_mirror boolean true

# users
d-i passwd/root-password password testtest
d-i passwd/root-password-again password testtest
d-i passwd/make-user boolean false

# partitioning
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

apt-cdrom-setup apt-setup/cdrom/set-first boolean false

# software
tasksel tasksel/first multiselect standard
d-i pkgsel/include string qemu-guest-agent wget openssh-server vim sudo cloud-init
popularity-contest popularity-contest/participate boolean false

# grub
d-i grub-installer/only_debian boolean true
d-i grub-installer/bootdev  string /dev/vda

# script
d-i preseed/late_command in-target sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /target/etc/ssh/sshd_config

# finish
d-i finish-install/reboot_in_progress note

To build this with Packer, you use packer build $NAME_OF_FILE.json. This will start the build process, which you can then view over VNC: -

$ packer build debian-10-kube.json
debian10: output will be in this color.

==> debian10: Retrieving ISO
==> debian10: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.6.0-amd64-netinst.iso
==> debian10: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.6.0-amd64-netinst.iso?checksum=sha256%3A2af8f43d4a7ab852151a7f630ba596572213e17d3579400b5648eba4cc974ed0
==> debian10: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.6.0-amd64-netinst.iso?checksum=sha256%3A2af8f43d4a7ab852151a7f630ba596572213e17d3579400b5648eba4cc974ed0 => /home/stuh84/git/packer/packer_cache/c9149179a5e3b2d2360476806e9a5d30854c84ea.iso
==> debian10: Creating required virtual machine disks
==> debian10: Starting HTTP server on port 8435
==> debian10: Found port for communicator (SSH, WinRM, etc): 3270.
==> debian10: Looking for available port between 5900 and 6000 on 0.0.0.0
==> debian10: Starting VM, booting from CD-ROM
    debian10: The VM will be run headless, without a GUI. If you want to
    debian10: view the screen of the VM, connect via VNC without a password to
    debian10: vnc://0.0.0.0:5924

Connect over VNC to this port, and we can see the image being built: -

VNC to Packer image - Debian

Once the install is finished, Ansible will apply the steps to customize the image and install the relevant dependencies.

Windows Build

Windows Container support for Kubernetes was first made available in Windows Server 2019, so we target that as our base operating system.

Most of the configuration was sourced from Stefan Scherer’s packer-windows repository, with a few changes made to suit my environment.

{
  "builders": [
    {
      "type": "qemu",
      "accelerator": "kvm",
      "communicator": "winrm",
      "format":"qcow2",
      "cpus": "2",
      "disk_size": "51200",
      "disk_interface":"virtio",
      "floppy_files": [
        "http/windows-2019/Autounattend.xml",
        "scripts/win-common/SetUpWinRM.ps1"
      ],
      "qemuargs": [
        [
          "-drive",
          "file=artifacts/qemu/win2019-kube/{{ .Name }},if=virtio,cache=writeback,discard=ignore,format=qcow2,index=1"
        ],
        [
          "-drive",
          "file=isos/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso,media=cdrom,index=2"
        ],
        [
          "-drive",
          "file=isos/virtio-win.iso,media=cdrom,index=3"
        ]
      ],
      "headless": true,
      "vnc_bind_address": "0.0.0.0",
      "iso_checksum": "549bca46c055157291be6c22a3aaaed8330e78ef4382c99ee82c896426a1cee1",
      "iso_url": "https://software-download.microsoft.com/download/pr/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso",
      "iso_target_path": "isos/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso",
      "memory": "4096",
      "output_directory": "artifacts/qemu/win2019-kube",
      "shutdown_command": "shutdown /s /t 30 /f",
      "shutdown_timeout": "15m",
      "vm_name": "windows-server-2019-standard-x64-eval",
      "winrm_password": "adm!nadm!n",
      "winrm_username": "Administrator",
      "winrm_timeout": "2h",
      "winrm_insecure": true,
      "winrm_use_ssl": true
    }
  ],
  "provisioners": [
    {
     "type": "ansible",
     "extra_arguments": [
       "-e",
       "kubernetes_version=v1.19.1",
       "-e",
       "ansible_shell_type=powershell",
       "-e",
       "ansible_shell_executable=None",
       "-e",
       "ansible_winrm_server_cert_validation=ignore"
     ],
     "use_proxy": false,
     "playbook_file": "ansible/windows-2019-kubernetes/main.yml",
     "user": "Administrator"
    },
    {
      "type": "powershell",
      "inline": [
        "C:\\Windows\\System32\\Sysprep\\sysprep.exe /quiet /generalize /oobe  /unattend:C:\\Program Files\\Cloudbase Solutions\\Cloudbase-Init\\conf\\Unattend.xml"
      ]
    }
  ]
}

A lot of this is very similar to the Debian image, with some exceptions. Some important points to note in this are: -

  • We use the floppy_files directive for supplying the Autounattend file
    • We cannot guarantee that the network drivers work correctly pre-install (due to using VirtIO drivers)
  • We supply a script to setup WinRM (taken from here)
    • Ansible uses WinRM for Windows, so this is required for us to make changes to the machine post-install
  • We present the virtio-win.iso ISO so that the machine can load drivers at install time
  • We customize the winrm parameters (including timeouts and whether we use SSL, and also ignoring if a cert is unsigned)

One interesting point for the Ansible stage is that we use use_proxy: false. This means that Ansible bypasses the Packer connection to the machine, and talks to it directly. This is based upon a comment I received on my Building Windows AWS AMIs using Packer and Ansible from James Sturtevant. Previously I used an Ansible connection plugin to effectively piggyback onto the Packer connection to the virtual machine. Avoiding the Packer proxy means that we do not need any custom plugins to interact with the machine.

Finally, we add a step at the end to sysprep the machine (i.e. generalize it ready for being cloned) using the Cloudbase-Init provided Unattend.xml. This ensures that Cloudbase-Init will run on the next boot of the machine (i.e. when it is cloned) so that we can customize the image with cluster-specific parameters.

The contents of the Autounattend.xml file that is used to install Windows in the first place looks like the below: -

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="windowsPE">
        <component name="Microsoft-Windows-PnpCustomizationsWinPE"
            publicKeyToken="31bf3856ad364e35" language="neutral"
            versionScope="nonSxS" processorArchitecture="amd64"
            xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State">

            <!--
                 This makes the VirtIO drivers available to Windows, assuming that
                 the VirtIO driver disk at https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso
                 (see https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/index.html#virtio-win-direct-downloads)
                 is available as drive E:
            -->
            <DriverPaths>
                <PathAndCredentials wcm:action="add" wcm:keyValue="2">
                    <Path>E:\viostor\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="3">
                    <Path>E:\NetKVM\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="4">
                    <Path>E:\Balloon\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="5">
                    <Path>E:\pvpanic\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="6">
                    <Path>E:\qemupciserial\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="7">
                    <Path>E:\qxldod\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="8">
                    <Path>E:\vioinput\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="9">
                    <Path>E:\viorng\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="10">
                    <Path>E:\vioscsi\2k19\amd64</Path>
                </PathAndCredentials>

                <PathAndCredentials wcm:action="add" wcm:keyValue="11">
                    <Path>E:\vioserial\2k19\amd64</Path>
                </PathAndCredentials>
            </DriverPaths>
        </component>

        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-International-Core-WinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <SetupUILanguage>
                <UILanguage>en-US</UILanguage>
            </SetupUILanguage>
            <InputLocale>en-US</InputLocale>
            <SystemLocale>en-US</SystemLocale>
            <UILanguage>en-US</UILanguage>
            <UILanguageFallback>en-US</UILanguageFallback>
            <UserLocale>en-US</UserLocale>
        </component>
        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <DiskConfiguration>
                <Disk wcm:action="add">
                    <CreatePartitions>
                        <CreatePartition wcm:action="add">
                            <Type>Primary</Type>
                            <Order>1</Order>
                            <Size>350</Size>
                        </CreatePartition>
                        <CreatePartition wcm:action="add">
                            <Order>2</Order>
                            <Type>Primary</Type>
                            <Extend>true</Extend>
                        </CreatePartition>
                    </CreatePartitions>
                    <ModifyPartitions>
                        <ModifyPartition wcm:action="add">
                            <Active>true</Active>
                            <Format>NTFS</Format>
                            <Label>boot</Label>
                            <Order>1</Order>
                            <PartitionID>1</PartitionID>
                        </ModifyPartition>
                        <ModifyPartition wcm:action="add">
                            <Format>NTFS</Format>
                            <Label>Windows 2019</Label>
                            <Letter>C</Letter>
                            <Order>2</Order>
                            <PartitionID>2</PartitionID>
                        </ModifyPartition>
                    </ModifyPartitions>
                    <DiskID>0</DiskID>
                    <WillWipeDisk>true</WillWipeDisk>
                </Disk>
            </DiskConfiguration>
            <ImageInstall>
                <OSImage>
                    <InstallFrom>
                        <MetaData wcm:action="add">
                            <Key>/IMAGE/NAME</Key>
                            <Value>Windows Server 2019 SERVERDATACENTERCORE</Value>
                        </MetaData>
                    </InstallFrom>
                    <InstallTo>
                        <DiskID>0</DiskID>
                        <PartitionID>2</PartitionID>
                    </InstallTo>
                </OSImage>
            </ImageInstall>
            <UserData>
                <ProductKey>
                    <!--
                        Windows Server Insider product key
                        See https://blogs.windows.com/windowsexperience/2017/07/13/announcing-windows-server-insider-preview-build-16237/
                    -->
                    <!--<Key></Key>-->
                    <WillShowUI>OnError</WillShowUI>
                </ProductKey>
                <AcceptEula>true</AcceptEula>
                <FullName>Vagrant</FullName>
                <Organization>Vagrant</Organization>
            </UserData>
        </component>
    </settings>
    <settings pass="specialize">
        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <OEMInformation>
                <HelpCustomized>false</HelpCustomized>
            </OEMInformation>
            <ComputerName>vagrant-2019</ComputerName>
            <TimeZone>Pacific Standard Time</TimeZone>
            <RegisteredOwner/>
        </component>
        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-ServerManager-SvrMgrNc" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <DoNotOpenServerManagerAtLogon>true</DoNotOpenServerManagerAtLogon>
        </component>
        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-IE-ESC" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <IEHardenAdmin>false</IEHardenAdmin>
            <IEHardenUser>false</IEHardenUser>
        </component>
        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-OutOfBoxExperience" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <DoNotOpenInitialConfigurationTasksAtLogon>true</DoNotOpenInitialConfigurationTasksAtLogon>
        </component>
        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Security-SPP-UX" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <SkipAutoActivation>true</SkipAutoActivation>
        </component>
    </settings>
    <settings pass="oobeSystem">
        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <AutoLogon>
                <Password>
                    <Value>adm!nadm!n</Value>
                    <PlainText>true</PlainText>
                </Password>
                <Enabled>true</Enabled>
                <Username>Administrator</Username>
            </AutoLogon>
            <FirstLogonCommands>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>cmd.exe /c powershell -Command "Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force"</CommandLine>
                    <Description>Set Execution Policy 64 Bit</Description>
                    <Order>1</Order>
                    <RequiresUserInput>true</RequiresUserInput>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>C:\Windows\SysWOW64\cmd.exe /c powershell -Command "Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force"</CommandLine>
                    <Description>Set Execution Policy 32 Bit</Description>
                    <Order>2</Order>
                    <RequiresUserInput>true</RequiresUserInput>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -File a:\SetUpWinRM.ps1</CommandLine>
                    <Description>Setup WinRM</Description>
                    <Order>3</Order>
                    <RequiresUserInput>true</RequiresUserInput>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Advanced\ /v HideFileExt /t REG_DWORD /d 0 /f</CommandLine>
                    <Order>4</Order>
                    <Description>Show file extensions in Explorer</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKCU\Console /v QuickEdit /t REG_DWORD /d 1 /f</CommandLine>
                    <Order>5</Order>
                    <Description>Enable QuickEdit mode</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Advanced\ /v Start_ShowRun /t REG_DWORD /d 1 /f</CommandLine>
                    <Order>6</Order>
                    <Description>Show Run command in Start Menu</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Advanced\ /v StartMenuAdminTools /t REG_DWORD /d 1 /f</CommandLine>
                    <Order>7</Order>
                    <Description>Show Administrative Tools in Start Menu</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKLM\SYSTEM\CurrentControlSet\Control\Power\ /v HibernateFileSizePercent /t REG_DWORD /d 0 /f</CommandLine>
                    <Order>8</Order>
                    <Description>Zero Hibernation File</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKLM\SYSTEM\CurrentControlSet\Control\Power\ /v HibernateEnabled /t REG_DWORD /d 0 /f</CommandLine>
                    <Order>9</Order>
                    <Description>Disable Hibernation Mode</Description>
                </SynchronousCommand>
                <!-- END WITH WINDOWS UPDATES -->
            </FirstLogonCommands>
            <OOBE>
                <HideEULAPage>true</HideEULAPage>
                <HideLocalAccountScreen>true</HideLocalAccountScreen>
                <HideOEMRegistrationScreen>true</HideOEMRegistrationScreen>
                <HideOnlineAccountScreens>true</HideOnlineAccountScreens>
                <HideWirelessSetupInOOBE>true</HideWirelessSetupInOOBE>
                <NetworkLocation>Home</NetworkLocation>
                <ProtectYourPC>1</ProtectYourPC>
            </OOBE>
            <UserAccounts>
                <AdministratorPassword>
                    <Value>adm!nadm!n</Value>
                    <PlainText>true</PlainText>
                </AdministratorPassword>
            </UserAccounts>
            <RegisteredOwner/>
        </component>
    </settings>
    <settings pass="offlineServicing">
        <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-LUA-Settings" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <EnableLUA>false</EnableLUA>
        </component>
    </settings>
    <cpi:offlineImage xmlns:cpi="urn:schemas-microsoft-com:cpi" cpi:source="wim:c:/wim/install.wim#Windows Server 2019 SERVERSTANDARD"/>
</unattend>

This was again taken from Stefan Scherer’s repository (here) but with a few changes (removing some of the scripts used, and making sure we run the SetUpWinRM.ps1 script).

We can now build this with Packer, and then watch the install take place over VNC: -

$ packer build windows-2019-kube.json
qemu: output will be in this color.

==> qemu: Retrieving ISO
==> qemu: Trying https://software-download.microsoft.com/download/pr/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso
==> qemu: Trying https://software-download.microsoft.com/download/pr/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso?checksum=sha256%3A549bca46c055157291be6c22a3aaaed8330e78ef4382c99ee82c896426a1cee1
==> qemu: https://software-download.microsoft.com/download/pr/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso?checksum=sha256%3A549bca46c055157291be6c22a3aaaed8330e78ef4382c99ee82c896426a1cee1 => isos/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso
==> qemu: Creating floppy disk...
    qemu: Copying files flatly from floppy_files
    qemu: Copying file: http/windows-2019/Autounattend.xml
    qemu: Copying file: scripts/win-common/SetUpWinRM.ps1
    qemu: Done copying files from floppy_files
    qemu: Collecting paths from floppy_dirs
    qemu: Resulting paths from floppy_dirs : []
    qemu: Done copying paths from floppy_dirs
==> qemu: Creating required virtual machine disks
==> qemu: Found port for communicator (SSH, WinRM, etc): 3887.
==> qemu: Looking for available port between 5900 and 6000 on 0.0.0.0
==> qemu: Starting VM, booting from CD-ROM
    qemu: The VM will be run headless, without a GUI. If you want to
    qemu: view the screen of the VM, connect via VNC without a password to
    qemu: vnc://0.0.0.0:5987
==> qemu: Overriding default Qemu arguments with QemuArgs...
==> qemu: Waiting 10s for boot...
==> qemu: Connecting to VM via VNC (0.0.0.0:5987)
==> qemu: Typing the boot command over VNC...
    qemu: Not using a NetBridge -- skipping StepWaitGuestAddress
==> qemu: Using winrm communicator to connect: 127.0.0.1
==> qemu: Waiting for WinRM to become available...

VNC to Packer image - Windows

Once the install is finished, Ansible will apply the steps to customize the image.

Ansible

As noted, we are using Ansible to customize the images after the operating system has been installed. This installs the correct dependencies (e.g. Docker, Kubernetes, Cloudbase-Init for Windows), ensures we have the correct paths and creates the relevant configuration files.

We make use of roles so that they can be reused for other purposes. Each role is created using ansible-galaxy init $ROLE_NAME (although we are not submitting these to Ansible Galaxy), which will create the following directory structure: -

$ROLE_NAME
├── defaults
├── files
├── handlers
├── meta
├── README.md
├── tasks
├── templates
├── tests
└── vars

This allows us to create our tasks in the correct place, create handlers, files and tests should we wish in the correct structure.

Linux Ansible Roles

The Linux deployment uses the following roles: -

  • base - This covers base packages that will be used either during install, or during day-to-day management of the cluster
  • docker - This installs the Docker engine which Kubernetes will use for deploying containers
  • kubernetes - This installs all the relevant Kubernetes dependencies

Base

For the base role, we have the following tasks in our tasks/main.yml file: -

---
# tasks file for docker
- name: Install base dependencies
  package:
    name:
      - apt-transport-https 
      - ca-certificates
      - curl 
      - gnupg2 
      - software-properties-common
      - tcpdump
      - wget
      - telnet
      - vim
      - dnsutils
      - mtr-tiny
      - python3-pip
    state: present
    update_cache: yes

The first 5 dependencies are used when adding other repositories (in the docker and kubernetes roles), whereas the rest are useful if any troubleshooting of the cluster is required.

Docker

The Docker role has the following tasks in our tasks/main.yml file: -

---
# tasks file for docker
- name: Add Docker Repository key
  apt_key:
    url: https://download.docker.com/linux/debian/gpg
    state: present

- name: Add Docker Repository
  apt_repository:
    repo: deb [arch=amd64] https://download.docker.com/linux/debian buster stable
    state: present

- name: Install Docker
  package:
    name:
      - docker-ce
      - docker-ce-cli 
      - containerd.io
    state: present
    update_cache: yes

This adds the Docker APT repository GPG key, adds the Debian Buster Docker repository, and then installs the latest version of Docker Community Edition, the Docker CLI and containerd.

Kubernetes

The Kubernetes role has the following tasks in our tasks/main.yml file: -

---
# tasks file for kubernetes
- name: Add Kubernetes Repository key
  apt_key:
    url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
    state: present

- name: Add Kubernetes
  apt_repository:
    repo: deb https://apt.kubernetes.io/ kubernetes-xenial main 
    state: present

- name: Install Kubernetes
  package:
    name:
      - kubelet
      - kubeadm
      - kubectl 
    state: present
    update_cache: yes

- name: Disable Swap
  shell: |
    swapoff -a

- name: Remove swap in fstab
  replace:
    path: /etc/fstab
    regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
    replace: '# \1'

- name: Bridge NF Filter sysctl IPv4
  ansible.posix.sysctl:
    name: net.bridge.bridge-nf-call-iptables
    value: '1'
    state: present

- name: Bridge NF Filter sysctl IPv6
  ansible.posix.sysctl:
    name: net.bridge.bridge-nf-call-ip6tables
    value: '1'
    state: present

Similar to the Docker role, we add the Kubernetes APT repository GPG key, add the Kubernetes repository (the kubernetes-xenial repository works for all recent versions of Debian and Ubuntu), and installs kubelet (the Kubernetes agent), kubeadm (the Admin tool to create Kubernetes clusters with) and kubectl (the Kubernetes CLI tool).

After this, we disable swap and comment out any swap entries in /etc/fstab as Kubernetes does not allow running with swap partitions enabled.

Finally, we use enable packets that traverse bridges in Linux to be evaluated by IPTables for both IPv4 and IPv6. A lot of the Docker and Kubernetes networking relies on IPTables manipulation, so the packets traversing bridges must go through IPTables as well.

Windows Ansible Roles

The Windows deployment uses the following roles: -

  • base - This covers setting some default values in the Windows registry for TLS
  • containers - This installs the Windows Containers feature and the Docker engine which Kubernetes will use for deploying containers
  • kubernetes - This installs all the relevant Kubernetes dependencies
  • cloudbase-init - This installs Cloudbase-Init so that we can leverage cloud-config and cloud-init to customize the machine on first boot

Base

For the base role, we have the following tasks in our tasks/main.yml file: -

---
# tasks file for base
  # Enable Strong security for Powershell (required for NuGet Provider)
  - name: Enable TLS 1.2 for 64-bit .Net Framework
    win_regedit:
      path: HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319
      name: SchUseStrongCrypto
      data: 1
      type: dword

  - name: Enable TLS 1.2 for 32-bit .Net Framework
    win_regedit:
      path: HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319
      name: SchUseStrongCrypto
      data: 1
      type: dword

This task enables TLS 1.2 support for DotNet in Windows. Without this, NuGet and Invoke-WebRequest will often fail. Powershell by default supports SSLv3 or lower, which many sites now have disabled.

Containers

For the containers role, we have the following tasks in our tasks/main.yml file: -

---
# tasks file for containers
  # Enable Containers feature
  - name: features | Enabling Containers Feature
    win_feature:
      name: Containers
      state: present
    register: windows_container_feature

  - name: reboot | Rebooting Server
    win_reboot:
      reboot_timeout_sec: 3600
    when: windows_container_feature.changed

  # Enable Docker for Windows
  - name: Add Nuget Package provider
    win_shell: Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force

  - name: Install Docker Powershell Module Repository for Windows
    win_psmodule:
      name: DockerMsftProvider
      state: present

  - name: Install Docker for Windows
    win_shell: Install-Package -Name docker -ProviderName DockerMsftProvider -Force
    register: docker_installed

  - name: Enable Docker service
    win_service:
      name: docker
      start_mode: auto
      state: started
    when: docker_installed.changed

  - name: Reboot Windows
    win_reboot:
      reboot_timeout_sec: 3600
    when: docker_installed.changed

The above does the following: -

  • Enables the Containers feature
  • Reboots the server if the Containers feature was installed (otherwise there is no need for a reboot)
  • Adds the ability to use NuGet packages
  • Adds the Microsoft Docker provider
  • Installs Docker for Windows
  • Enables Docker for Windows
  • Reboots the machine

After this, the machine will now have Docker installed, using native Windows containers. This is not the same as Docker Desktop, which runs a lightweight Linux virtual machine for creating and interacting with Linux-based containers.

Kubernetes

For the Kubernetes role, we have the following tasks in our tasks/main.yml file: -

---
# tasks file for kubernetes
- name: Create Kubernetes directory
  win_file:
    path: C:\k
    state: directory

- name: Add Kubernetes directory to path
  ansible.windows.win_path:
    elements:
      - 'C:\k'

- name: Download kubelet
  win_get_url:
    url: https://dl.k8s.io/{{ kubernetes_version }}/bin/windows/amd64/kubelet.exe
    dest: C:\k\kubelet.exe

- name: Download kubeadm
  win_get_url:
    url: https://dl.k8s.io/{{ kubernetes_version }}/bin/windows/amd64/kubeadm.exe
    dest: C:\k\kubeadm.exe

- name: Download wins
  win_get_url:
    url: https://github.com/rancher/wins/releases/download/v0.0.4/wins.exe
    dest: C:\k\wins.exe

- name: Create Docker host network
  ansible.windows.win_shell: docker network create -d nat host

- name: Create wins service
  ansible.windows.win_shell: wins.exe srv app run --register

- name: Start wins service
  ansible.windows.win_service:
    name: rancher-wins
    state: restarted

- name: Create other Kubernetes (and related) directories
  win_file:
    path: "{{ item }}"
    state: directory
  loop:
    - 'C:\var\log\kubelet'
    - 'C:\var\lib\kubelet\etc\kubernetes'
    - 'C:\etc\kubernetes\pki'
    - 'C:\Program Files\nssm'
    - 'C:\tmp'

- name: Symbolic Link for PKI directory
  ansible.windows.win_shell: New-Item -path C:\var\lib\kubelet\etc\kubernetes\pki -type SymbolicLink -value C:\etc\kubernetes\pki\

- name: Download nssm
  win_get_url:
    url: https://k8stestinfrabinaries.blob.core.windows.net/nssm-mirror/nssm-2.24.zip
    dest: "C:\\nssm.zip"

- name: Unzip nssm
  community.windows.win_unzip:
    src: "C:\\nssm.zip"
    dest: "C:\\tmp"

- name: Move nssm executable to correct location
  win_copy:
    src: "C:\\tmp\\nssm-2.24\\win64\\nssm.exe"
    dest: "C:\\Program Files\\nssm\\nssm.exe"
    remote_src: true

- name: Add NSSM directory to path
  ansible.windows.win_path:
    elements:
      - "C:\\Program Files\\nssm\\"

- name: Create Kubelet firewall rule
  community.windows.win_firewall_rule:
    name: Kubelet
    localport: 10250
    action: allow
    direction: in
    protocol: tcp
    state: present
    enabled: yes

Most of the steps above are taken from the Windows node preparation Powershell scripts in the Windows SIG Tools repository for Kubernetes. This can be seen here.

Some of the steps differ, in that the Powershell script also starts and registers the Kubelet as a service. It also prepares the Kubelet configuration. These tasks can take place when the node boots for the first time, so they are omitted from this playbook.

To summarize what it does: -

  • Creates a directory for Kubernetes binaries called C:\k
  • Adds it to the $PATH variable
  • Downloads the Windows version of kubelet and kubeadm
  • Downloads wins
    • wins is a tool from Rancher that allows containers to talk to the Windows host they are running on
  • Creates a Docker network in nat mode so that containers can talk to the internet initially
  • Starts the wins service
  • Creates all the directories we need
  • Links C:\var\lib\kubelet\etc\kubernetes\pki to C:\etc\kubernetes\pki\
  • Downloads nssm (the Non-Sucking Service Manager)
  • Unzips nssm, moves it to the correct directory, and adds the directory to the $PATH variable
  • Adds a firewall entry in for the Kubelet on TCP:10250 so that the Control Plane nodes can talk to it

After this, all the relevant dependencies are installed, ready for joining a Kubernetes cluster.

Cloudbase-Init

As noted, Cloudbase-Init provides cloud-init functionality to Windows. This means that we can run commands on first boot of the image, providing customization that is only required once (e.g. setting the hostname, configuring the Kubernetes cluster it will connect to, etc).

For the Cloudbase-Init role, we have the following tasks in our tasks/main.yml file: -

---
# tasks file for cloudbase-init
  - name: Get Cloudbase-Init
    win_get_url:
      url: https://cloudbase.it/downloads/CloudbaseInitSetup_x64.msi
      dest: C:\Users\Administrator\cloudbase-init.msi

  - name: Install Cloudbase-Init
    win_package:
      path: C:\Users\Administrator\cloudbase-init.msi

  - name: Add Cloudbase Init config files
    win_copy:
      src: "{{ item }}"
      dest: "C:\\Program Files\\Cloudbase Solutions\\cloudbase-init\\conf\\{{ item }}"
    loop:
      - cloudbase-init.conf
      - cloudbase-init-unattend.conf

In this we download the Cloudbase-Init MSI, install it, and then provide configuration files that are customized for our purpose. These configuration files customize what kind of cloud-init format, what cloud-init modules are available and more.

For more details on these configuration files, see the Configuring Cloudbase-Init of my “Using Terraform and Cloud-Init to deploy and automatically monitor Proxmox instances - Windows” post.

Once this is installed, Packer will then be able to run the sysprep command using the Cloudbase-Init included Unattend.xml file: -

C:\Windows\System32\Sysprep\sysprep.exe 
    /quiet /generalize /oobe  
    /unattend:C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\Unattend.xml

Videos

Below are a couple of videos/screen captures of the process. Both are quite long, with each install taking around 15 minutes, so feel free to skip to the parts that are relevant to you! Both are best viewed in 1080p resolution, and full screen where possible.

Building a Linux-based Kubernetes image for KVM using Packer and Ansible

Building a Windows-based Kubernetes image for KVM using Packer and Ansible

Next steps

The next step after this will be to move the generated images to your KVM storage location. For me this is in /var/lib/libvirt/images (the default install location).

# Move the Linux image
$ mv artifacts/qemu/debian10-kube/debian10 /var/lib/libvirt/images/debian10-kube.qcow2

# Move the Windows image
$ mv artifacts/qemu/win2019-kube/windows-server-2019-standard-x64-eval /var/lib/libvirt/images/debian2019-kube.qcow2

Once this is done, we can create our Terraform configuration to deploy the Kubernetes cluster. This will be covered in Part 2 of this series.

Summary

In this, we’ve seen how we can use Packer and Ansible to create both Linux and Windows images. While Ansible is well known for configuration management of Linux instances, it is just as capable of managing Windows, BSD, networking equipment and more.

The next part will use our images to create three Kubernetes Control Plane nodes (running Linux), three Linux Kubernetes workers and one Windows Kubernetes worker. This will all be managed by Terraform and Cloud-Init (using the native Cloud-Init packages in Debian, and Cloudbase-Init for Windows).

To see the JSON files, Ansible roles and everything else used in this post, see my Packer Repository on Gitlab