use flatcar as base OS on capo clusters

[cornelius-keller]

Towards #426

User stories

• As a giant swarm customer I want to have a secure and container optimized operating system in my clusters so that I don't need to worry about the performance and security of my Kubernetes clusters

TODO

Upstream qemu build boot on openstack but doesn't contain the valid kernel parameters to make ignition work. We have to either

• use upstream image-builder repository and make a flatcar image for openstack ( image-source: https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_openstack_image.img.bz2) or
• patch the already existing qemu-flatcar image to make ignition work on openstack

Tasks

• try if image-builder qemu images work on openstack #780

additional details:

• flatcar.oem.id isn't set as kernel argument) coreos-metadata service stops and therefore no ignition configuration is being processed

  coreos-metadata[682]: Caused by: Couldn't find flag 'flatcar.oem.id' or 'coreos.oem.id' in cmdline file (/proc/cmdline)
  

• below listed kubeadmconfigtemplate (which got generated by CAPI/CAPO) on a local machine works fine

`kubeadmconfigtemplate` flatcar linux

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha5
kind: OpenStackMachineTemplate
metadata:
  annotations:
    meta.helm.sh/release-name: galaxy-cluster
    meta.helm.sh/release-namespace: org-giantswarm
  labels:
    app: cluster-openstack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 0.10.1
    application.giantswarm.io/team: rocket
    cluster.x-k8s.io/cluster-name: galaxy
    giantswarm.io/cluster: galaxy
    giantswarm.io/organization: giantswarm
    helm.sh/chart: cluster-openstack-0.10.1
  name: galaxy-galaxy-flatcar
  namespace: org-giantswarm
  ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: galaxy
      uid: dd9f6431-14dd-43f6-8687-257d756b0919
spec:
  template:
    spec:
      cloudName: openstack
      flavor: n1.medium
      identityRef:
        kind: Secret
        name: openstack-cloud-config
      image: "Flatcar Container Linux stable-3139.2.0-kube-v1.21.10-mario"
      rootVolume:
        diskSize: 50
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  annotations:
    meta.helm.sh/release-name: galaxy-cluster
    meta.helm.sh/release-namespace: org-giantswarm
  labels:
    app: cluster-openstack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 0.10.1
    application.giantswarm.io/team: rocket
    cluster.x-k8s.io/cluster-name: galaxy
    giantswarm.io/cluster: galaxy
    giantswarm.io/organization: giantswarm
    helm.sh/chart: cluster-openstack-0.10.1
  name: galaxy-galaxy-flatcar-lon-1
  namespace: org-giantswarm
  ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: galaxy
      uid: dd9f6431-14dd-43f6-8687-257d756b0919
spec:
  clusterName: galaxy
  replicas: 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: galaxy
      cluster.x-k8s.io/deployment-name: galaxy-galaxy-flatcar-lon-1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: cluster-openstack
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/version: 0.10.1
        application.giantswarm.io/team: rocket
        cluster.x-k8s.io/cluster-name: galaxy
        cluster.x-k8s.io/deployment-name: galaxy-galaxy-flatcar-lon-1
        giantswarm.io/cluster: galaxy
        giantswarm.io/organization: giantswarm
        helm.sh/chart: cluster-openstack-0.10.1
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: galaxy-galaxy-flatcar-lon-1
      clusterName: galaxy
      failureDomain: gb-lon-1
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
        kind: OpenStackMachineTemplate
        name: galaxy-galaxy-flatcar
      version: v1.22.8
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  annotations:
    meta.helm.sh/release-name: galaxy-cluster
    meta.helm.sh/release-namespace: org-giantswarm
  labels:
    app: cluster-openstack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 0.10.1
    application.giantswarm.io/team: rocket
    cluster.x-k8s.io/cluster-name: galaxy
    giantswarm.io/cluster: galaxy
    giantswarm.io/organization: giantswarm
    helm.sh/chart: cluster-openstack-0.10.1
  name: galaxy-galaxy-flatcar-lon-1
  namespace: org-giantswarm
  ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: galaxy
      uid: dd9f6431-14dd-43f6-8687-257d756b0919
spec:
  template:
    spec:
      ignition:
        containerLinuxConfig:
          strict: false
          additionalConfig: |
            storage:
              links:
              # For some reason enabling services via systemd.units doesn't work on Flatcar CAPI AMIs.
              - path: /etc/systemd/system/multi-user.target.wants/coreos-metadata.service
                target: /usr/lib/systemd/system/coreos-metadata.service
              - path: /etc/systemd/system/multi-user.target.wants/kubeadm.service
                target: /etc/systemd/system/kubeadm.service
              files:
                - contents:
                    source: |
                      ssh-ed25519 <content from KubeadmConfigTemplate>
                  path: /etc/ssh/trusted-user-ca-keys.pem
                  permissions: "0600"
                - contents:
                    source: |
                      # Use most defaults for sshd configuration.
                      Subsystem sftp internal-sftp
                      ClientAliveInterval 180
                      UseDNS no
                      UsePAM yes
                      PrintLastLog no # handled by PAM
                      PrintMotd no # handled by PAM
                      # Non defaults (#100)
                      ClientAliveCountMax 2
                      PasswordAuthentication no
                      TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
                      MaxAuthTries 5
                      LoginGraceTime 60
                      AllowTcpForwarding no
                      AllowAgentForwarding no
                  path: /etc/ssh/sshd_config
                  permission: "0600"
            systemd:
              units:
              - name: kubeadm.service
                dropins:
                - name: 10-flatcar.conf
                  contents: |
                    [Unit]
                    # kubeadm must run after coreos-metadata populated /run/metadata directory.
                    Requires=coreos-metadata.service
                    After=coreos-metadata.service
                    [Service]
                    # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                    Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                    # To make metadata environment variables available for pre-kubeadm commands.
                    EnvironmentFile=/run/metadata/*
      preKubeadmCommands:
      - envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp
      - mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml
      - 'files="/etc/ssh/trusted-user-ca-keys.pem /etc/ssh/sshd_config"; for f in $files; do tmpFile=$(mktemp); cat "${f}" | base64 -d > ${tmpFile}; if [ "$?" -eq 0 ]; then mv ${tmpFile} ${f};fi;  done;'
      - systemctl restart sshd
      format: ignition
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            cloud-provider: external
            node-labels: giantswarm.io/node-pool=galaxy-flatcar-lon-1
          name: ${HOSTNAME}
      postKubeadmCommands:
        - systemctl restart sshd
      users:
        - name: giantswarm
          groups: sudo
          sudo: ALL=(ALL) NOPASSWD:ALL

[cornelius-keller]

@ericgraf please add some links to track upstream progress on flatcar on openstack

[ericgraf]

Response from Kinvolk

We're not currently working on CAPI OpenStack support (and afaik are not planning to; please keep me honest).

Slack thread: https://gigantic.slack.com/archives/C9EUYTKM5/p1643724191649329

[erkanerol]

We need to check collectors in node-exporter. We have some problems with the ubuntu image now. giantswarm/node-exporter-app#136

[bavarianbidi]

imo we have to fix the image build issue by our own.
Once this is done we have to check the state of the known blocking issues and then continue planning a desired shiping date (incl. some refinement stuff what else has to be done by our side)

rocket issues

• qemu image build for flatcar isn't working - need some investigation

  2022/04/12 13:00:49 packer-builder-qemu plugin: stderr:
  ==> flatcar: Error finding "./packer/qemu/linux/flatcar/http/": stat ./packer/qemu/linux/flatcar/http/: no such file or directory
  ==> flatcar: Deleting output directory...
  2022/04/12 13:00:49 [INFO] (telemetry) ending qemu
  ==> Wait completed after 22 seconds 435 milliseconds
  2022/04/12 13:00:49 machine readable: error-count []string{"1"}
  ==> Some builds didn't complete successfully and had errors:
  2022/04/12 13:00:49 machine readable: flatcar,error []string{"Error finding \"./packer/qemu/linux/flatcar/http/\": stat ./packer/qemu/linux/flatcar/http/: no such file or directory"}
  Build 'flatcar' errored after 22 seconds 435 milliseconds: Error finding "./packer/qemu/linux/flatcar/http/": stat ./packer/qemu/linux/flatcar/http/: no such file or directory
  

blocking issues

• base64 encoded files are not decoded on disk - upstream tracking issue

provider independent implementation

• CAPI-App PR (WIP) to use ignition

provider specific implementation (phoenix)

• giantswarm/cluster-aws
• giantswarm/cluster-api-provider-aws-app

general notable issues/PRs

• flatcar tracking issue - CAPI support

[invidian]

Qemu builds should be fixed with kubernetes-sigs/image-builder#829 I think.

[bavarianbidi]

Qemu builds should be fixed with kubernetes-sigs/image-builder#829 I think.

image build was successful. will try to use a CAPO deployment based on flatcar later on.

[bavarianbidi]

by using a flatcar image (build from image-builder PR 829), instance got provisioned but doesn't continue with orchestration.
After setting the log-level on capo to debug, the generated user_data seems to be invalid (by running ignition-validate against the base64 decoded content).

Not sure as this happens as the ignition version in user-data is set to 2.3.0.

[kopiczko]

Proposed upstream change for OCCM kubernetes/cloud-provider-openstack#1928

[glitchcrab]

Pawel needs to tidy up his mess. Ubuntu testing is moving on, plenty of testing to do yet

[kopiczko]

I have my cluster-openstack branch here giantswarm/cluster-openstack#85

It works ok for creating a fresh cluster with both Ubuntu and Flatcar image. For Flatcar you have to set ingintion.enabled: true.

The problem is when upgrading CP nodes from Ubuntu to Flatcar:

      Reason: Upgrade "pawe1-cluster" failed: cannot patch "pawe1" with kind KubeadmControlPlane: admission webhook "validation.kubeadmcontrolplane.controlplane.cluster.x-k8s.io" denied the request: KubeadmControlPlane.controlplane.cluster.x-k8s.io "pawe1" is invalid: [spec.kubeadmConfigSpec.useExperimentalRetryJoin: Forbidden: cannot be modified, spec.kubeadmConfigSpec.format: Forbidden: cannot be modified]

I tried to edit the CR directly and:

# kubeadmcontrolplanes.controlplane.cluster.x-k8s.io "pawe1" was not valid:
# * spec.kubeadmConfigSpec.useExperimentalRetryJoin: Forbidden: cannot be modified

I don't have a solution to that yet.

BTW this needs to be disabled because useExperimentalRetryJoin is not supported with format: ignition.

[invidian]

If you need, I guess useExperimentalRetryJoin could be implemented by Ignition. The only reason this hasn't been implemented is because we wanted to have something shippable, so we did not targeted experimental features.

[kopiczko]

@invidian that would be ideal.

But I guess this will be too long for us to wait for the next CAPI release with that feature included.

I worked around that by disabling the webhook temporarily.

[kopiczko]

Dropping this note, because I already forgot it and had to rediscover. We need to set proper OEM in grub.cfg after the image is built. This is how I do this:

guestmount -m /dev/sda6 -a ./image /mnt

echo 'set oem_id="openstack"' > /mnt/grub.cfg
chmod 644 /mnt/grub.cfg

guestunmount /mnt

We have to do that so the image pulls ignition form user-data on OpenStack.

The OEM partition is completely empty when processed by image-builder.

@invidian do you have an idea if we can somehow add this to image-builder? Maybe adding an env var like FLATCAR_OEM=openstack?

[invidian]

@kopiczko I think it should be fine to follow https://flatcar-linux.org/docs/latest/setup/customization/other-settings/#adding-custom-kernel-boot-options in image-builder for Flatcar, as we boot the Flatcar instance while building the image, then reset it to the fresh state.

[kopiczko]

@invidian I guess there are technical possibilities to achieve this, but what would be the interface? Because it's still a QEMU not OpenStack image from image-builder POV.

So right now you build QEMU image like:

make FLATCAR_CHANNEL=stable FLATCAR_VERSION=3139.2.3 build-qemu-flatcar

I'm thinking about adding extra FLATCAR_OEM env var like:

make FLATCAR_CHANNEL=stable FLATCAR_OEM=openstack  FLATCAR_VERSION=3139.2.3 build-qemu-flatcar

Does that make sense to you?

[invidian]

How about adding build-openstack-flatcar target which can be based on build-qemu-flatcar with extra config required for OEM stuff?

[kopiczko]

I created an upstream issue for this kubernetes-sigs/image-builder#937

[kopiczko]

I opened another issue for the race between kubeadm and containerd: kubernetes-sigs/image-builder#939

[kopiczko]

We also need to add ignition support to mc-bootstrap and cluster-api-app:

• https://github.com/giantswarm/mc-bootstrap/pull/209
• Enable ignition feature gate cluster-api-app#84

[kopiczko]

This is an internal umbrella issue for upstream issues #1264

[cornelius-keller]

done, image builidng and testing tracked in follow up issues.