Elemental Operator - Wrangler -> Kubebuilder migration

[alexander-demicev]

Elemental Operator - Wrangler -> Kubebuilder migration

Established

TBD (do we actually need this section for the time being?)

Status

TBD (probably here it would be enough to keep track of PRs and GH issues, just a list of links, not to duplicate data)

Context

This ADR explains the path of migrating the Elemental Operator from Wrangler to Kubebuiler.

What is Wrangler?

Wrangler is a framework for building custom kubernetes APIs. It provides a similar experience as client-go
where all client interactions with the custom resources are generated. Generated client code is static meaning it has to be generated for every custom resource.
The controller implementation is based on callbacks allowing to react on object changes.

What is kubebuilder?

Kubebuilder is also a framework for building custom kubernetes APIs. It's goal is to reduce the amount of boilerplate and reduce the complexity of building
kubernetes APIs. Kubebuilder comes with a CLI that helps with scaffolding the project and
provides a set of tools for generating controller manifests(CRDs, RBAC resources, webhooks). Client part of Kubebuilder is based on controller-runtime, this
means that client is dynamic and there is no need to generate big chunks of code of every resources.
The controller implementation is based on a process called reconciliation where reconcile loops are
triggered on each event related to the custom resource or other watched additional resources. This approach simplifies controller development by not requiring developers to set up callbacks manually.

Motivation

Using Kubebuilder will improve the overall quality of the Elemental Operator.

• Both Kubebuilder and controller-runtime are backed by a big community. This ensures the projects are well maintained and there is always an ongoing work on improving them.
  An example of these tools usage is Cluster API and all it's providers.
• Both tools are greatly documented: Kubebuilder link, controller-runtime link.
• Using kubebuilder will increase the maintainability by simplifying codebase design. Dynamic client requires less boilerplate and reconcile loops will make code readable as it's executed sequentially and there is no need to
  setup event handlers manually.
• controller-runtime provides a great tool for testing code, envtest library and [setup-envtest] binary will setup an instance of etcd and the Kubernetes API
  server allowing unit tests to use real kubernetes client and avoid working with the fake one.
• Usually projects built with Kubebuilder have a similar project structure as a result of scaffolding provided by kubebuilder CLI.

Goals

• Migrate Elemental Operator to the new codebase, built using Kubebuilder.

Non-Goals

• Introduce breaking API changes making custom resources incompatible after migration.

Design details

1. Set up second container

In order to not break anything in the current implementation we plan to work on migration in a separate branch as it's a big change. The migration of the controllers can be done in steps where we replace the controllers one
by one making sure that nothing breaks at any point of migration.
To be able to do so as a first step we can add a new command to the operator binary and run it as second container near current one in the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elemental-operator
  namespace: cattle-elemental-system
spec:
...
  template:
    ...
    spec:
      containers:
      - args:
        - operator
        ...
        name: elemental-operator
      - args:
        - ctrl-runtime-operator
        name: elemental-operator-ctrl-runtime
        ...

Another reason for having a second container is that options for configuring controller-runtime manager are a bit different from
what we pass to the Wrangler controllers, a separate command with it's own arguments will make the integration cleaner and not require us
touch any existing code.

2. Prepare Makefile

Before we can start writing new controllers we should also ensure the required binaries for code generation and testing are present. Kubebuilder provides binary for generating operator manifests from code like CRDs, RBAC
objects, webhooks, etc. and controller-runtime will require a binary for downloading etcd and Kubernetes API
server for testing. These binaries can be managed with Makefile:

GO_INSTALL := ./scripts/go_install.sh
CONTROLLER_GEN_VER := v0.9.2
CONTROLLER_GEN_BIN := controller-gen
CONTROLLER_GEN := $(abspath $(TOOLS_BIN_DIR)/$(CONTROLLER_GEN_BIN)-$(CONTROLLER_GEN_VER))
CONTROLLER_GEN_PKG := sigs.k8s.io/controller-tools/cmd/controller-gen

SETUP_ENVTEST_VER := v0.0.0-20211110210527-619e6b92dab9
SETUP_ENVTEST_BIN := setup-envtest
SETUP_ENVTEST := $(abspath $(TOOLS_BIN_DIR)/$(SETUP_ENVTEST_BIN)-$(SETUP_ENVTEST_VER))
SETUP_ENVTEST_PKG := sigs.k8s.io/controller-runtime/tools/setup-envtest

$(CONTROLLER_GEN):
  GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) $(CONTROLLER_GEN_PKG) $(CONTROLLER_GEN_BIN) $(CONTROLLER_GEN_VER)

$(SETUP_ENVTEST):
  GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) $(SETUP_ENVTEST_PKG) $(SETUP_ENVTEST_BIN) $(SETUP_ENVTEST_VER)

.PHONY: generate
generate: $(CONTROLLER_GEN)
  $(MAKE) generate-go
  $(MAKE) generate-manifests

.PHONY: generate-manifests
generate-manifests: $(CONTROLLER_GEN)
  $(CONTROLLER_GEN) \
    paths=./api/... \
    paths=./controllers/... \
    crd:crdVersions=v1 \
    rbac:roleName=manager-role \
    output:crd:dir=./config/crd/bases \
    output:rbac:dir=./config/rbac \
    output:webhook:dir=./config/webhook \
    webhook

.PHONY: generate-go
generate-go: $(CONTROLLER_GEN)
  $(CONTROLLER_GEN) \
    object:headerFile=$(ROOT)scripts/boilerplate.go.txt \
    paths=./api/...

3. Include new manifests to the helm chart.

We are using helm chart and it means that we have to include new changes in the helm chart, make generate will generate the CRDs for us, we can place them in the chart directory. Make task will generate CRDs in a
separate directory and it will have the following structure:

config/
  crd/
    bases/
      elemental.cattle.io_$CRD_NAME1.yaml
      elemental.cattle.io_$CRD_NAME2.yaml
      elemental.cattle.io_$CRD_NAME3.yaml
      elemental.cattle.io_$CRD_NAME4.yaml

It will be easier for us to bundle the assets into one file before moving them to the chart. We can use Kustomize as it not only bundles the resources for us but also can apply some customizations in case we need them. Kustomize is also the default tool for releasing manifests for
projects using Kubebuilder.

The Makefile change can look like this:

KUSTOMIZE_VER := v4.5.2
KUSTOMIZE_BIN := kustomize
KUSTOMIZE := $(abspath $(TOOLS_BIN_DIR)/$(KUSTOMIZE_BIN)-$(KUSTOMIZE_VER))
KUSTOMIZE_PKG := sigs.k8s.io/kustomize/kustomize/v4

$(KUSTOMIZE): 
  CGO_ENABLED=0 GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) $(KUSTOMIZE_PKG) $(KUSTOMIZE_BIN) $(KUSTOMIZE_VER)

build-crds: $(KUSTOMIZE)
  $(KUSTOMIZE) build config/crd > chart/templates/crds.yaml

4. Start implementing the controllers.

First, we have to add our API definition to the api/ directory, in this case it's located in the root of the project. Example of the API package from the Kubebuiler tutorial.

Next, we have to setup controller in controllers package(link to example) and add it to the main(another example).

Finally we can run make generate and try out the new controller.

5. Switch off old controllers.

Once new controllers are in place, we can switch off old controllers by removing or commenting them out.

pkg/operator/operator.go

  managedos.Register(ctx, cl, o.DefaultRegistry)
  // registration.Register(ctx, cl)
  machineinventory.Register(ctx, cl)
  managedosversionchannel.Register(ctx, o.requeuer, cl)
  machineinventoryselector.Register(ctx, cl)

Risks and Mitigations

The migration might break existing functionality after upgrade. In order to minimize the risks we have to:

• Avoid changing any user configurable API fields while migrating. This will ensure that the API scheme will stay the same and provide us backward compatibility for the v1beta1 version no matter if we are running new or old
  controllers.
• Avoid adding new API resources to the v1beta1 to make sure that all resource controller have both Wrangler and Kubebuiler implementations.

Useful links

• Draft PR for introducing Kubebuilder.