Support Rancher Manager as a cluster installation add-on

[JacieChao]

Feature description

We are consider to add a Rancher Manager as a cluster installation add-on. Users can enable the add-on when installing the cluster which can help them to quickly start using Rancher in different cloud environments. AutoK3s will rely on the capabilities of K3s Helm Controller to realize this.
We will add a built-in add-on manifest(Rancher Manager Latest) for AutoK3s. Users can choose which version are they going to use when creating a cluster.

We need to support customizing the add-on as users needed, such as changing value settings, changing to a private repo chart, etc.

With this design, we can support other kinds of custom add-ons too. And users can also manage their own add-ons to install after K3s installation.

Storage

We need to add a new struct to save add-on template configurations.

Name	Data Type
name	Text
description	Text
manifest	Blob

We support several environment variables in the manifest file.

• internal variables. These internal variables are supported like hostname, node IP, instance-id and so on for cloud providers. AutoK3s will replace these kinds of variables before distributing the manifest to the first master server node.
• customized variables. This kind of variable can be replaced by the --set args.

For example:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: rancher
  namespace: kube-system
spec:
  repo: https://releases.rancher.com/server-charts/latest
  version: {{ .Version | default "" }}
  set:
    hostname: "{{ providerTemplate "public-ip-address" }}:30443"

The default Version value is an empty string. The Version variables will be replaced by autok3s create ... --enable rancher --set rancher.Version=v2.7.3.

Cluster add-on installation

For creating a cluster, we can use the Enable option to save which add-ons have been enabled. And this option is designed to save enabled add-ons for a cluster.

Plugin management

Users can create/update/delete add-ons both from CLI and UI.

If we want to switch to RPM, the version field is required. Users can also set the chart version, and we support switching the chart version when creating a cluster.

Default Template

We support several default template variables currently, including public-ip-address, internal-ip-address and local-hostname

The default setting will be like {{ providerTemplate "public-ip-address" }}
providerTemplate uses the default template parser in AutoK3s
"public-ip-address" is the default parameters that describe above.

The default template above will return the public-ip for the first master node instance.

CLI support

We will add the autok3s add-ons command for this feature and the following sub-command:

• create
• update
• rm
• ls

We can support change settings when creating/updating add-ons by manifest.

autok3s add-ons create <name> --description xxxx --from xxx.yaml --set xxx=xxxx --set xxx=xxxx
autok3s add-ons update <name> -f xxx.yaml --set xxx=xxx --set xxx=xxx

The cluster creation CLI is like below:

autoK3s create ... --enable xxxx --set xxxx.xxxx=xxxx

specified plugin and version, e.g. --enable rancher --set rancher.Version=v2.6.12 or use --enable rancher to install the latest version of rancher.

UI support

Add New Tab with Settings > Add-ons, and the page will show the following columns:

• name
• description

On the list page, we will have a Create and Delete button. The Create page is a normal page, but the following fields are specified.

• manifest: The input should validate yaml file and also support uploading file from local disk.

For each item, need to add Update and Delete actions.

For Create cluster page, in the Add-on Options tab, add an add-on dropdown list. When users choose an add-on, the UI shows an additional input to specify variables with key=value.

[orangedeng]

Instead of chart repo and version, I would prefer using raw manifest to implement this feature. The manifest file can also contains the HelmChart crd as content to do the same thing and it can also create some workloads/configurations without helm support.

[JacieChao]

Sure. This makes sense. I will change the name.
If we are using raw manifest, do we need to handle different versions of manifest with the same configuration?
For example, to support the rancher manager, do we need to create two add-ons like the following two manifests?

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: rancher
  namespace: kube-system
spec:
  repo: https://releases.rancher.com/server-charts/latest
  version: v2.6.12
  set:
    hostname: xxxxx

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: rancher
  namespace: kube-system
spec:
  repo: https://releases.rancher.com/server-charts/latest
  version: v2.7.3
  set:
    hostname: xxxxx

[orangedeng]

That depends on how we implement the feature. If we treat the manifest as template and the user can deploy the manifest with customized variables while create k3s cluster, it could be only one manifest list like following:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: rancher
  namespace: kube-system
spec:
  repo: https://releases.rancher.com/server-charts/latest
  version: ${Version}
  set:
    hostname: xxxxx

autok3s create .... --enable rancher --set Version=v2.7.3

[JacieChao]

--enable is designed to enable multiple add-ons. Would it be ok to use --enable rancher,myapp --set rancher.Version=v2.7.3?

[orangedeng]

Sure. It makes sense. It would be great that the user can set default args in manifest and overrides them when creating cluster.

[orangedeng]

It would be nicer that creating add-ons from manifest file like autok3s create add-ons <name> -f xxxx.yaml --set xxx=xxx --set xxx=xxx

[orangedeng]

The scripts seems dangerous to me. Can you give an example to explain how to use the scripts parameter?

[JacieChao]

The script is using like this

#!/bin/bash

ec2_ip=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4)
port=30443

users can use ${ec2_ip}:${port} as hostname put it into set values.

The script is just used to initialize something like environment values for chart setting.

[orangedeng]

when the scripts get executed? Will it get executed every node of your k3s cluster?

[JacieChao]

The script will just run on the server node. Yes, the script is dangerous to execute on the host.
I changed it to env-args. If users want to use variables in their manifest, they can set env-args with name=value. AutoK3s will get the environment value and fill the manifest file with new value.

[orangedeng]

I would prefer to keep it simple enough to users. Just give them a limited list of variables(like node ip/node hostname and etc) that we can get from the context and let the user use in their manifests. If the users want to have complex operation on their resource, they should use a helm chart to do so.

[JacieChao]

Sure. I will fix it.

[ly5156]

UI Demo

1. list page:

2. create/edit page

3. create cluster Add-on Options tab

[orangedeng]

Maybe it needs a kv map for add-on to store the default value of variables.

[JacieChao]

Will it be ok that the default value can be set in the manifest like ${Version:""}? The new variables will replace the default value by --set args.

[orangedeng]

Sure. But what kind of template script is it? kind of like bash script. It needs to make sure that we have the template engine to vendor the manifest.

[JacieChao]

We can use Masterminds/sprig as the template engine. The template script will use like below:

version: {{ .Version | default "" }}