Multiple issues in Multus docs

[brandond]

Refs:

• K3s Multus + Whereabouts doesn't work k3s#10545
• CNI bin dir changes with K3s version k3s#10869

1. The docs should recommend deploying a HelmChart resource with the correct options, instead of telling people to create a values file and then run helm repo add / helm install. There's no reason to have people do that by hand when we have a resource for this.

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: multus
  namespace: kube-system
spec:
  repo: https://rke2-charts.rancher.io
  chart: rke2-multus
  targetNamespace: kube-system
  valuesContent: |-
    config:
      fullnameOverride: multus
      cni_conf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        binDir: /var/lib/rancher/k3s/data/cni
        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
    rke2-whereabouts:
      fullnameOverride: whereabouts
      enabled: true
      cniConf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        binDir: /var/lib/rancher/k3s/data/cni

2. The Whereabouts docs do not work out of the box because whereabouts will not look under /var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d for config. There are two ways to fix this; either one will work:
   1.ln -s /var/lib/rancher/k3s/agent/etc/cni /etc/ on every node in the cluster
   2. Reference the correct paths in the NetworkAttachmentDefinition configuration_path and kubeconfig:

      apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: whereabouts-conf
      spec:
        config: '{
            "cniVersion": "1.0.0",
            "name": "whereabouts-conf",
            "type": "macvlan",
            "master": "eth0",
            "mode": "bridge",
            "ipam": {
              "type": "whereabouts",
              "range": "172.17.0.0/24",
              "exclude": [
                "172.17.0.0/25"
              ],
              "gateway": "172.17.0.1",
              "configuration_path": "/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.conf",
              "kubernetes": {
                "kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
              }
            }
          }'