Remove bootstrapping of namespaces by integrating them into the helmfile workflow

[OlleLarsson]

Warning

This is public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request
• business confidential information, such as customer names

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• kind/adr

What does this PR do / why do we need this PR?

This PR removes the bootstrap step used to install namespaces in a
cluster. The step has been integrated into the new helmfile setup and
namespaces are now managed by helm.

Two new releases are added: "admin-namespaces" and
"dev-namespaces", the latter of which is only relevant for wc clusters
and it only includes the "alertmanager" namespace as of now. The
"admin-namespace" includes all admin namespaces execpt for the "kube-"
namespaces. The "kube-" namespaces are not managed by helm nor do they
get any PSA or "owner=operator" labels set. Furthermore those namespaces
are exempted from the OPA-Gatekeeper validating and mutating webhooks!

Note, namespaces are not removed by Helm! If you destroy the
namespace releases the namespace resources will be left behind.

Furthermore, this PR sets the restricted PSA labels on the "thanos"
namespace and the privileged PSA labels on the "falco" namespace in SC.
In order set the restricted labels on the "thanos" namespace, seccomp
profile and drop all capabilities is set on the thanos components.

• Fixes Remove bootstrap - helmify #1882

Additional information to reviewers

Screenshots

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts are not affected)
  • The metrics names did change (Grafana dashboards and Prometheus alerts were fixed)
• Logs checks:
  • The logs do not show any errors after the change
• Network Policy checks:
  • Any changed pod is covered by Network Policies
  • The change does not cause any dropped packages in the NetworkPolicy Dashboard
• Pod Security Policy checks:
  • Any changed pod is covered by Pod Security Admission
  • Any changed pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any pods to be blocked by Pod Security Admission or Policies
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Bug checks:
  • The bug fix is covered by regression tests

[simonklb]

I thought it was worth mentioning because it popped up in my head when reading the title:
If I recall correctly, the reason why we left out namespace creation from Helm before was to prevent an accident that uninstalled the "namespace Helm release" which would cascade delete everything else in the cluster. I assume that with helm.sh/resource-policy: keep this will not be a risk?

[OlleLarsson]

I thought it was worth mentioning because it popped up in my head when reading the title: If I recall correctly, the reason why we left out namespace creation from Helm before was to prevent an accident that uninstalled the "namespace Helm release" which would cascade delete everything else in the cluster. I assume that with helm.sh/resource-policy: keep this will not be a risk?

Good point.

So, the helm resource policy keep could be set on all of the namespaces created by the release and that would, as you say, prevent them from being deleted with helm uninstall.

ATM, the keep resource policy has only been set on the kube* namespaces but for those we are thinking of moving out from this release. So we'd need to either set the owner=operator in some other way or configure gatekeeper in such a way that those namespaces are excluded by default (despite them not having the owner label)

[OlleLarsson]

It shouldn't be possible to modify this label of a namespace, so customers shouldn't be able to abuse it.

[OlleLarsson]

Noticied that we didn't set any psa labels on the falco namespace here which I assume that we want to do.

[OlleLarsson]

Still doing some testing but feel free to take a look

[davidumea]

LGTM

[viktor-f]

Looks good, but I have not tested it.
Just one minor thing.