How to lint HelmRelease output?

[jceb]

I've been managing multiple Flux installations in production for about 2 years. I'm very satisfied with how it operates, but I continually encounter the question of how to adequately lint the complete Kubernetes configuration before Flux applies it to the cluster. Handling HelmRelease objects is particularly challenging, as they generate cluster configurations only when the helm-controller unpacks the charts in the cluster. These configurations don't pass through any linter like datree or kubeconform, leading to errors, particularly Kubernetes API incompatibilities, being detected late in the process and causing disruptions.

I couldn't find much information about linting in the documentation. I would greatly appreciate your insights and best practices on this topic.

I have considered the following options, but none of them seem to be a perfect fit:

1. Use HelmRelease objects: Flux supports this, but linting doesn't seem to be possible.
2. Avoid HelmRelease objects and instead use helm template to render the charts directly into the local Git repository. Kustomize combines the output with the rest of the cluster configuration. This is supported by Flux and allows for linting, but it becomes more difficult to manage different chart versions in development, staging, and production environments + it requires a manual "render" step that could easily be forgotten or increase complexity with a separate CI pipeline and separate target repository .
3. Use a Kustomize plugin that accepts HelmRelease objects and renders them locally for linting, while Flux operates in the cluster (as in option 1). This would be an excellent solution, but as far as I understand, there's no such plugin available. Maybe this helm-controller could be adjusted to become such a renderer since it's not trivial to build a custom "script" that takes into account specific HelmRelease features like post-render kustomizations ...

[hiddeco]

Your best option is likely to wait for the refinements we have planned for Q2, in which part of what makes the helm-controller will API-wise be available to the flux binary. This will allow us to build likewise tooling as currently exists for the kustomize-controller (think flux diff, etc), and thereby make it easier to ensure things can be linted and/or debugged.

[jceb]

Great, thank you! That's good to know!

[tijmenvandenbrink]

This would indeed be very helpful. +1 for this feature.

[fheslouin]

@hiddeco could you point me out to the issue or tracking for this feature please ? I can't find any information about it.

[kingdonb]

The Helm Controller changes mentioned above are now targeted for Q3 on the roadmap:

https://fluxcd.io/roadmap/#flux-helm-ga-q3-2023

I don't see a specific issue about flux diff for helmreleases, but I believe it's still part of this effort plan.

[fheslouin]

Thanks @kingdonb for your answers. The roadmap on this topic is not really clear to me as well.

However I figure out that the Helmrelease controller was able to lint (values.yaml <-> values.schema.json validation) a chart before deploying it, the documentation is not clear about it and I can't see any parameters to enable or disable linting.

I don't know what feature is going to ship the v2.1.0, as you said probably the diff for helmrelease ?!

[kingdonb]

This will allow us to build likewise tooling as currently exists for the kustomize-controller (think flux diff, etc), and thereby make it easier to ensure things can be linted and/or debugged.

I'm not sure what you mean by this, the only "linting" that is performed by Helm controller at apply time right now as I understand it is the server-side dry-run and Kustomize post-renderer. This could have the appearance of a linter in function but it's quite a bit more than that, server-side validation ensures that the resources in the Helm chart can all be applied and are valid before any apply takes place.

You can read all about those features in the Helm Controller and HelmRelease docs:
https://fluxcd.io/flux/components/helm/helmreleases/

With a warning that this doc hasn't been updated for 2.0.0, there is a little bit more detail about it in the Helm Controller description here:

https://fluxcd.io/flux/flux-e2e/#microservice-architecture

Doesn't really answer your question, but if I can get my head around exactly what your question is, we can try to answer and address it there, or below in the docs. I think you're saying that Flux already lints charts, and Helm Controller will balk at some things that normal Helm users might not see (like duplicate keys), but those are errors not simple linting violations. Flux processes those manifests through Kustomize, and something between the YAML parser and the Kustomize build/server-side apply step is balking at an error, and refusing to proceed.

That "linting feature" can't be disabled @fheslouin, if I understood you right, it is a side-effect of how Flux applies YAML in general against the Kubernetes API, it must be syntactically valid and pass a server-side dry run, or else Flux will not execute the apply, either because syntactic ambiguity needs to be resolved (even though Helm CLI would not balk at all), or due to the fact that Kubernetes would not allow all of the applies to proceed atomically.

We have a Flux e2e doc where we try to avoid discussing implementation details, but do describe any user-facing impact of internal Flux processes, and this server-side apply + kustomize postrender pipeline is a good candidate to expand that doc. It's user-facing and it's a thing that if you're not aware it exists, you will likely not understand why Flux's behavior differs from the Helm CLI. I'll make a note that we should expand that part of the docs. You might find something helpful in there, (but reading it over myself at a glance, I didn't see anything about either of these issues in the current version.)

We could use a bit more detail here as well:

• https://fluxcd.io/flux/flux-e2e/#helm-controller-reconciles-helmrelease-resources