Custom validators

[WitoDelnat]

This describes (extensively) how we could approach custom validators.

Requirements

• It should be isomorphic, that is they should run both on NodeJs and the browser.
• It should be possible to create them in an external repository, that is out-of-tree from monokle-core.
• It should be secure to load third-party code.

Proposal

1. Create plugin: custom validation API

This is a temporary poc repository: https://github.com/WitoDelnat/bookish-octo-bassoon

There is a working POC with following interface:

// Notice how custom API has a clean isolated export (the /custom postfix)
import { defineRule, definePlugin } from "@monokle/validation/custom"

const noEmptyAnnotations = defineRule("RCA001", {
  name: "no-empty-annotations",
  description: "Require annotations as metadata.",
  help: "Add any annotation to the Kubernetes resource.",
  validate({ resources }, { report }) {
    resources.forEach((resource) => {
      const annotations = Object.entries(
        resource.content.metadata?.annotations ?? {}
      );
      const hasAnnotations = annotations.length > 0;

      if (!hasAnnotations) {
        report(resource, { path: "metadata.annotations" });
      }
    });
  },
});

export default definePlugin("annotations", {
  description: "Validates your annotations",
  rules: {
    noEmptyAnnotations,
  },
});

Besides metadata, you will see a validate function that has RuleContext and RuleApi parameters:

note: I think we should not expose our Resource directly but instead resource = { $id: resource.id, ...resource.content}.

export declare type RuleContext = {
    /**
     * Resources that should be revalidatd.
     */
    resources: Resource[];
    /**
     * All resources - even those not targeted by incremental revalidation.
     */
    allResources: Resource[];
    /**
     * The global settings object.
     */
    settings?: any;
};

export declare type ReportArgs = {
    path: string; // the path that we should use as an error region. We should maybe also support directly an error region if you work with a parsed document (see below)
    message?: string; // defaults to the description
};
export declare type RuleApi = {
    report(resource: Resource, args: ReportArgs): void; // report an error.
    parse(resource: Resource): Document.Parsed<ParsedNode>; // Access a parsed YAML document so you can implement a visitor. Bad DX but very customizable
};

Important to note is that we the plugin should be the default export for this plugin module.

2. Create bundle: compile typescript and rollup ESM bundler

tsc && rollup --config rollup.config.js

The following configuration is used - it's quite basic.

import { nodeResolve } from "@rollup/plugin-node-resolve";
import { terser } from "rollup-plugin-terser";

export default [
  {
    input: "build/main.js",
    output: [
      {
        file: "plugin.js",
        format: "esm",
      },
    ],
    plugins: [nodeResolve(), terser()],
  },
];

3. Serve the plugin

See below how this would eventually be done but for this poc it's done simple..

npx serve -p 4111 --cors

3. Load the plugin

Different implementation is needed for NodeJs and Browser. This is because NodeJs does not yet support importing ESM modules using HTTP (docs).

Browser

export const VALIDATOR = createMonokleValidator(async pluginName => {
  if (pluginName !== 'annotations') {
    throw new Error('validator_not_found');
  }

  const annotationsPlugin = await import(
    'http://localhost:4111/plugin.js' as unknown as any
  );
  return new SimpleCustomValidator(annotationsPlugin.default, RESOURCE_PARSER);
});

NodeJs

There are two options to work around the restriction:

/**
 * The second attempt, does not use experimental features
 */
async function importWithDataUrl(url: string) {
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(`Error fetching ${url}: ${response.statusText}`);
  }
  const source = await response.text();
  const buff = Buffer.from(source);
  const encodedSource = buff.toString("base64");
  const dataUrl = `data:text/javascript;base64,${encodedSource}`;

  const module = await import(dataUrl);
  return module;
}

/**
 * The first attempt , though it requires the "--experimental-vm-modules" flag to be set.
 *
 * You can do this with an env variable:
 * @config export NODE_OPTIONS="--experimental-vm-modules"
 */
async function importWithVm(url: string) {
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(`Error fetching ${url}: ${response.statusText}`);
  }
  const source = await response.text();
  const context = vm.createContext({});
  const module = new (vm as any).SourceTextModule(source, {
    identifier: url,
    context,
  });
  await module.link(noop);
  await module.evaluate();
  return module.namespace;
}

4. Configure plugins and validate resources

No change needed. Current validation configuration is powerful enough.

Miscellaneous

Codegen types from Kubernetes (core + CRDs)

I've written a script that generates types and typeguards to go from resource.content: any to a fully typed object. This will give intellisense to anyone trying to write a plugin which is a drastic DX improvement.

CLI to manage plugin lifecycle

This would put all pieces together. Below is a basic concept but it has to be iterated on.

# Skaffolds a plugin repository
## note: this can be done exactly how vite does it: https://github.com/vitejs/vite/tree/main/packages/create-vite
monokle plugin validation init    

# Generates typescript from /crd directory
monokle plugin validation codegen 

# Under the hood creates an ESM bundle with rollup 
monokle plugin validation build

# Uploads an unreviewed release? - something to easily test
##note: needs authentication with Monokle Cloud?
monokle plugin validation preview

# Submit repo for an audit by a Monokle member to be published properly.
# These could have a public variant to appear in an easy to explore and install gallery?
##note: needs authentication with Monokle Cloud?
monokle plugin validation submit (--public)

Maybe this is not needed for now as some npm scripts and a submit form would do the trick as well?

Security

For now a combination of manual review and signature should be sufficient? I doubt there will be so many submissions that manual review will be a bottleneck and it's the simplest solution. The signature can be a shasum of the bundle which we then use together with a HTTP Signature header.

Maybe shasum is not needed when we upload manually to our own infra?

Tasks to be done

Plugins [5-6 days]

• Add validation API to monokle-core. [POC available 0.5 day]
• Add create-monokle-plugin repository to bootstrap plugin repository. This should be a copy paste of create-vite. It should have a flag --validator and later can be extended with a flag --template. For now this repository can include the rollup config, codegen script and everything just plainly put in it. Let's get some usage before we do a touch-up of internals. [POC repo available - 1 day]
• Document lifecycle using npx create monokle-plugin --validator, npm run codegen and npm run build - let's avoid CLI for now. npm run preview can run the local server which should be sufficient to test. [POC available - 0.5 day]
• Add infrastructure to serve plugins. For example https://plugins.monokle.com/assets/validation/my-custom-plugin. [0.5 day]
• Add admin script to upload a reviewed plugin [1 day]
• Extend @monokle/validation to fetch a plugin [1 day]

At this point it works with CLI or by adding it to the monokle.validation.yaml.

Plugins configurable in browser [5-8 days]

• Add install / dynamic configure from loaded plugins within the UI to help users configure rules. UX needed.
• This could be taken into consideration when we work on the v2.0 settings of validation which would cut the time to implement quite a lot.
• Can be done in parallel with above plugins.

Plugins gallery [5-8 days]

• A gallery to explore and install new plugins would be a cool nice to have but not immediately needed [5-8 days].

[olensmar]

Thanks @WitoDelnat - i'm trying to understand what the actual workflow/steps would be for both a creator of a plugin and a consumer.. could you provide some example? i.e. "Step by step instructions to create a new validator that can be used by others, "Step by step instructions to use a custom validator when validating resources with monokle cli"?

Also - as an additional layer of security on the node side - could we consider using https://github.com/patriksimek/vm2 for sandboxing the actual plugin (we already use vm2 in Monokle for sandboxing template code)? or does that not play nicely with your proposed approach!?

[WitoDelnat]

Yeah, in retrospect I should've documented it more in the user's experience instead how I chronologically build the POC..

The customer experience

As a user, I want to create a custom annotations validator that checks for the existence of metadata.annotations.

1. Create plugin repository

npx create monokle-plugin --template validator-typescript annotations

2. [advanced] Create plugin using CRDs

Add my custom resource definitions to /crds

npm run codegen

I now see src/__generated__ contains types of resources I could use.

3. Define plugin and rules

I explore the skaffolded repository and see the already present sample definePlugin and update it with my own metadata. I see an example rule and copy paste example-rule, rename it to noMissingAnnotations.ts and implement it. Finally I register the rule in definePlugin.

4. Build and test my plugin

npm run build
npm run preview

I can now try my custom validator using the CLI as follows:

# monokle.validation.yaml option 1 - if we do not recognise the name we try to load it. @preview goes to localhost and otherwise to plugins.monokle.com
plugins:
  annotations@preview: true
rules:
  annotations/no-missing-annotations: "err"

# monokle.validation.yaml option 2
plugins:
  annotations:
    enabled: true
    src: http://localhost:4177/plugin.js
rules:
  annotations/no-missing-annotations: "err"

5. Submit plugin

We got you covered. Within a couple of days we'll have review it and upload it to plugins.monokle.com!

[WitoDelnat]

I'm creating a separate thread for security 👇

Also - as an additional layer of security on the node side - could we consider using https://github.com/patriksimek/vm2 for sandboxing the actual plugin (we already use vm2 in Monokle for sandboxing template code)? or does that not play nicely with your proposed approach!?

[WitoDelnat]

I've read quite a bit about security, both automated static analysis tools and attempts to isolation are insufficient. They would add on top of the manual review instead of replace it. For now I think only the manual review is sufficient to keep it simple - let's see if people want this.

All these things can be escaped if you know your way around it. Even things like V8 isolation are not sufficient and all need proper virtualisation to be secure. Even Cloudflare open source their advanced workerd runtime with explicit mentions that this is not secure on its own.

For our own infrastructure (once we get to GitHub Bot) it might be worth exploring lightweight virtualisation. I read about Kata firecrackers although not sure if they are compatible with GKE. That'll be a nice challenge for Hamid!

[olensmar]

ok - i'm thinking it will be likely that users will want to create validators that are internal to their org/project/etc which might not be something they want to publish in a public repo/project.. i.e. no auditing from our end. How would they host/publish those validators and use them in their (internal) pipelines?

[WitoDelnat]

If they write an internal script that they run on their internal pipelines then it becomes very simple: nothing needs to be done. If the author is the only consumer then there is no benefit to any malicious intent.

Though the complication arrises when we add Monokle (GitHub) Bot which will run it on our pipelines. That's why we will need to explore lightweight virtualisation or this is a blocker. It's more work than using these vm2 tools etc but it's the only thing that will actually provide security instead of a smoke screen of false sense of security.

As for publishing, I'd still recommend that we host them and they can use a CLI to upload private scripts themselves (plus add authentication to the CLI so we can keep an audit log of exactly who uploaded it). The reason is that we can restrict from where we download content on the web application where credentials are at risk so we have more control over this. This is a premium feature so we have the uploader's credit card for if we detect malicious usage - that is you are not anonymous and can't use a throwaway account.

[WitoDelnat]

I was just thinking to add a timeout to limit impact of bad behavior. For instance a 1ms / resource per rule timeout (e.g. 50 resources has to be finished under 50ms) or something like that. As the name says this is really a SimpleCustomValidator for those small checks on certain properties so it should not do anything remotely special.

Your fix remark also made me realise that we should make a copy of the resources we pass in the plugin to avoid malicious rewrite of resources. This should be forbidden.

[olensmar]

Would it make sense to add the optional possibility for validators to provide "quick-fixes" for the errors they find? Most (all?) current OPA validations could be fixed automatically by adding/setting properties/etc - which would be easy to add and extremely helpful in scenarios like code-scanning (where we could generate PRs based on quick-fixes) and in editors where we could have keyboard-actions for applying quick-fixes!?

[WitoDelnat]

Yeah. I was toying with that in this design a while ago, I anticipated a fix({ problem: ValidationResult, resource }, api): void that would allow to rewrite resources. I've been starting to back-peddle a bit on the quick-fix feature as it's usefulness might be limited.

It's more a discussion for elsewhere, but shortly I think it's a fair assumption that the majority of Kubernetes users works with Helm, Kustomize or another higher-level tool and not on vanilla Kubernetes resources. At that point validated resources are read-only and quick-fix won't be feasible. I'd like to focus more on the higher-level tool workflows for now as there will be more value here - but again this is just an assumption I'm playing around with which has to be discussed with the team.