KCC Composition Sample (jinja2) for Eventarc

experiments/compositions/samples/Eventarc/README.md

@@ -0,0 +1,61 @@
+# Eventarc with GCS Trigger - KCC Composition
+
+This repository provides a KCC Compositions approach to setting up an event-driven workflow that triggers a Cloud Workflow when objects are finalized in a Google Cloud Storage (GCS) bucket.
+
+**Key Components**
+
+* **KCC Composition:** Manages the creation of the Eventarc trigger and associated resources.
+* **Custom Resource Definition (CRD):** Defines a CRD for `EventarcConfig` to simplify Eventarc trigger configuration.
+* **Custom Resource (CR):** Provides a custom resource to specify the Eventarc trigger configuration.
+* **Storage Notification:** Configures a `StorageNotification` to publish events from the GCS bucket to a Pub/Sub topic.
+* **Service Account:** Creates a service account with necessary permissions for Eventarc and Pub/Sub.
+
+**Prerequisites**
+
+* **Google Cloud Project:** With necessary APIs enabled (Cloud Storage, Pub/Sub, Eventarc, Cloud Workflows).
+* **Kubernetes Cluster:** With Config Connector installed.
+* **Service Account:** With IAM permissions to manage GCS buckets, Pub/Sub topics, Eventarc triggers, and Cloud Workflows.
+* **Cloud Workflow:** The workflow you want to trigger.
+
+**Why Getters and the Context API are Used**
+
+This configuration utilizes KCC's `GetterConfiguration` and `Context` API to improve resource management and streamline deployments.
+
+* **Getters (`GetterConfiguration`)**
+
+ Getters allow the composition to extract values from resources within the cluster. In this case, they ensure that the `StorageNotification` resource is created only after the GCS bucket and Pub/Sub topic are fully available, preventing dependency errors.
+
+* **Context API (`Context`)**
+
+ The Context API provides a way to define contextual information, such as the project ID, that can be accessed by the composition. This avoids redundant specification of the project ID for each resource and promotes centralized configuration management.
+
+**Deployment Steps**
+
+1. **Apply `sa.yaml`:** Creates the service account and grants it necessary permissions.
+
+ ```bash
+ kubectl apply -f sa.yaml -n config-control
+
+2. **Apply `eventarcconfigs-crd.yaml`:** Creates the CRD (Custom Resource Definition) for EventarcConfig.
+
+ ```bash
+ kubectl apply -f eventarcconfigs-crd.yaml -n config-control
+
+3. **Apply `eventarc-composition.yaml`:** Creates the KCC composition and the CRD for EventarcConfig.
+
+ ```bash
+ kubectl apply -f eventarc-composition.yaml -n config-control
+
+4. **Apply `facade.yaml`:** Creates the EventarcConfig custom resource, triggering the composition to create the Eventarc trigger, GCS bucket, Pub/Sub topic, and StorageNotification.
+
+ ```bash
+ kubectl apply -f facade.yaml -n config-control
+
+To delete the resources, delete the YAML files in reverse order:
+
+```bash
+kubectl delete -f facade.yaml -n config-control
+kubectl delete -f eventarc-composition.yaml -n config-control
+kubectl delete -f eventarcconfigs-crd.yaml -n config-control
+kubectl delete -f sa.yaml -n config-control
+```

experiments/compositions/samples/Eventarc/eventarc-composition.yaml

@@ -0,0 +1,135 @@
+apiVersion: composition.google.com/v1alpha1
+kind: Context
+metadata:
+ name: context
+ namespace: config-control
+spec:
+ project: # Replace with your project ID
+
+---
+apiVersion: composition.google.com/v1alpha1
+kind: GetterConfiguration
+metadata:
+ name: gcs-bucket-getter
+ namespace: config-control
+spec:
+ valuesFrom:
+ - name: bucket
+ resourceRef:
+ group: storage.cnrm.cloud.google.com
+ version: v1beta1
+ kind: StorageBucket
+ resource: storagebuckets
+ name: # Replace with your bucket name
+ fieldRef:
+ - path: ".metadata.name"
+ as: name
+
+---
+apiVersion: composition.google.com/v1alpha1
+kind: GetterConfiguration
+metadata:
+ name: pubsub-topic-getter
+ namespace: config-control
+spec:
+ valuesFrom:
+ - name: topic
+ resourceRef:
+ group: pubsub.cnrm.cloud.google.com
+ version: v1beta1
+ kind: PubSubTopic
+ resource: pubsubtopics
+ name: # Replace with your topic name
+ fieldRef:
+ - path: ".metadata.name"
+ as: name
+
+---
+apiVersion: composition.google.com/v1alpha1
+kind: Composition
+metadata:
+ name: complete-eventarc-composition
+ namespace: config-control
+spec:
+ inputAPIGroup: eventarcconfigs.idp.mycompany.com
+ expanders:
+ # STAGE 2: Create the StorageBucket and PubSubTopic resources
+ - type: jinja2 # create-gcs-bucket
+ name: create-gcs-bucket
+ template: |
+ apiVersion: storage.cnrm.cloud.google.com/v1beta1
+ kind: StorageBucket
+ metadata:
+ name: {{ eventarcconfigs.spec.bucketName }}
+ namespace: {{ eventarcconfigs.spec.namespace }}
+ spec:
+ uniformBucketLevelAccess: true
+ - type: jinja2 # create-pubsub-topic
+ name: create-pubsub-topic
+ template: |
+ apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
+ kind: PubSubTopic
+ metadata:
+ name: {{ eventarcconfigs.spec.topicName }}
+ namespace: {{ eventarcconfigs.spec.namespace }}
+
+ # STAGE 3: Wait for the StorageBucket and PubSubTopic to be ready
+ - type: getter # get-gcs-bucket
+ version: v0.0.1
+ name: get-gcs-bucket
+ template: ""
+ configref:
+ name: gcs-bucket-getter
+ namespace: # YOUR NAMESPACE
+
+ - type: getter # get-pubsub-topic
+ version: v0.0.1
+ name: get-pubsub-topic
+ template: ""
+ configref:
+ name: pubsub-topic-getter
+ namespace: # YOUR NAMESPACE
+
+ # STAGE 4: Create the EventarcTrigger and StorageNotification using extracted values
+ - type: jinja2
+ name: create-eventarc-trigger
+ template: |
+ apiVersion: eventarc.cnrm.cloud.google.com/v1beta1
+ kind: EventarcTrigger
+ metadata:
+ name: {{ eventarcconfigs.spec.triggerName }}
+ namespace: {{ context.spec.namespace }}
+ spec:
+ destination:
+ workflowRef:
+ external: "projects/{{ context.spec.project }}/locations/{{eventarcconfigs.spec.location}}/workflows/{{eventarcconfigs.spec.workflowName}}"
+ location: {{eventarcconfigs.spec.location}}
+ serviceAccountRef:
+ name: eventarctrigger-iam-gsa
+ transport:
+ pubsub:
+ topicRef:
+ name: {{ values.topic.name }}
+ namespace: config-control
+ matchingCriteria:
+ - attribute: "type"
+ value: "google.cloud.pubsub.topic.v1.messagePublished"
+ projectRef:
+ external: "projects/{{ context.spec.project }}"
+
+ - type: jinja2 # create-storage-notification
+ name: create-storage-notification
+ template: |
+ apiVersion: storage.cnrm.cloud.google.com/v1beta1
+ kind: StorageNotification
+ metadata:
+ name: bucket-notification
+ namespace: {{ eventarcconfigs.spec.namespace }}
+ spec:
+ bucketRef:
+ name: {{ values.bucket.name }}
+ topicRef:
+ name: {{ values.topic.name }}
+ eventTypes:
+ - "OBJECT_FINALIZE"
+ payloadFormat: JSON_API_V1

experiments/compositions/samples/Eventarc/eventarcconfigs-crd.yaml

@@ -0,0 +1,34 @@
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+ name: eventarcconfigs.idp.mycompany.com
+spec:
+ group: idp.mycompany.com
+ versions:
+ - name: v1
+ served: true
+ storage: true
+ schema:
+ openAPIV3Schema:
+ type: object
+ properties:
+ spec:
+ type: object
+ properties:
+ triggerName:
+ type: string
+ workflowName:
+ type: string
+ location:
+ type: string
+ topicName:
+ type: string
+ bucketName:
+ type: string
+ scope: Namespaced
+ names:
+ plural: eventarcconfigs
+ singular: eventarcconfig
+ kind: EventarcConfig
+ shortNames:
+ - eac

experiments/compositions/samples/Eventarc/facade.yaml

@@ -0,0 +1,12 @@
+apiVersion: idp.mycompany.com/v1
+kind: EventarcConfig
+metadata:
+ name: ea-test
+ namespace: config-control
+spec:
+ triggerName: bucket-trigger
+ workflowName: workflow-1 # Replace with your workflow path
+ location: us-central1 # Or your desired location
+ serviceAccount: your-service-account # Replace with your service account name
+ topicName: eventarc-topic-1 # Replace with your Pub/Sub topic name
+ bucketName: my-gcs-bucket-101 # Replace with your GCS bucket name

experiments/compositions/samples/Eventarc/sa.yaml

@@ -0,0 +1,49 @@
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMServiceAccount
+metadata:
+ annotations:
+ cnrm.cloud.google.com/project-id: #YOUR_PROJECT_ID
+ name: eventarctrigger-iam-gsa
+ namespace: #YOUR_NAMESPACE
+spec:
+ displayName: ExampleGSA-EventArc-POC
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+ name: eventarctrigger-iam-gsa-wf
+ namespace: config-control
+spec:
+ memberFrom:
+ serviceAccountRef:
+ name: eventarctrigger-iam-gsa
+ role: roles/workflows.admin
+ resourceRef:
+ kind: Project
+ external: #YOUR_PROJECT_ID
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+ name: eventarctrigger-iam-gsa-ev
+ namespace: config-control
+spec:
+ memberFrom:
+ serviceAccountRef:
+ name: eventarctrigger-iam-gsa
+ role: roles/eventarc.admin
+ resourceRef:
+ kind: Project
+ external: #YOUR_PROJECT_ID
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+ name: gcs-bucket-pubsub-publisher
+ namespace: config-control
+spec:
+ member: serviceAccount:service-{#YOUR_PROJECT_ID}@gs-project-accounts.iam.gserviceaccount.com
+ role: roles/pubsub.publisher
+ resourceRef:
+ kind: PubSubTopic
+ name: #YOUR_PUBSUB_TOPIC