[WIP] Add tracing with OpenTelemetry #932

meobilivang · 2023-05-26T18:44:22Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Add Distributed Tracing to provide more insights into requests propagated through CAPG. This would enhance the observability of the internal behaviors of CAPG, and offers more clues for developers to troubleshoot the system during development besides the fragmented container logs.

Refer to the project's Proposal for more details on the implementation.

Which issue(s) this PR fixes:

Resolves [EPIC] Improve the supportability of CAPG (LFX Spring 2023 Mentorship) #810
Part of Spring '23 LFX Mentorship

Special notes for your reviewer:

This PR integrates new components into Tiltfile to launch the observability stack (OpenTelemetry Collector & Jaeger)

TODOs:

squashed commits
includes documentation
adds unit tests
Proposal

Release note:

NONE

cc @richardcase @cpanato

k8s-ci-robot · 2023-05-26T18:44:31Z

Hi @meobilivang. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

meobilivang · 2023-05-26T19:09:47Z

Tiltfile

My python environment has the Black linter by default so it just reformats the whole file 😗

meobilivang · 2023-05-26T19:11:55Z

Tiltfile

    )

    k8s_yaml(blob(yaml))

+
+def observability():


setting up OTEL Collector + Jaeger using helm chart

meobilivang · 2023-05-26T19:13:30Z

hack/observability/opentelemetry/values.yaml

+      tls:
+        insecure: true
+    #export to Cloud Trace on GCP
+    googlecloud:


Refer to GCP Cloud trace exporter implementation for OTEL Collector

richardcase · 2023-06-01T20:06:44Z

/ok-to-test

meobilivang · 2023-06-05T14:59:21Z

/retest

meobilivang · 2023-06-05T15:27:15Z

exp/controllers/gcpmanagedcluster_controller.go

+	ctx, span := telemetry.Tracer().Start(
+		ctx, "controllers.GCPManagedClusterReconciler.Reconcile",
+		trace.WithAttributes(
+			attribute.String("name", req.Name),


TODO: Add ReconcileId as an attribute: controller.ReconcileIDFromContext(ctx)

k8s-ci-robot · 2023-06-06T23:35:35Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2024-01-21T02:11:59Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-02-20T02:25:15Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-03-21T03:11:01Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2024-03-21T03:11:06Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

richardcase · 2024-08-13T07:52:10Z

/reopen

k8s-ci-robot · 2024-08-13T07:52:14Z

@richardcase: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2024-08-13T07:52:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: meobilivang
Once this PR has been reviewed and has the lgtm label, please assign cpanato for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

richardcase · 2024-08-13T07:52:37Z

/remove-lifecycle rotten

k8s-triage-robot · 2024-11-11T08:40:00Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

meobilivang added 14 commits May 4, 2023 05:01

install Go otel packages

1ad8b12

set up TracerProvider + tracer object

f955a09

add enableTracing flag

dcfb15b

instrument GCP machine + GCPCluster controllers

49c1ea2

helm charts for jaeger all-in-one + otel collector

deb44dc

templates to set up dev env

bb98f0e

instrument cloud/scope

bc4ea9f

instrument exp/controllers

eb25369

instrument cloud/services/compute

1b98f8e

instrument cloud/services/container

ea54fe7

add sampling rate

560fd0a

Tilt file

ba6261e

debug blocking tracing connection

173a97f

helm charts for jaeger + otel collector

f4a0554

k8s-ci-robot requested review from dims and richardcase May 26, 2023 18:44

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 26, 2023

meobilivang added 3 commits May 26, 2023 18:52

bump go package

fc2812d

bump packages

e4da793

Merge branch 'main' into add-tracing-otel

085b48f

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 26, 2023

meobilivang commented May 26, 2023

View reviewed changes

Tiltfile

)

k8s_yaml(blob(yaml))

def observability():

Copy link

Contributor Author

meobilivang May 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setting up OTEL Collector + Jaeger using helm chart

meobilivang commented May 26, 2023

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 1, 2023

reformat tilt file

bbc5bf5

meobilivang commented Jun 5, 2023

View reviewed changes

meobilivang force-pushed the add-tracing-otel branch from 690d64e to bbc5bf5 Compare June 5, 2023 16:01

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 6, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 21, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 20, 2024

k8s-ci-robot closed this Mar 21, 2024

k8s-ci-robot reopened this Aug 13, 2024

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 13, 2024

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add tracing with OpenTelemetry #932

[WIP] Add tracing with OpenTelemetry #932

meobilivang commented May 26, 2023

k8s-ci-robot commented May 26, 2023

meobilivang May 26, 2023 •

edited

Loading

meobilivang May 26, 2023

meobilivang May 26, 2023

richardcase commented Jun 1, 2023

meobilivang commented Jun 5, 2023

meobilivang Jun 5, 2023 •

edited

Loading

k8s-ci-robot commented Jun 6, 2023

k8s-triage-robot commented Jan 21, 2024

k8s-triage-robot commented Feb 20, 2024

k8s-triage-robot commented Mar 21, 2024

k8s-ci-robot commented Mar 21, 2024

richardcase commented Aug 13, 2024

k8s-ci-robot commented Aug 13, 2024

k8s-ci-robot commented Aug 13, 2024

richardcase commented Aug 13, 2024

k8s-triage-robot commented Nov 11, 2024

[WIP] Add tracing with OpenTelemetry #932

Are you sure you want to change the base?

[WIP] Add tracing with OpenTelemetry #932

Conversation

meobilivang commented May 26, 2023

k8s-ci-robot commented May 26, 2023

meobilivang May 26, 2023 • edited Loading

Choose a reason for hiding this comment

meobilivang May 26, 2023

Choose a reason for hiding this comment

meobilivang May 26, 2023

Choose a reason for hiding this comment

richardcase commented Jun 1, 2023

meobilivang commented Jun 5, 2023

meobilivang Jun 5, 2023 • edited Loading

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 6, 2023

k8s-triage-robot commented Jan 21, 2024

k8s-triage-robot commented Feb 20, 2024

k8s-triage-robot commented Mar 21, 2024

k8s-ci-robot commented Mar 21, 2024

richardcase commented Aug 13, 2024

k8s-ci-robot commented Aug 13, 2024

k8s-ci-robot commented Aug 13, 2024

richardcase commented Aug 13, 2024

k8s-triage-robot commented Nov 11, 2024

meobilivang May 26, 2023 •

edited

Loading

meobilivang Jun 5, 2023 •

edited

Loading