use flagger with kuma

[Gidi233]

Describe the bug

When I tried the Kuma Canary Deployments example, I got the following error:

Events:
  Type     Reason  Age                    From     Message
  ----     ------  ----                   ----     -------
  Warning  Synced  17m                    flagger  podinfo-primary.test not ready: waiting for rollout to finish: observed deployment generation less than desired generation
  Warning  Synced  16m                    flagger  podinfo-primary.test not ready: waiting for rollout to finish: 0 of 2 (readyThreshold 100%) updated replicas are available
  Warning  Synced  16m (x3 over 17m)      flagger  Error checking metric providers: prometheus not avaiable: running query failed: request failed: Get "http://prometheus-server.mesh-observability:80/api/v1/query?query=vector%281%29": dial tcp 10.96.250.180:80: connect: connection refused
  Normal   Synced  16m                    flagger  Initialization done! podinfo.test
  Normal   Synced  10m                    flagger  New revision detected! Scaling up podinfo.test
  Warning  Synced  9m32s                  flagger  canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 2 (readyThreshold 100%) updated replicas are available
  Normal   Synced  9m2s                   flagger  Starting canary analysis for podinfo.test
  Normal   Synced  9m2s                   flagger  Pre-rollout check acceptance-test passed
  Normal   Synced  9m2s                   flagger  Advance podinfo.test canary weight 5
  Warning  Synced  6m32s (x5 over 8m32s)  flagger  Halt advancement no values found for kuma metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found
  Warning  Synced  6m2s                   flagger  Rolling back podinfo.test failed checks threshold reached 5
  Warning  Synced  6m2s                   flagger  Canary failed! Scaling down podinfo.test

the Flagger logs:

{"level":"info","ts":"2024-08-05T03:17:05.094Z","caller":"flagger/main.go:149","msg":"Starting flagger version 1.38.0 revision b6ac5e19aa7fa2949bbc8bf37a0f6c1e31b1745d mesh provider kuma"}
{"level":"info","ts":"2024-08-05T03:17:05.095Z","caller":"clientcmd/client_config.go:659","msg":"Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work."}
{"level":"info","ts":"2024-08-05T03:17:05.095Z","caller":"clientcmd/client_config.go:659","msg":"Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work."}
{"level":"info","ts":"2024-08-05T03:17:05.100Z","caller":"flagger/main.go:441","msg":"Connected to Kubernetes API v1.30.0"}
{"level":"info","ts":"2024-08-05T03:17:05.100Z","caller":"flagger/main.go:294","msg":"Waiting for canary informer cache to sync"}
{"level":"info","ts":"2024-08-05T03:17:05.100Z","caller":"cache/shared_informer.go:313","msg":"Waiting for caches to sync for flagger"}
{"level":"info","ts":"2024-08-05T03:17:05.202Z","caller":"cache/shared_informer.go:320","msg":"Caches are synced for flagger"}
{"level":"info","ts":"2024-08-05T03:17:05.202Z","caller":"flagger/main.go:301","msg":"Waiting for metric template informer cache to sync"}
{"level":"info","ts":"2024-08-05T03:17:05.202Z","caller":"cache/shared_informer.go:313","msg":"Waiting for caches to sync for flagger"}
{"level":"info","ts":"2024-08-05T03:17:05.302Z","caller":"cache/shared_informer.go:320","msg":"Caches are synced for flagger"}
{"level":"info","ts":"2024-08-05T03:17:05.302Z","caller":"flagger/main.go:308","msg":"Waiting for alert provider informer cache to sync"}
{"level":"info","ts":"2024-08-05T03:17:05.302Z","caller":"cache/shared_informer.go:313","msg":"Waiting for caches to sync for flagger"}
{"level":"info","ts":"2024-08-05T03:17:05.403Z","caller":"cache/shared_informer.go:320","msg":"Caches are synced for flagger"}
{"level":"error","ts":"2024-08-05T03:17:05.405Z","caller":"flagger/main.go:208","msg":"Metrics server http://prometheus-server.mesh-observability:80 unreachable running query failed: request failed: Get \"http://prometheus-server.mesh-observability:80/api/v1/query?query=vector%281%29\": dial tcp 10.96.250.180:80: connect: connection refused","stacktrace":"main.main\n\t/workspace/cmd/flagger/main.go:208\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
{"level":"info","ts":"2024-08-05T03:17:05.405Z","caller":"server/server.go:45","msg":"Starting HTTP server on port 8080"}
{"level":"info","ts":"2024-08-05T03:17:05.405Z","caller":"controller/controller.go:191","msg":"Starting operator"}
{"level":"info","ts":"2024-08-05T03:17:05.405Z","caller":"controller/controller.go:200","msg":"Started operator workers"}
{"level":"info","ts":"2024-08-05T03:18:08.802Z","caller":"controller/controller.go:312","msg":"Synced test/podinfo"}
{"level":"info","ts":"2024-08-05T03:18:15.426Z","caller":"router/kubernetes_default.go:175","msg":"Service podinfo-canary.test created","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:18:15.445Z","caller":"router/kubernetes_default.go:175","msg":"Service podinfo-primary.test created","canary":"podinfo.test"}
{"level":"error","ts":"2024-08-05T03:18:15.446Z","caller":"controller/events.go:39","msg":"Error checking metric providers: prometheus not avaiable: running query failed: request failed: Get \"http://prometheus-server.mesh-observability:80/api/v1/query?query=vector%281%29\": dial tcp 10.96.250.180:80: connect: connection refused","canary":"podinfo.test","stacktrace":"github.com/fluxcd/flagger/pkg/controller.(*Controller).recordEventErrorf\n\t/workspace/pkg/controller/events.go:39\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).advanceCanary\n\t/workspace/pkg/controller/scheduler.go:207\ngithub.com/fluxcd/flagger/pkg/controller.CanaryJob.Start.func1\n\t/workspace/pkg/controller/job.go:35"}
{"level":"info","ts":"2024-08-05T03:18:15.463Z","caller":"canary/deployment_controller.go:323","msg":"Deployment podinfo-primary.test created","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:18:15.464Z","caller":"controller/events.go:45","msg":"podinfo-primary.test not ready: waiting for rollout to finish: observed deployment generation less than desired generation","canary":"podinfo.test"}
{"level":"error","ts":"2024-08-05T03:18:45.413Z","caller":"controller/events.go:39","msg":"Error checking metric providers: prometheus not avaiable: running query failed: request failed: Get \"http://prometheus-server.mesh-observability:80/api/v1/query?query=vector%281%29\": dial tcp 10.96.250.180:80: connect: connection refused","canary":"podinfo.test","stacktrace":"github.com/fluxcd/flagger/pkg/controller.(*Controller).recordEventErrorf\n\t/workspace/pkg/controller/events.go:39\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).advanceCanary\n\t/workspace/pkg/controller/scheduler.go:207\ngithub.com/fluxcd/flagger/pkg/controller.CanaryJob.Start.func1\n\t/workspace/pkg/controller/job.go:39"}
{"level":"info","ts":"2024-08-05T03:18:45.419Z","caller":"controller/events.go:45","msg":"podinfo-primary.test not ready: waiting for rollout to finish: 0 of 2 (readyThreshold 100%) updated replicas are available","canary":"podinfo.test"}
{"level":"error","ts":"2024-08-05T03:19:15.416Z","caller":"controller/events.go:39","msg":"Error checking metric providers: prometheus not avaiable: running query failed: request failed: Get \"http://prometheus-server.mesh-observability:80/api/v1/query?query=vector%281%29\": dial tcp 10.96.250.180:80: connect: connection refused","canary":"podinfo.test","stacktrace":"github.com/fluxcd/flagger/pkg/controller.(*Controller).recordEventErrorf\n\t/workspace/pkg/controller/events.go:39\ngithub.com/fluxcd/flagger/pkg/controller.(*Controller).advanceCanary\n\t/workspace/pkg/controller/scheduler.go:207\ngithub.com/fluxcd/flagger/pkg/controller.CanaryJob.Start.func1\n\t/workspace/pkg/controller/job.go:39"}
{"level":"info","ts":"2024-08-05T03:19:15.434Z","caller":"router/kubernetes_default.go:175","msg":"Service podinfo.test created","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:19:15.434Z","caller":"controller/scheduler.go:257","msg":"Scaling down Deployment podinfo.test","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:19:15.468Z","caller":"router/kuma.go:105","msg":"TrafficRoute podinfo created","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:19:15.484Z","caller":"controller/events.go:33","msg":"Initialization done! podinfo.test","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:25:15.439Z","caller":"controller/events.go:33","msg":"New revision detected! Scaling up podinfo.test","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:25:45.429Z","caller":"controller/events.go:45","msg":"canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 2 (readyThreshold 100%) updated replicas are available","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:26:15.436Z","caller":"controller/events.go:33","msg":"Starting canary analysis for podinfo.test","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:26:15.444Z","caller":"controller/events.go:33","msg":"Pre-rollout check acceptance-test passed","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:26:15.459Z","caller":"controller/events.go:33","msg":"Advance podinfo.test canary weight 5","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:26:45.435Z","caller":"controller/events.go:45","msg":"Halt advancement no values found for kuma metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:27:15.441Z","caller":"controller/events.go:45","msg":"Halt advancement no values found for kuma metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:27:45.431Z","caller":"controller/events.go:45","msg":"Halt advancement no values found for kuma metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:28:15.440Z","caller":"controller/events.go:45","msg":"Halt advancement no values found for kuma metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:28:45.438Z","caller":"controller/events.go:45","msg":"Halt advancement no values found for kuma metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:29:15.441Z","caller":"controller/events.go:45","msg":"Rolling back podinfo.test failed checks threshold reached 5","canary":"podinfo.test"}
{"level":"info","ts":"2024-08-05T03:29:15.458Z","caller":"controller/events.go:45","msg":"Canary failed! Scaling down podinfo.test","canary":"podinfo.test"}

To Reproduce

The steps and definition are the same as in https://docs.flagger.app/tutorials/kuma-progressive-delivery?fallback=true

Additional context

• Flagger version:1.38.0
• Kubernetes version: 1.30.0
• Service Mesh provider:kuma