-
Notifications
You must be signed in to change notification settings - Fork 438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gloo fails after upgrade from 1.14.27 to 1.15.x #8968
Comments
Related issue: istio/istio#46321
We introduced Envoy 1.26 as part of: #8454 Internal slack context: https://solo-io-corp.slack.com/archives/G01EERAK3KJ/p1702392220039389 |
Thanks for writing up this issue. This is indeed a bug due to the changes in the Envoy 1.26 release, however it will only occur when using Our recommended approach is to rely on gRPC EDS instead of REST. Is that a feasible solution for now? If so, can you confirm that when using gRCP EDS, that this NACK no longer occurs? |
Hi, I can confirm setting Shall I close this issue, or do you want to keep it open until the root cause is fixed? |
Glad to hear that it solves your issue. We also have moved to generally recommending against using resteds so hopefully you will also get a nice performance gain with this change! That being said please do leave this issue open as we would like to fix it and update our tests to catch this proactively in the future. |
Alright we will be fixing this issue, updating docs around when to use useresteds (we have some comments on history and why we dont use it on the function but not in docs), and then will scope out what testing effort we would need to have confidence in this feature going forwards. |
Update:
Remaining work
|
Scoping test refactoring
|
Update
Implementation Details
// set Type = EDS if we have endpoints for the upstream
if eps, ok := upstreamRefKeyToEndpoints[upstream.GetMetadata().Ref().Key()]; ok && len(eps) > 0 {
xds.SetEdsOnCluster(out, t.settings)
}
ClusterNames: []string{defaults.GlooRestXdsName},
|
UpdateBecause of the complications with automated testing around this functionality, we have decided to proceed with the following two PRs:
REST EDS is not recommended for use, as it is less performant and does not offer any benefits over the default GRPC EDS. As we do not expect further enhancements to the way Gloo Edge handles REST EDS, we are proceeding with manual validation for the 1.15.x change |
We're time boxing the remaining work on this issue to be completed by EOD Jan 12. The goal is to complete as much of this checklist as possible:
|
Please note that Gloo Edge 1.16 and 1.17 are not susceptible to this issue, as they depend on envoy-gloo/envoy-gloo-ee 1.27, each of which has included the REST config subscription extension in their extensions build config for some time: |
Closing this issue as we have merged the relevant work. The fix will be available in Gloo Edge OSS 1.15.20 and Gloo Edge Enterprise 1.15.11 |
Thank you for the awesome work! |
Gloo Edge Product
Open Source
Gloo Edge Version
v1.15.x
Kubernetes Version
v1.24
Describe the bug
Upgrading from Gloo 1.14.x to 1.15.x causes Gloo to fail entirely. Everything starts, glooctl check says everything is OK, but Gloo doesn't route anything. I get warning as below for all upstreams:
It's a very long list.
Expected Behavior
Gloo working after upgrade
Steps to reproduce the bug
Upgrade Gloo from 1.14.27 to 1.15.x (I tried all patch versions).
Additional Environment Detail
Using consul upstreams discovery (but gloo fails for both consul and kube upstreams).
Running in EKS.
Helm values:
Additional Context
No response
The text was updated successfully, but these errors were encountered: