Bad Gateway Error using nginx Ingress Controller #3848

exaucae · 2023-05-19T14:28:51Z

exaucae
May 19, 2023

Issue submitter TODO list

I've looked up my issue in FAQ
I've searched for an already existing issues here
I've tried running master-labeled docker image and the issue still persists there
I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

Kafka UI is not rendered. Host responds with 502 Error code.

Expected behavior

Kafka UI is rendered

Your installation details

Kafka-UI version 0.7.0

# values.yaml
yamlApplicationConfig:
  kafka:
    clusters:
      - name: bazz
        bootstrapServers: kafka-headless:9092
ingress:
  enabled: true
  annotations:
    external-dns.alpha.kubernetes.io/hostname: kafka-ui.foo.bar
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  host: kafka-ui.foo.bar
  ingressClassName: nginx
nodeSelector:
  role: runners

Steps to reproduce

helm install your-release-name charts/kafka-ui -f values.yml --namespace kafka
curl kafka-ui.foo.bar

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

Screenshots

No response

Logs

>kubectl get svc --namespace kafka
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      
kafka            ClusterIP   xxxxxx           <none>        9092/TCP                     
kafka-headless   ClusterIP   None             <none>        9092/TCP,9094/TCP,9093/TCP  
kafka-kafka-ui   ClusterIP   xxxxxx           <none>        80/TCP

>kubectl get ingress --namespace kafka
NAME             CLASS   HOSTS             ADDRESS  PORTS                                                      
kafka-kafka-ui   nginx   kafka-ui.foo.bar  xxx      80

>kubectl get pods --namespace kafka
NAME                              READY   STATUS    RESTARTS   
kafka-0                           1/1     Running   0          
kafka-kafka-ui-86d9dfc595-vggzp   1/1     Running   0

>kubectl logs kafka-kafka-ui-86d9dfc595-vggzp --namespace kafka | tail
Kubeconfig user entry is using deprecated API version client.authentication.k8s.io/v1alpha1. Run 'aws eks update-kubeconfig' to update.
2023-05-19 15:14:44,704 DEBUG [parallel-2] c.p.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: bazz
2023-05-19 15:14:44,710 DEBUG [parallel-1] c.p.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: bazz
2023-05-19 15:15:14,704 DEBUG [parallel-2] c.p.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: bazz
2023-05-19 15:15:14,711 DEBUG [parallel-1] c.p.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: bazz
2023-05-19 15:15:44,704 DEBUG [parallel-2] c.p.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: bazz
2023-05-19 15:15:44,710 DEBUG [parallel-1] c.p.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: gamma-events
2023-05-19 15:16:14,704 DEBUG [parallel-2] c.p.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: bazz
2023-05-19 15:16:14,711 DEBUG [parallel-1] c.p.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: bazz
2023-05-19 15:16:44,704 DEBUG [parallel-2] c.p.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: bazz
2023-05-19 15:16:44,710 DEBUG [parallel-1] c.p.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: bazz

Additional context

All cluster resources are up and running fine. From pod to ingress. And Bootstrap servers were successfully reached.

No port mapping was overridden, all defaults are in use. I postulate it shouldn't be the issue root cause. a forwarding from ingress -> service -> pod look on point.

No error logged by the container when hitting the endpoint kafka-ui.foo.bar .

Answered by exaucae

May 25, 2023

Here's a positive update, @Haarolean : the issue is fixed. It was due a misconfiguration in the ingress backend protocol. I had it to HTTPS (see values.yml above) but kafka backend service wasn't set up for HTTPS. Reverting to HTTP backend protocol was enough. Although a better fix would be to setup kafka-ui more securely.

In the debugging process, I found the meaning of the logs I was referring to in my previous comment. It translates to this:

You may be running the spring boot app on http only and then hitting it via https (or your browser is redirecting to https), so the http version is not parsed appropriately.

In hope it helps someone else.

Finally, at this very place of showingSer…

View full answer

2023-05-19T14:29:33Z

github-actions[bot]
bot May 19, 2023

Hello there exaucae! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

0 replies

Haarolean · 2023-05-19T16:08:33Z

Haarolean
May 19, 2023

I suggest looking closer into your infra, it's not like it's possible to anyone to look into your infra with k8s and helm

0 replies

exaucae · 2023-05-20T12:42:42Z

exaucae
May 20, 2023
Author

I've checked the infra before drafting #3847. It sure costs me nothing do that again : )

Meanwhile I've got this interesting log in the kafka-ui container. It appears when I hit the host URL kafka-ui.foo.bar. Any chance this hints us in the right way?

I'm curious about how it can try to decode something when I open a URL in the browser. Perphaps it failed to get something from the Kafka cluster?

2023-05-20 12:16:14,698 WARN  [reactor-http-epoll-2] r.n.h.s.HttpServerOperations: [afd40da4, L:/192.168.44.214:8080 - R:/192.168.88.213:52156] Decoding failed: FULL_REQUEST(decodeResult: failure(java.lang.IllegalArgumentException: text is empty (possibly HTTP/0.9)), version: HTTP/1.0, content: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 0, cap: 0))
64
GET /bad-request HTTP/1.0
63
2023-05-20 12:16:14,704 WARN  [reactor-http-epoll-3] r.n.h.s.HttpServerOperations: [ec664d1d, L:/192.168.44.214:8080 - R:/192.168.88.213:52160] Decoding failed: FULL_REQUEST(decodeResult: failure(java.lang.IllegalArgumentException: Invalid separator), version: HTTP/1.0, content: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 0, cap: 0))
62
GET /bad-request HTTP/1.0
61
2023-05-20 12:16:14,709 WARN  [reactor-http-epoll-4] r.n.h.s.HttpServerOperations: [66920159, L:/192.168.44.214:8080 - R:/192.168.88.213:52172] Decoding failed: FULL_REQUEST(decodeResult: failure(java.lang.IllegalArgumentException: text is empty (possibly HTTP/0.9)), version: HTTP/1.0, content: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 0, cap: 0))
60
GET /bad-request HTTP/1.0
59
2023-05-20 12:16:14,999 WARN  [reactor-http-epoll-1] r.n.h.s.HttpServerOperations: [9c907dd1, L:/192.168.44.214:8080 - R:/192.168.88.213:52188] Decoding failed: FULL_REQUEST(decodeResult: failure(java.lang.IllegalArgumentException: text is empty (possibly HTTP/0.9)), version: HTTP/1.0, content: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 0, cap: 0))
58
GET /bad-request HTTP/1.0
57
2023-05-20 12:16:15,010 WARN  [reactor-http-epoll-2] r.n.h.s.HttpServerOperations: [5c60f01e, L:/192.168.44.214:8080 - R:/192.168.88.213:52196] Decoding failed: FULL_REQUEST(decodeResult: failure(java.lang.IllegalArgumentException: text is empty (possibly HTTP/0.9)), version: HTTP/1.0, content: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 0, cap: 0))
56
GET /bad-request HTTP/1.0
55
2023-05-20 12:16:15,016 WARN  [reactor-http-epoll-3] r.n.h.s.HttpServerOperations: [93a3c128, L:/192.168.44.214:8080 - R:/192.168.88.213:52198] Decoding failed: FULL_REQUEST(decodeResult: failure(java.lang.IllegalArgumentException: text is empty (possibly HTTP/0.9)), version: HTTP/1.0, content: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 0, cap: 0))
54
GET /bad-request HTTP/1.0

1 reply

Haarolean May 22, 2023

Any chance this hints us in the right way?

Most likely, your ingress is not configured properly.

Perphaps it failed to get something from the Kafka cluster?

No, you would get stacktraces in this case.

We have a docker-compose example with nginx as a reverse proxy here which works fine.

Also, we use nginx with k8s for our dev env which also works fine:

NAME                     CLASS            HOSTS                                   ADDRESS                                                                      PORTS   AGE
internal-master          nginx-internal   master.internal.kafka-ui.provectus.io                                                                                80      365d
kafka-ui                 nginx            www.kafka-ui.provectus.io               xxx.eu-central-1.elb.amazonaws.com                                           80      473d

exaucae · 2023-05-25T13:57:47Z

exaucae
May 25, 2023
Author

Here's a positive update, @Haarolean : the issue is fixed. It was due a misconfiguration in the ingress backend protocol. I had it to HTTPS (see values.yml above) but kafka backend service wasn't set up for HTTPS. Reverting to HTTP backend protocol was enough. Although a better fix would be to setup kafka-ui more securely.

In the debugging process, I found the meaning of the logs I was referring to in my previous comment. It translates to this:

You may be running the spring boot app on http only and then hitting it via https (or your browser is redirecting to https), so the http version is not parsed appropriately.

In hope it helps someone else.

Finally, at this very place of showingServer errors, it my be useful to have a console.log. Sometimes, you don't have direct access to the cluster. But the browser logs can help you quickstart the debugging in the right way.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bad Gateway Error using nginx Ingress Controller #3848

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 1 reply

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Bad Gateway Error using nginx Ingress Controller #3848

exaucae May 19, 2023

Issue submitter TODO list

Describe the bug (actual behavior)

Expected behavior

Your installation details

Steps to reproduce

Screenshots

Logs

Additional context

Replies: 4 comments · 1 reply

github-actions[bot] bot May 19, 2023

Haarolean May 19, 2023

exaucae May 20, 2023 Author

Haarolean May 22, 2023

exaucae May 25, 2023 Author

exaucae
May 19, 2023

Replies: 4 comments 1 reply

github-actions[bot]
bot May 19, 2023

Haarolean
May 19, 2023

exaucae
May 20, 2023
Author

exaucae
May 25, 2023
Author