HTTP3/QUIC not working - please switch to BoringSSL (or LibreSSL or quicTLS)

[the-hotmann]

I have read these issues:

• please add httpv3 protocol  #485
• Image nginx: 1.25.4-alpine3.18 does not have support for HTTPS enabled, and after container startup, an error message is reported using HTTP 3 #867

and think that most people now think, that this dockerized version of Nginx will support HTTP3/QUIC - but it does not, since it is using OpenSSL. The The OpenSSL Compatibility Layer at least does not work for me.

Since Nginx itself supports HTTP3/QUIC, but OpenSSL does not LINK this dockerized version of Nginx (which I love!) does not support HTTP3/QUIC, becasue both things must support it:

• the nginx-version
• the SSL/TLS library used

OpenSSL plans to support HTTP3 for servers from the end of 2024 - but just experimental first (in v3.4.x).
Since this is the current situation I would love to ask to add an additional build (especially the alpine ones) with the addition -boringssl which people (liek me) can use to use and test with HTTP3/QUIC before somewhen OpenSSL supports it.

Note:

• enabeling http3/quic in the config works just fine, BUT your browser (or curl) will not actually use it!
• bascially there are two options: 1.) wait ½-1year and use it with OpenSSL. 2.) use BoringSSL to use it right now!

I used this curl command to verify the actual HTTP Version the server is using:

curl -sIk --http3 https://sub.dom.tld -o/dev/null -w '%{http_version}\n'

or

curl -sIk --http3-only https://sub.dom.tld -o/dev/null -w '%{http_version}\n'

Alternatively you could use the Browsers Dev-Tools to check which protocol actually is getting used - but I prefer the curl version. (curl version should be newer than v8.0.0)

Also please keep in mind, that if you want to use HTTP3/QUIC you need to allow the udp-protocol on Port :443:

services:

  nginx:
    image: nginx:1-alpine-slim
    container_name: nginx
    hostname: nginx
    ports:
      - "443:443"
      - "443:443/udp"
    volumes:
      - "./nginx/templates/:/etc/nginx/templates/:ro"
      - "./nginx/ssl/:/etc/ssl/own/:ro"
    healthcheck:
      test: ["CMD-SHELL", "nc -vz -w1 $(hostname) 443"]
      interval: 1s
      timeout: 1s
      retries: 30
    deploy:
      resources:
        limits:
          memory: 500M
    restart: unless-stopped

if you just open the port :443 this applies to the tcp-protocol only!

I would love to get some feedback from the maintainer of this awesome package and I am ofc open for discussion. :)

[thresheek]

Hi @the-hotmann!

nginx in current docker images works absolutely fine with http3 using openssl via compatibility layer shim nginx team developed.

I suppose something is not configured right in your scenario, so let me illustrate it with my test setup:

$ cat nginx.conf
user nginx;
worker_processes 1;
error_log /dev/stderr info;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    access_log /dev/stdout combined;

    server {
    listen 443 quic reuseport;
    listen 443 ssl;

    add_header Alt-Svc 'h3=":443"; ma=86400';

    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/key.pem;
    location / { return 200 'http3: $http3\n'; }
    }
}

$ ls -1 ssl/
cert.pem
key.pem

$ docker run -d -v $(pwd)/nginx.conf:/etc/nginx/nginx.conf -v $(pwd)/ssl/:/etc/nginx/ssl/ nginx:1.27.1-alpine
2012dd7d365555ddee7c0977d24327fa77160524cfa6a0fe3ade087f8d2236ef

$ docker inspect 2012dd7d365555ddee7c0977d24327fa77160524cfa6a0fe3ade087f8d2236ef | jq '.[].NetworkSettings.Networks.bridge.IPAddress'
"172.17.0.2"

$ docker run -ti --rm alpine/curl-http3:latest curl -v -k --http3 https://172.17.0.2:443/
*   Trying 172.17.0.2:443...
* Server certificate:
*  subject: C=XX; ST=StateName; L=CityName; O=CompanyName; OU=CompanySectionName; CN=CommonNameOrHostname
*  start date: Sep 30 16:58:10 2024 GMT
*  expire date: Sep 28 16:58:10 2034 GMT
*  issuer: C=XX; ST=StateName; L=CityName; O=CompanyName; OU=CompanySectionName; CN=CommonNameOrHostname
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Connected to 172.17.0.2 (172.17.0.2) port 443
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://172.17.0.2:443/
* [HTTP/3] [0] [:method: GET]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: 172.17.0.2]
* [HTTP/3] [0] [:path: /]
* [HTTP/3] [0] [user-agent: curl/8.10.1-DEV]
* [HTTP/3] [0] [accept: */*]
> GET / HTTP/3
> Host: 172.17.0.2
> User-Agent: curl/8.10.1-DEV
> Accept: */*
>
* Request completely sent off
< HTTP/3 200
< server: nginx/1.27.1
< date: Mon, 30 Sep 2024 17:12:34 GMT
< content-type: text/plain
< content-length: 10
< alt-svc: h3=":443"; ma=86400
<
http3: h3
* Connection #0 to host 172.17.0.2 left intact

As you can see, nginx and curl talk http3 here.

[the-hotmann]

@thresheek thank you for your swift response. I ofc will test again and report back.

[the-hotmann]

This is my config:

server {

        # Dont show nginx version
        server_tokens           off                                                     ;

        # Listener for HTTP2 & HTTP3
        listen                  443 quic reuseport                                      ;
        listen                  443 ssl                                                 ;

        # HTTP2 & HTTP3 STUFF
        http2                   on                                                      ;
        http3                   on                                                      ;
        http3_hq                on                                                      ;
        quic_retry              on                                                      ;
        #quic_gso               on                                                      ;
        #ssl_early_data         on                                                      ;

        # Server & SSL Settings
        server_name             ${PUBLIC_HOST}                                          ;
        ssl_certificate         /etc/ssl/own/wildcard_domain.cert                       ;
        ssl_certificate_key     /etc/ssl/own/wildcard_domain.key                        ;
        ssl_protocols           TLSv1.2 TLSv1.3                                         ;
        ssl_ciphers             HIGH:!aNULL:!eNULL:!3DES:!ADH:!CAMELLIA:!DSS:!ECDSA:!EXP:!IDEA:!MD5:!PSK:!RC4:!RSA:!SEED:!SHA1;
        ssl_session_timeout     10m                                                     ;
        ssl_session_cache       shared:MozSSL:10m                                       ;
        ssl_dhparam             /etc/ssl/own/dhparam.pem                                ;
        keepalive_timeout       70                                                      ;

        # HEADERS
        proxy_hide_header       Strict-Transport-Security                               ;
        add_header              Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
        # HTTP3 Headers
        add_header              QUIC-Status $http3                                      ;
        add_header              x-quic 'h3'                                             ;
        add_header              alt-svc 'h3=":$server_port"; ma=86400'                  ;

        # PROXY BUFFERS
        proxy_busy_buffers_size         512k    ;
        proxy_buffers                   4 512k  ;
        proxy_buffer_size               256k    ;

        fastcgi_buffer_size             128k    ;
        fastcgi_buffers                 4 256k  ;
        fastcgi_busy_buffers_size       256k    ;

        # HTML BASICS
        root /var/www/html;
        index index.html;

        location / { return 200 'http3: $http3\n'; }


}

But when I run:

docker run -it --rm alpine/curl-http3:latest curl -v -k --http3 https://domain.tld:443/

I get the following:

* Host domain.tld:443 was resolved.
* IPv6: (none)
* IPv4: ###IPv4###
*   Trying ###IPv4###:443...
*   Trying ###IPv4###:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=DE; ST=###CENSORED###; L=###CENSORED###; O=###CENSORED###; CN=*.###CENSORED###
*  start date: Jan 19 09:56:52 2024 GMT
*  expire date: Jan 23 23:59:59 2025 GMT
*  issuer: C=DE; O=Deutsche Telekom Security GmbH; CN=Telekom Security ServerID OV Class 2 CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Connected to domain.tld (###IPv4###) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://domain.tld:443/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: domain.tld]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.10.1-DEV]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: domain.tld
> User-Agent: curl/8.10.1-DEV
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
< server: nginx
< date: Tue, 01 Oct 2024 14:01:50 GMT
< content-type: application/octet-stream
< content-length: 8
< strict-transport-security: max-age=31536000; includeSubDomains
< x-quic: h3
< alt-svc: h3=":443"; ma=86400
<
http3:
* Connection #0 to host domain.tld left intact

Notice this:

* ALPN: curl offers h2,http/1.1

why does curl (the very same container you ran not offer h3?

Thanks for looking into this! :)

P.S.:

• I run the alpine-slim Images (does this change anything?)
• I only get HTTP2
• my config is generated from /etc/nginx/templates and envsubst-ed to /etc/nginx/conf.d. I did not edit the /etc/nginx/nginx.conf directly, but used the nginx-container as advertised on docker-hub.

[thresheek]

You probably have some other configuration you didnt mention in this snippet. This works fine for me (I've only changed the SSL certificate locations):

user nginx;
worker_processes 1;
error_log /dev/stderr info;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    access_log /dev/stdout combined;

server {

        # Dont show nginx version
        server_tokens           off                                                     ;

        # Listener for HTTP2 & HTTP3
        listen                  443 quic reuseport                                      ;
        listen                  443 ssl                                                 ;

        # HTTP2 & HTTP3 STUFF
        http2                   on                                                      ;
        http3                   on                                                      ;
        http3_hq                on                                                      ;
        quic_retry              on                                                      ;
        #quic_gso               on                                                      ;
        #ssl_early_data         on                                                      ;

        # Server & SSL Settings
        server_name             test;
        ssl_certificate         /etc/nginx/ssl/cert.pem;
        ssl_certificate_key     /etc/nginx/ssl/key.pem;
        ssl_protocols           TLSv1.2 TLSv1.3                                         ;
        ssl_ciphers             HIGH:!aNULL:!eNULL:!3DES:!ADH:!CAMELLIA:!DSS:!ECDSA:!EXP:!IDEA:!MD5:!PSK:!RC4:!RSA:!SEED:!SHA1;
        ssl_session_timeout     10m                                                     ;
        ssl_session_cache       shared:MozSSL:10m                                       ;
        ssl_dhparam             /etc/nginx/ssl/dhparam.pem                                ;
        keepalive_timeout       70                                                      ;

        # HEADERS
        proxy_hide_header       Strict-Transport-Security                               ;
        add_header              Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
        # HTTP3 Headers
        add_header              QUIC-Status $http3                                      ;
        add_header              x-quic 'h3'                                             ;
        add_header              alt-svc 'h3=":$server_port"; ma=86400'                  ;

        # PROXY BUFFERS
        proxy_busy_buffers_size         512k    ;
        proxy_buffers                   4 512k  ;
        proxy_buffer_size               256k    ;

        fastcgi_buffer_size             128k    ;
        fastcgi_buffers                 4 256k  ;
        fastcgi_busy_buffers_size       256k    ;

        # HTML BASICS
        root /var/www/html;
        index index.html;

        location / { return 200 'http3: $http3\n'; }


}
}

$ docker run -ti --rm alpine/curl-http3:latest curl -v -k --http3-only https://172.17.0.3:443/
*   Trying 172.17.0.3:443...
* Server certificate:
*  subject: C=XX; ST=StateName; L=CityName; O=CompanyName; OU=CompanySectionName; CN=CommonNameOrHostname
*  start date: Sep 30 16:58:10 2024 GMT
*  expire date: Sep 28 16:58:10 2034 GMT
*  issuer: C=XX; ST=StateName; L=CityName; O=CompanyName; OU=CompanySectionName; CN=CommonNameOrHostname
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Connected to 172.17.0.3 (172.17.0.3) port 443
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://172.17.0.3:443/
* [HTTP/3] [0] [:method: GET]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: 172.17.0.3]
* [HTTP/3] [0] [:path: /]
* [HTTP/3] [0] [user-agent: curl/8.10.1-DEV]
* [HTTP/3] [0] [accept: */*]
> GET / HTTP/3
> Host: 172.17.0.3
> User-Agent: curl/8.10.1-DEV
> Accept: */*
>
* Request completely sent off
< HTTP/3 200
< server: nginx
< date: Tue, 01 Oct 2024 16:43:53 GMT
< content-type: text/plain
< content-length: 10
< strict-transport-security: max-age=31536000; includeSubDomains
< quic-status: h3
< x-quic: h3
< alt-svc: h3=":443"; ma=86400
<
http3: h3
* Connection #0 to host 172.17.0.3 left intact

[the-hotmann]

Yes, there is another vhost, but this should not affect this one. Which nginx image exactly are you using?

[thresheek]

It's nginx:1.27.1-alpine.

[the-hotmann]

Ok, I use nginx:1-alpine-slim. But I guess this was not the problem.

I found the issue. If I use network_mode: host it works. If I use ports (network_mode: bridge), it does not:

Works:

services:

  nginx:
    image: nginx:1-alpine-slim
    container_name: nginx
    network_mode: host
    volumes:
      - "./nginx/templates/:/etc/nginx/templates/:ro"
      - "./nginx/ssl/:/etc/ssl/own/:ro"
    healthcheck:
      test: ["CMD-SHELL", "nc -vz -w1 $(hostname) 443"]
      interval: 1s
      timeout: 1s
      retries: 30
    deploy:
      resources:
        limits:
          memory: 500M
    restart: unless-stopped

does not work:

services:

  nginx:
    image: nginx:1-alpine-slim
    container_name: nginx
    hostname: nginx
    ports:
      - "443:443/tcp"
      - "443:443/udp"
    volumes:
      - "./nginx/templates/:/etc/nginx/templates/:ro"
      - "./nginx/ssl/:/etc/ssl/own/:ro"
    healthcheck:
      test: ["CMD-SHELL", "nc -vz -w1 $(hostname) 443"]
      interval: 1s
      timeout: 1s
      retries: 30
    deploy:
      resources:
        limits:
          memory: 500M
    restart: unless-stopped

Could you please try it again with ports and see if it still works for you?

Thanks! If network_mode: bridge breaks HTTP3, then this at least should be added to the docs, or it should be recommended to use network_mode: host - which I really dislike, since especially nginx can so nicely be completely isolated.

Thanks in advance!

[thresheek]

I am using a bridge network, since I don't specify anything in particular to change this mode.

This is confirmed by e.g. a docker inspect in #935 (comment), since I'm looking for a network named bridge.

[the-hotmann]

I probably should add, that I am using nftables (previously iptables), but also there I have enabled udp port 443 + the default docker rules, that automatically apply, if you have nftables/iptables installed.

[the-hotmann]

This issue seems to be related to: moby/moby#15127

What partially fixed the issue (now I can reproduce your case):

services:

  nginx:
    image: nginx:1-alpine-slim
    container_name: nginx
    hostname: nginx
    ports:
      - "443:443/tcp"
      - "123.123.123.123:443:443/udp"
    volumes:
      - "./nginx/templates/:/etc/nginx/templates/:ro"
      - "./nginx/ssl/:/etc/ssl/own/:ro"
    healthcheck:
      test: ["CMD-SHELL", "nc -vz -w1 $(hostname) 443"]
      interval: 1s
      timeout: 1s
      retries: 30
    deploy:
      resources:
        limits:
          memory: 500M
    restart: unless-stopped

In the issue was described, that opening udp solely will not work, but you need to bind it to your public ip (or local IP, if you want to access it locally).

So replace:

      - "443:443/udp"

with

      - "123.123.123.123:443:443/udp"

(use your hosts IP instead of 123.123.123.123)

When I not use docker curl-http2 container on my host I get HTTP3, if I use it remotely it does not work and falls back to HTTP2.

@thresheek I wonder how it could work for you, if you never tagged it on the hosts IP..
Also: if you use this on a public server, does it still work, when connecting from another public server?
Because from another public server I never was able to establish HTTP3.

[thresheek]

I've actually tried both - inside the bridged network from another container as posted here in this issue, and publishing the ports (not ip:ports), and testing the access from the other machine...

For that matter, I'm testing on Ubuntu 22.04 aarch64, with:

 Client: Docker Engine - Community
 Version:    26.1.4
 Server Version: 26.1.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-119-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: aarch64

And the "remote" machine I also tested from runs Alpine Linux.

[the-hotmann]

My machine runs Ubuntu 22 aswell.
Not aarch64, but amd64. I dont know if this makes a difference or not (should not), but I currently guess that docker does not play nicely with iptables/nftables when it comes to udp.

As for now, I would like to keep this issue open and inform here about news.
Thanks for the assistance!