All requests return 400 error when using Linux swag in front of roadrunner-based Shlink #1796
-
How Shlink is set up
SummaryAfter upgrading to latest version, everything stopped working. When I tried to access any URL, I was receiving version: "3"
services:
shlink:
image: shlinkio/shlink:stable
pull_policy: always
container_name: shlink
restart: unless-stopped
tty: true
stdin_open: true
external_links:
- mysql:mysql
environment:
IS_HTTPS_ENABLED: "true"
DEFAULT_DOMAIN: ${SHLINK_DOMAIN}
DB_DRIVER: ${SHLINK_DB_DRIVER}
DB_NAME: ${SHLINK_DB_NAME}
DB_USER: ${SHLINK_DB_USER}
DB_PASSWORD: ${SHLINK_DB_PASSWORD}
DB_HOST: ${SHLINK_DB_HOST}
GEOLITE_LICENSE_KEY: ${SHLINK_GEOLITE_KEY}
DEFAULT_INVALID_SHORT_URL_REDIRECT: ${SHLINK_INVALID_REDIRECT}
DEFAULT_REGULAR_404_REDIRECT: ${SHLINK_404_REDIRECT}
DEFAULT_BASE_URL_REDIRECT: ${SHLINK_URL_REDIRECT}
networks:
- swag-network
networks:
swag-network:
name: swag
I tried a lot of times different ways to fix it, including testing different tags of the docker image. the one that is working for me is Apparently, it seems an issue during the migration to the new docker runtime. Follow the Docker logs after starting the Current behaviorRight now, I can use my instance with Expected behaviorI expect that it works with the new runner :D How to reproduceI'm not sure. I'm using the same instance for a long long time (about 1.5 years). In the past I had to update my env vars during an upgrade, but since then I never touched anything. Watchtower always could handle the updates for me. If I return to any tag that uses Roadrunner runtime, the issue starts happening again. Obs: I never tried to rollback to a previous version. |
Beta Was this translation helpful? Give feedback.
Replies: 13 comments 8 replies
-
It would be good to know what is the response body for those 400 errors, because according to your logs, the server started successfully, but it is not registering any incoming request, which makes me think there's something in front of Shlink returning those errors. |
Beta Was this translation helpful? Give feedback.
-
|
Ohh that's exactly the response body :)
And I'm pretty sure that this response came from the shlink service, because if I turn it off, I receive the default NGINX 502 error (I'm using the Linuxserver's SWAG docker with NGINX as the reverse proxy for this service) One (maybe important) information that I forgot to mention:
|
Beta Was this translation helpful? Give feedback.
-
|
Can you also share the request and response headers? Nginx can return 400s when configured as reverse proxy and sending too big headers. Also, please check nginx logs, as the reason for the error will be most likely there. Perhaps openswoole accepts a bigger request by defaault and roadrunner is rejecting it. |
Beta Was this translation helpful? Give feedback.
-
|
Of course. I did a cleanup in the nginx logs. After trying to access a URL, this was the content of the The Regarding the request and response headers, these are all headers:
And these are the headers for a working request (using docker tag
|
Beta Was this translation helpful? Give feedback.
-
|
Can you try to put together a reproducible scenario via docker compose, with nginx as reverse proxy and Shlink 3.6 with roadrunner? Also, sharing your nginx config would be helpful. |
Beta Was this translation helpful? Give feedback.
-
|
I will create a new environment to try to reproduce the issue at the end of the day :) while, follow my nginx config: |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I could reproduce the issue in a new (fresh) environment. VM Configuration:
Obs: the VM that I had the initial issue is running Alpine Linux with a better hardware configuration. Docker-Compose YAML: version: "3"
services:
shlink:
image: shlinkio/shlink:stable
pull_policy: always
container_name: shlink
restart: unless-stopped
tty: true
stdin_open: true
depends_on:
- mysql
external_links:
- mysql:mysql
environment:
IS_HTTPS_ENABLED: "true"
DEFAULT_DOMAIN: ${SHLINK_DOMAIN}
DB_DRIVER: ${SHLINK_DB_DRIVER}
DB_NAME: ${SHLINK_DB_NAME}
DB_USER: ${SHLINK_DB_USER}
DB_PASSWORD: ${SHLINK_DB_PASSWORD}
DB_HOST: ${SHLINK_DB_HOST}
GEOLITE_LICENSE_KEY: ${SHLINK_GEOLITE_KEY}
DEFAULT_INVALID_SHORT_URL_REDIRECT: ${SHLINK_INVALID_REDIRECT}
DEFAULT_REGULAR_404_REDIRECT: ${SHLINK_404_REDIRECT}
DEFAULT_BASE_URL_REDIRECT: ${SHLINK_URL_REDIRECT}
networks:
- swag-network
swag:
image: "ghcr.io/linuxserver/swag:latest"
pull_policy: always
container_name: swag
restart: unless-stopped
tty: true
stdin_open: true
ports:
- "80:80"
- "443:443"
volumes:
- ${SWAG_PATH}:/config
environment:
PUID: 1000
PGID: 1000
TZ: "America/Sao_Paulo"
URL: ${SWAG_DOMAIN}
EXTRA_DOMAINS: ${SWAG_EXTRA_DOMAINS}
VALIDATION: "dns"
DNSPLUGIN: "cloudflare"
EMAIL: ${SWAG_EMAIL}
STAGING: "false"
cap_add:
- NET_ADMIN
networks:
- swag-network
mysql:
image: mysql:latest
pull_policy: always
container_name: mysql
restart: unless-stopped
tty: true
stdin_open: true
environment:
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_ROOT_PASSWORD: ${MYSQL_PASSWORD}
volumes:
- ${MYSQL_PATH}:/var/lib/mysql
networks:
- swag-network
networks:
swag-network:
name: swag.env file with the environment variables: NGINX configuration file ( Other configurations I did:
To test, I tried to open Obs: it's important to mention that I used my real domain during this test. |
Beta Was this translation helpful? Give feedback.
-
|
I think the issue is somewhere in I put together a simple example using plain nginx as reverse proxy, over HTTP, and everything works just fine. version: "3"
services:
shlink:
image: shlinkio/shlink:3.6.0
container_name: shlink
restart: unless-stopped
tty: true
stdin_open: true
depends_on:
- mysql
links:
- mysql
environment:
IS_HTTPS_ENABLED: "true"
DEFAULT_DOMAIN: ${SHLINK_DOMAIN}
DB_DRIVER: ${SHLINK_DB_DRIVER}
DB_NAME: ${SHLINK_DB_NAME}
DB_USER: ${SHLINK_DB_USER}
DB_PASSWORD: ${SHLINK_DB_PASSWORD}
DB_HOST: ${SHLINK_DB_HOST}
GEOLITE_LICENSE_KEY: ${SHLINK_GEOLITE_KEY}
DEFAULT_INVALID_SHORT_URL_REDIRECT: ${SHLINK_INVALID_REDIRECT}
DEFAULT_REGULAR_404_REDIRECT: ${SHLINK_404_REDIRECT}
DEFAULT_BASE_URL_REDIRECT: ${SHLINK_URL_REDIRECT}
swag:
image: "nginx:1.25"
container_name: swag
restart: unless-stopped
tty: true
stdin_open: true
ports:
- "8888:80"
volumes:
- ./nginx.conf:/etc/nginx/conf.d/default.conf
cap_add:
- NET_ADMIN
links:
- shlink
mysql:
image: mysql:latest
container_name: mysql
restart: unless-stopped
tty: true
stdin_open: true
environment:
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_ROOT_PASSWORD: ${MYSQL_PASSWORD}What I find very weird is that you are not getting any error logged in your nginx when it is returning a 400 response, because nginx always logs errors with INFO level. Make sure there's nothing else preventing nginx logs, as that's the only way to find the root cause. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I created an issue in the |
Beta Was this translation helpful? Give feedback.
-
|
I generally have the same issue, but no real time right now to mess with this. Running shlinksrv:3.5.2 on Docker fine, all works ok. NGINX access.log: Nothing of use in error.log Messed with a bunch of NGINX changes, but nothing really lead me anywhere. My NGINX config: I'll see another day. I sense the new version has some sort of new web server and nginx needs to adjust somehow to this. |
Beta Was this translation helpful? Give feedback.
-
|
The only idea that comes to my mind is trying with docker tag 3.5.4-roadrunner That version has roadrunner native logs enabled, while v3.6 has them disabled because it introduces a way to log requests consistently between openswoole and roadrunner. If for any reason, roadrunner is rejecting the request from nginx, causing this 400, maybe we see something different in those logs, that is not present in 3.6, because the request is rejected before triggering any Shlink code. |
Beta Was this translation helpful? Give feedback.
-
|
That's great.. thanks I'll try tag 3.5.4-roadrunner I didn't have the time, but I wanted to see what I could gat from the console of my swag container via some curl statements. Maybe the shlink server would respond directly.. I think the server responds to API requests.. just didn't have those details ready. The only other thing in the mix is a .conf file in swag/nginx that needs an obscure update. |
Beta Was this translation helpful? Give feedback.
-
|
@acelaya I'm trying right now with 3.5.4-roadrunner to see if I can find anything, but it's not logging any error. Here we have the end of the docker logs for this image: If I try to access any URL, it give me a |
Beta Was this translation helpful? Give feedback.
-
|
Hi @acelaya I tried to comment out line by line of the |
Beta Was this translation helpful? Give feedback.
-
|
I also set that when trying with vanila nginx, and it works #1796 (comment) In fact, that header is important for multi-domain to work properly, so it's needed. My only guess is that something invalid is being set in |
Beta Was this translation helpful? Give feedback.
-
|
Hmm that's interesting :D I noticed that I was already setting this Host header in my nginx config file (#1796 (comment)) I removed from my config file and readded in the |
Beta Was this translation helpful? Give feedback.
-
|
Okay, I got it :) When I followed the documentation to put shlink behind a reverse proxy, I copied everything to my swag-nginx file. But swag set all headers for me. When I copied the doc, I duplicated a lot of headers. I don't know what causes the issue, because before it was working :D but after removing all |
Beta Was this translation helpful? Give feedback.
Okay, I got it :) When I followed the documentation to put shlink behind a reverse proxy, I copied everything to my swag-nginx file. But swag set all headers for me. When I copied the doc, I duplicated a lot of headers. I don't know what causes the issue, because before it was working :D but after removing all
proxy_set_headerfrom my config file, everything worked.