Dozzle 8.6.1 : Container is consired as stopped by Dozzle, while it is correctly running #3337

omasseau · 2024-10-21T11:02:45Z

🔍 Check for existing issues

Completed

How is Dozzle deployed?

Standalone Deployment

📦 Dozzle version

8.6.1

✅ Command used to run Dozzle

  # Dozzle logs
  docker-logs:
    profiles: [all, logs]
    image: amir20/dozzle:v8.6.1
    restart: unless-stopped
    deploy:
      resources:
        limits:
          # strict minimum required: 10m
          memory: 25m
    networks:
      gateway_internal_network:
        # if an ip address is not explicitly defined, docker will attribute to the container an available ip from the defined subnet
        ipv4_address: ${DOZZLE__DOCKER_SUBNET_IP:-}
    ports:
      - '${DOZZLE__HTTP_PORT:-9999}:8080'
    logging:
      # logging options
      driver: "json-file"
      options:
        max-file: "25"
        max-size: "10m"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DOZZLE_BASE=/logs
      - DOZZLE_LEVEL=debug
      - DOZZLE_NO_ANALYTICS=true

🐛 Describe the bug / provide steps to reproduce it

When displaying Dozzle page, one of my container is shown as stopped, while it is running.
I never had this problem with version 7 of Dozzle.

Here you can see the running containers in docker-desktop :

You can clearly see that the container 'bdoc-gateway-proxy-1' is running.

Now if I go in Dozzle, you can see it is not listed :

If I ask Dozzle to show stopped containers, then the container appears in the list :

Strangely it is shown in white and not grey like should be a stopped container (like the 'esignature-1' container which is correctly detected as stopped).

For information, the container that is consirered as stopped seems random. It is not always the same.

Note that docker ps command correctly shows the container :

So does docker stats :

💻 Environment

C:\WINDOWS\system32>docker system info
Client:
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.0-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.3-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  0.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.10
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.2.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 6
  Running: 5
  Paused: 0
  Stopped: 1
 Images: 28
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.48GiB
 Name: docker-desktop
 ID: 50871aac-08fe-4c83-8128-15f5cb1b647f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

📸 If applicable, add screenshots to help explain your bug

No response

📜 If applicable, attach your Dozzle logs. You many need to enable debug mode. See https://dozzle.dev/guide/debugging.

2024-10-21T10:46:31.104094372Z {"level":"info","time":"2024-10-21T10:46:31Z","message":"Dozzle version v8.6.1"}
2024-10-21T10:46:31.104155318Z {"level":"debug","filterArgs":{},"time":"2024-10-21T10:46:31Z","message":"Creating local client"}
2024-10-21T10:46:31.199182737Z {"level":"debug","filter":{},"host":"docker-desktop","time":"2024-10-21T10:46:31Z","message":"Listing containers"}
2024-10-21T10:46:31.232269302Z {"level":"debug","time":"2024-10-21T10:46:31Z","message":"Adding local Docker Engine"}
2024-10-21T10:46:31.236471591Z {"level":"info","clients":1,"time":"2024-10-21T10:46:31Z","message":"Connected to Docker"}
2024-10-21T10:46:31.236485065Z {"level":"debug","filter":{},"host":"docker-desktop","time":"2024-10-21T10:46:31Z","message":"Listing containers"}
2024-10-21T10:46:31.236487532Z {"level":"debug","host":"docker-desktop","time":"2024-10-21T10:46:31Z","message":"docker store subscribing docker events"}
2024-10-21T10:46:31.236489547Z {"level":"info","time":"2024-10-21T10:46:31Z","message":"Accepting connections on :8080"}
2024-10-21T10:46:31.264279754Z {"level":"debug","id":"8943fd1b5094","time":"2024-10-21T10:46:31Z","message":"Finding container"}
2024-10-21T10:46:31.313580180Z {"level":"debug","containers":6,"time":"2024-10-21T10:46:31Z","message":"finished initializing container store"}
2024-10-21T10:46:31.313606512Z {"level":"debug","id":"bb2e6ad23d9e","time":"2024-10-21T10:46:31Z","message":"Finding container"}
2024-10-21T10:46:31.346499764Z {"level":"debug","filter":{},"host":"docker-desktop","time":"2024-10-21T10:46:31Z","message":"Listing containers"}
2024-10-21T10:46:31.389474686Z {"level":"debug","id":"bb2e6ad23d9e","time":"2024-10-21T10:46:31Z","message":"container started"}
2024-10-21T10:46:31.810248553Z {"level":"debug","id":"83116018d688","time":"2024-10-21T10:46:31Z","message":"Finding container"}
2024-10-21T10:46:31.833824071Z {"level":"debug","filter":{},"host":"docker-desktop","time":"2024-10-21T10:46:31Z","message":"Listing containers"}
2024-10-21T10:46:31.896171970Z {"level":"debug","id":"83116018d688","time":"2024-10-21T10:46:31Z","message":"container started"}
2024-10-21T10:46:32.040908468Z {"level":"debug","id":"472e08f43f02","time":"2024-10-21T10:46:32Z","message":"Finding container"}
2024-10-21T10:46:32.052941605Z {"level":"debug","filter":{},"host":"docker-desktop","time":"2024-10-21T10:46:32Z","message":"Listing containers"}
2024-10-21T10:46:35.056652381Z {"level":"debug","id":"ced78352ac69","time":"2024-10-21T10:46:35Z","message":"Finding container"}
2024-10-21T10:46:35.079270720Z {"level":"debug","filter":{},"host":"docker-desktop","time":"2024-10-21T10:46:35Z","message":"Listing containers"}
2024-10-21T10:46:35.123283289Z {"level":"debug","id":"ced78352ac69","time":"2024-10-21T10:46:35Z","message":"container started"}
2024-10-21T10:47:02.051214941Z {"level":"debug","filter":{},"host":"docker-desktop","time":"2024-10-21T10:47:02Z","message":"Listing containers"}
2024-10-21T10:47:02.099829503Z {"level":"debug","container":"83116018d688","host":"docker-desktop","time":"2024-10-21T10:47:02Z","message":"starting to stream stats"}
2024-10-21T10:47:02.099864105Z {"level":"debug","container":"bb2e6ad23d9e","host":"docker-desktop","time":"2024-10-21T10:47:02Z","message":"starting to stream stats"}
2024-10-21T10:47:02.100022560Z {"level":"debug","container":"472e08f43f02","host":"docker-desktop","time":"2024-10-21T10:47:02Z","message":"starting to stream stats"}
2024-10-21T10:47:02.100154186Z {"level":"debug","container":"ced78352ac69","host":"docker-desktop","time":"2024-10-21T10:47:02Z","message":"starting to stream stats"}
2024-10-21T10:47:02.100285403Z {"level":"debug","container":"8943fd1b5094","host":"docker-desktop","time":"2024-10-21T10:47:02Z","message":"starting to stream stats"}
2024-10-21T10:47:02.100421958Z {"level":"debug","host":"docker-desktop","time":"2024-10-21T10:47:02Z","message":"starting to listen to docker events"}
2024-10-21T10:47:02.461806850Z {"level":"debug","hit":false,"data":"[]releases.Release","time":"2024-10-21T10:47:02Z","message":"Cache hit"}
2024-10-21T10:48:24.655335788Z {"level":"debug","id":"472e08f43f02","since":"0001-01-01T00:00:00Z","stdType":"all","host":"docker-desktop","time":"2024-10-21T10:48:24Z","message":"Streaming logs for container"}
2024-10-21T10:48:34.302581862Z {"level":"debug","allocated":"3.5 MB","totalAllocated":"9.8 MB","system":"17 MB","routines":42,"time":"2024-10-21T10:48:34Z","message":"runtime mem stats"}
2024-10-21T10:48:52.122852674Z {"level":"debug","id":"472e08f43f02","since":"0001-01-01T00:00:00Z","stdType":"all","host":"docker-desktop","time":"2024-10-21T10:48:52Z","message":"Streaming logs for container"}
2024-10-21T10:49:53.602335614Z {"level":"debug","allocated":"3.3 MB","totalAllocated":"11 MB","system":"17 MB","routines":40,"time":"2024-10-21T10:49:53Z","message":"runtime mem stats"}
2024-10-21T10:50:05.240333827Z {"level":"debug","id":"472e08f43f02","since":"0001-01-01T00:00:00Z","stdType":"all","host":"docker-desktop","time":"2024-10-21T10:50:05Z","message":"Streaming logs for container"}
2024-10-21T10:53:46.267002377Z {"level":"debug","allocated":"3.1 MB","totalAllocated":"14 MB","system":"18 MB","routines":41,"time":"2024-10-21T10:53:46Z","message":"runtime mem stats"}
2024-10-21T11:00:25.792119240Z {"level":"debug","id":"8943fd1b5094","since":"2024-10-21T10:46:31Z","stdType":"all","host":"docker-desktop","time":"2024-10-21T11:00:25Z","message":"Streaming logs for container"}

The text was updated successfully, but these errors were encountered:

omasseau · 2024-10-21T11:14:00Z

PS: I tried to restart docker-desktop does it does not fix the problem.

omasseau · 2024-10-21T12:42:58Z

Note : Updating to latest version of docker-desktop does not change anything.

Here is docker system info output with the latest version:

PS C:\Users\maol> docker system info
Client:
 Version:    27.2.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 6
  Running: 5
  Paused: 0
  Stopped: 1
 Images: 28
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.48GiB
 Name: docker-desktop
 ID: 50871aac-08fe-4c83-8128-15f5cb1b647f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

omasseau · 2024-10-21T13:14:33Z

Here is another example, where another container is considered as stopped :

amir20 · 2024-10-21T16:07:32Z

This sound like a bug from Dozzle. But am not sure what is causing it and how come to no one else has reported it. I also use Dozzle everyday and haven't ran in to this issue.

Strangely it is shown in white and not grey like should be a stopped container (like the 'esignature-1' container which is correctly detected as stopped).

Yes, that is very strange. The li only turns gray if status has changed to exited. So I wonder what Dozzle thinks the status is.

PS: I tried to restart docker-desktop does it does not fix the problem

That's even more strange because if it was a bug then Dozzle should at least be reset.

What I think is happening is that at some point of the containers goes to a state and Dozzle doesn't know how to handle it. Dozzle has it's own memory database for performance and I am wondering if it is becoming out of sync some how. But restart Dozzle would presumably fix that.

Your logs look fine.

I think the best idea I can think of is creating a random API like /api/store/dump that would show the internal state of Dozzle.

Before doing that, can you think of a way I can reproduce this? Doesn't seem so since I haven't really seen it happen.

amir20 · 2024-10-21T16:27:38Z

@omasseau Try amir20/dozzle:pr-3338. Next time this happens go to /api/debug/store. This only works if DEBUG is enabled.

Send me that and tell me which container is not working. If the data looks right, then it is most likely a UI bug.

omasseau · 2024-10-21T17:40:32Z

@amir20 Thanks, I will do that asap ;)

omasseau · 2024-10-21T17:52:06Z

Here it is :

The problematic container is :
"id": "6638ca65c359",
"name": "bdu-bdoc-gateway-keycloak-1",

The result of /api/debug/store :
store.json

amir20 · 2024-10-21T18:01:43Z

"state": "created", Hmmmmm 🤔

I am guessing docker ps --format "{{.ID}}: {{ .Names }} {{.State}}" for you has also created? Can you confirm?

omasseau · 2024-10-21T18:03:05Z

Another example :

The problematic container is :
"id": "55fc5eed688e",
"name": "bdu-bdoc-gateway-proxy-1"

The result of /api/debug/store :
store.json

docker ps --format "{{.ID}}: {{ .Names }} {{.State}}" correctly show it as running

amir20 · 2024-10-21T18:10:45Z

Hm ok. So docker ps right.

How are you getting in this state by the way? Are you tearing down containers and recreating them? I am looking at the code.

omasseau · 2024-10-21T18:19:26Z

Hm ok. So docker ps right.

How are you getting in this state by the way? Are you tearing down containers and recreating them? I am looking at the code.

I'm just doing a simple :
docker compose -f docker-compose.yml up
after making sure any previous containers are stopped and deleted in docker-desktop

The docker-compose.yml contains services which are activated through profiles.
The profiles are set before launching docker compose by creating the COMPOSE_PROFILES environnement variable.
Exemple :
COMPOSE_PROFILES=keycloak,logs,proxy

And in my docker-compose.yml I have associated profiles declared like that :

  docker-logs:
    profiles: [all, logs]
    image: amir20/dozzle:pr-3338

I'm gonna try by not providing profiles just to see if it changes anything.

omasseau · 2024-10-21T18:28:25Z

I can confirm it is not related to using docker compose with profiles.

I have the same problem when starting the containers with a simple
docker compose -f docker-compose.yml up

omasseau · 2024-10-21T18:32:32Z

FYI, I just tried with amir20/dozzle:v7.0.7 and it works fine (all other things being equal).
So its seems at least related to a change between version 7 and 8 of Dozzle.

amir20 · 2024-10-21T18:38:37Z

I also have a container store in v7. So something else must have changed. I am trying to figure out how a container can get into a "created" state. Because that seems to be the culprit.

amir20 · 2024-10-21T18:47:24Z

Can you do docker system events next time you do docker compose -f docker-compose.yml up?

I don't even see a created event. I see create and then follow up with start. So something else is happening that I can't figure out. I also don't have Windows which might be the cause.

omasseau · 2024-10-21T18:57:24Z

Here it is :

docker system event output :
docker-system-events.log

amir20 · 2024-10-21T19:01:55Z

You clearly have 2024-10-21T20:54:38.678963125+02:00 container start com.docker.compose.service=bdoc-gateway-proxy

I don't understand. I am going to add more debugging. Hold on.

amir20 · 2024-10-21T19:06:06Z

Wait, I think I know what it might be.

amir20 · 2024-10-21T19:12:05Z

OK, I think you have Dozzle as part of a larger compose file. That would make sense because, on startup, I only check running containers. However, if a container is transitioning from created to started, it is not captured.

I tried to fix that in #3338. Can you pull amir20/dozzle:pr-3338 and try again?

omasseau · 2024-10-21T20:07:33Z

@amir20 Indeed Dozzle is one the of the services declared in the compose file.
After some quick tests and a few restarts the problem seems to be fixed with amir20/dozzle:pr-3338 :)

amir20 · 2024-10-21T20:13:51Z

Woohoo. Nice find. I'll merge.

amir20 · 2024-10-21T20:29:27Z

New version released.

It also makes sense why most people didn't report this because I think most people are running Dozzle as a separate stack.

omasseau added the bug Something isn't working label Oct 21, 2024

amir20 mentioned this issue Oct 21, 2024

fix: fixes an edge case where a container might get stuck in created state #3338

Merged

amir20 closed this as completed in #3338 Oct 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dozzle 8.6.1 : Container is consired as stopped by Dozzle, while it is correctly running #3337

Dozzle 8.6.1 : Container is consired as stopped by Dozzle, while it is correctly running #3337

omasseau commented Oct 21, 2024 •

edited

Loading

omasseau commented Oct 21, 2024

omasseau commented Oct 21, 2024

omasseau commented Oct 21, 2024 •

edited

Loading

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024

omasseau commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024 •

edited

Loading

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024 •

edited

Loading

omasseau commented Oct 21, 2024 •

edited

Loading

omasseau commented Oct 21, 2024

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024 •

edited

Loading

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

Dozzle 8.6.1 : Container is consired as stopped by Dozzle, while it is correctly running #3337

Dozzle 8.6.1 : Container is consired as stopped by Dozzle, while it is correctly running #3337

Comments

omasseau commented Oct 21, 2024 • edited Loading

🔍 Check for existing issues

How is Dozzle deployed?

📦 Dozzle version

✅ Command used to run Dozzle

🐛 Describe the bug / provide steps to reproduce it

💻 Environment

📸 If applicable, add screenshots to help explain your bug

📜 If applicable, attach your Dozzle logs. You many need to enable debug mode. See https://dozzle.dev/guide/debugging.

omasseau commented Oct 21, 2024

omasseau commented Oct 21, 2024

omasseau commented Oct 21, 2024 • edited Loading

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024

omasseau commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024 • edited Loading

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024 • edited Loading

omasseau commented Oct 21, 2024 • edited Loading

omasseau commented Oct 21, 2024

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024 • edited Loading

amir20 commented Oct 21, 2024

amir20 commented Oct 21, 2024

omasseau commented Oct 21, 2024 •

edited

Loading

omasseau commented Oct 21, 2024 •

edited

Loading

omasseau commented Oct 21, 2024 •

edited

Loading

omasseau commented Oct 21, 2024 •

edited

Loading

omasseau commented Oct 21, 2024 •

edited

Loading

omasseau commented Oct 21, 2024 •

edited

Loading