Add support for containerd container runtime

[lhotari]

GKE has switched to use containerd as the default container runtime in GKE node version 1.19.

With the release of GKE node version 1.19, the Container-Optimized OS with Docker (cos) variant is deprecated. Please migrate to the Container-Optimized OS with Containerd (cos_containerd) variant, which is now the default GKE node image. For instructions, see Containerd images.

On Ubuntu, microk8s also uses containerd.

[lhotari]

It would be useful to implement the containerd support in a way that support for other OCI container runtimes such as CRI-O could be added later. CRI-O is used in OpenShift.

[joscabmar]

It is not working in AKS either:

Verifying target pod ... ✔
Launching profiler ... ❌

When inspecting the logs in the kubectl flame pod:

kubectl logs kubectl-flame-cc7b0469-504b-4c90-a5b5-6dd797675d91-6c2lw
{"type":"progress","data":{"time":"2021-06-16T09:08:36.020796231Z","stage":"started"}}
{"type":"error","data":{"reason":"open /var/lib/docker/image/overlay2/layerdb/mounts/containerd://400e37fbed1e7a36429210bb1c9842bf94d2974ededfff703bd48bb78c810a1b/mount-id: no such file or directory"}}

As you can see, it is trying to place the Container ID in the path. But that does not work with containerd. I entered in the overlayfs directory to check the schema and I found this:

$ sudo ls -la /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots
total 2432
drwx------ 606 root root 12288 Jun 16 07:55 .
drwx------   3 root root  4096 May 19 19:39 ..
drwx------   4 root root  4096 May 19 19:39 1
drwx------   4 root root  4096 May 19 19:39 10
drwx------   4 root root  4096 May 19 19:41 100
drwx------   4 root root  4096 May 19 19:41 101
drwx------   4 root root  4096 May 19 19:41 102
drwx------   4 root root  4096 May 19 19:41 103
drwx------   4 root root  4096 May 19 19:41 104
drwx------   4 root root  4096 May 19 19:41 105
drwx------   4 root root  4096 May 19 19:41 106
...

The Kubernetes version is 1.20.7:

kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-21T20:23:45Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"6b3f9b283463c1d5a2455df301182805e65c7145", GitTreeState:"clean", BuildDate:"2021-05-19T22:28:47Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

[lhotari]

Since kubectl-flame doesn't support containerd, I'd like to introduce a workaround for using async-profiler. kubectl-flame's java profiling support uses async-profiler. I have scripts in https://github.com/lhotari/k8s-diagnostics-toolbox for creating flamegraphs in k8s environment for Java containers. Just in case someone is looking for workarounds for the lack of containerd support in kubectl-flame.

[joscabmar]

Since kubectl-flame doesn't support containerd, I'd like to introduce a workaround for using async-profiler. kubectl-flame's java profiling support uses async-profiler. I have scripts in https://github.com/lhotari/k8s-diagnostics-toolbox for creating flamegraphs in k8s environment for Java containers. Just in case someone is looking for workarounds the lack of containerd support in kubectl-flame.

Thanks a lot for your help. I will have a look at them!

[postol]

Hi, is there any plan to resolve this issue as containerd runtime is a new standard now?

[vojtechmasa]

I am getting a similar error for CRI-O. Is it better to file a separate ticket or track it within this one?

{"type":"error","data":{"reason":"open /var/lib/docker/image/overlay2/layerdb/mounts/cri-o://9dd32a9df46b05fabc788fa1a07530fa918b1004f131e7f04f33f16a2c9737f4/mount-id: no such file or directory"}}

[u6f6o]

same here:

{"type":"error","data":{"reason":"open /var/lib/docker/image/overlay2/layerdb/mounts/containerd://31adf43e1c203be147fd94392217b655f43b507de8e4497cc75e1e6ecf0561fd/mount-id: no such file or directory"}}```

[ahmadbabaeimoghadam]

Same here:

{"type":"progress","data":{"time":"2023-10-09T11:37:13.823655136Z","stage":"started"}}
{"type":"error","data":{"reason":"open /var/lib/docker/image/overlay2/layerdb/mounts/containerd://268edb097c8c19eae337cd8fedf7fb969946c80c08dac0af370a3efb8a2276b1/mount-id: no such file or directory"}}

[Najiva]

same here