prometheus-k8s-0 keep Terminating ？

[trustnote-wang]

k8s version: 1.16.15

kube-prometheus version : release-0.4

This is my first time deploying kube-prometheus，but it not work，pod Prometheus-k8s-0 keeps restarting and terminating after deployment

NAME READY STATUS RESTARTS AGE
alertmanager-main-0 0/2 Terminating 0 4s
alertmanager-main-1 2/2 Terminating 0 14s
grafana-58dc7468d7-26xb2 1/1 Running 0 84m
kube-state-metrics-765c7c7f95-s9dgw 3/3 Running 0 84m
node-exporter-78jzl 2/2 Running 0 84m
node-exporter-9nfd8 0/2 Pending 0 84m
node-exporter-b4bv4 2/2 Running 0 84m
node-exporter-bjf6x 2/2 Running 0 84m
node-exporter-c289g 2/2 Running 0 84m
node-exporter-d6xqh 2/2 Running 0 84m
node-exporter-f5wls 2/2 Running 0 84m
node-exporter-g4qp4 2/2 Running 0 84m
node-exporter-j92zw 2/2 Running 0 84m
node-exporter-jjt78 2/2 Running 0 84m
node-exporter-lpmmg 0/2 Pending 0 84m
node-exporter-nmv9t 2/2 Running 0 84m
node-exporter-vtzrc 2/2 Running 0 84m
node-exporter-wtctm 2/2 Running 0 84m
node-exporter-xcbsl 2/2 Running 0 84m
node-exporter-xkjdj 2/2 Running 0 84m
node-exporter-xsh62 2/2 Running 0 84m
prometheus-adapter-5cd5798d96-jx6qz 1/1 Running 0 84m
prometheus-k8s-0 0/3 ContainerCreating 0 2s
prometheus-k8s-1 0/3 ContainerCreating 0 1s
prometheus-operator-5f75d76f9f-g72s4 1/1 Running 0 87m

error message：

Normal Scheduled default-scheduler Successfully assigned monitoring/prometheus-k8s-0 to qde02.test.com
Normal Pulled 5s kubelet, qde02.test.com Container image "quay.io/prometheus/prometheus:v2.11.0" already present on machine
Warning Failed 5s kubelet, qde02.test.com Error: cannot find volume "config-out" to mount into container "prometheus"
Normal Pulled 5s kubelet, qde02.test.com Container image "rancher/coreos-prometheus-config-reloader:v0.38.1" already present on machine
Warning Failed 5s kubelet, qde02.test.com Error: cannot find volume "config" to mount into container "prometheus-config-reloader"
Normal Pulled 5s kubelet, qde02.test.com Container image "rancher/coreos-configmap-reload:v0.0.1" already present on machine
Warning Failed 5s kubelet, qde02.test.com Error: cannot find volume "prometheus-k8s-rulefiles-0" to mount into container "rules-configmap-reloader"

alertmanager error message:

Normal Pulled 3s kubelet, qde02.test.com Container image "quay.io/prometheus/alertmanager:v0.18.0" already present on machine
Normal Created 3s kubelet, qde02.test.com Created container alertmanager
Warning Failed 3s kubelet, qde02.test.com Error: failed to start container "alertmanager": Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/var/lib/kubelet/pods/a0b7a19d-338f-4140-9923-68dcc15bf1ca/volumes/kubernetes.ioempty-dir/alertmanager-main-db" to rootfs at "/alertmanager" caused: stat /var/lib/kubelet/pods/a0b7a19d-338f-4140-9923-68dcc15bf1ca/volumes/kubernetes.ioempty-dir/alertmanager-main-db: no such file or directory: unknown
Normal Pulled 3s kubelet, qde02.test.com Container image "rancher/coreos-configmap-reload:v0.0.1" already present on machine
Warning Failed 3s kubelet, qde02.test.com Error: cannot find volume "config-volume" to mount into container "config-reloader"

[csyyy106]

Add this to fix it

[trustnote-wang]

Add this to fix it

HI ，I tried it, but it didn't work