Test scenarios for auto detach of pod's volume when the node is down

Workload type: Deployment

Volume Attachment Recovery Policy : Immediate:

• Have a workload deployed using longhorn volume
• change the terminationGracePeriodSeconds: 480 in the yaml of the workload.
• Power down the node where the pod is attached on
• The replacement pod is created after 5 minutes from when the original pod goes into Unknown state. And this replacement pod is in "creating/pending" state
• the volume is able to attach/mount and workload come up in about 30 seconds
• data is accessible after the pod comes up active

Volume Attachment Recovery Policy: wait:

• Have a workload deployed using longhorn volume
• change the terminationGracePeriodSeconds: 480 in the yaml of the workload.
• Power down the node where the pod is attached on
• The replacement pod is created after 5 minutes from when the original pod goes into Unknown state. And this replacement pod is in "creating/pending" state
• the volume is able to attach/mount and workload come up only after about 15 minutes.
• data is accessible after the pod comes up active

Volume Attachment Recovery Policy: never:

• Have a workload deployed using longhorn volume
• change the terminationGracePeriodSeconds: 480 in the yaml of the workload.
• Power down the node where the pod is attached on
• The replacement pod is created after 5 minutes from when the original pod goes into Unknown state. And this replacement pod is in "creating/pending" state
• the volume is NOT able to attach/mount and workload come up even for an hour.
• Delete the pod stuck in "Unknown" status
• Replacement pod comes up in about 1 minute
• data is accessible after the pod comes up active

Workload type: Stateful set

Volume Attachment Recovery Policy: Immediate:

• Have a stateful set workload deployed using volume claim template using longhorn storage class.
• Number of pods - 6
• 2 pods are scheduled on N1.
• Power off N1. The pods go into Unknown state.
• replacement pods are NOT created
• force delete these pods in "Unknown" state
• replacement pods are created
• first pod comes up in about 30 seconds.
• data is accessible in the volume.
• It takes another 1 minute approx for the second pod to come up.
• data is accessible in the volume.

Volume Attachment Recovery Policy: wait:

• Have a stateful set workload deployed using volume claim template using longhorn storage class.
• Number of pods - 6
• 1 pod is scheduled on N1.
• Power off N1. The pod goes into Unknown state.
• replacement pod is NOT created
• force delete this pod in "Unknown" state
• replacement pod is created
• It comes up in about 30 seconds.
• data is accessible in the volume.

Volume Attachment Recovery Policy: never:

• Have a stateful set workload deployed using volume claim template using longhorn storage class.
• Number of pods - 6
• 2 pods are scheduled on N1.
• Power off N1. The pods go into Unknown state.
• replacement pods are NOT created
• force delete these pods in "Unknown" state
• replacement pods are created
• first pod comes up in about 30 seconds.
• data is accessible in the volume.
• It takes another 1 minute approx for the second pod to come up.
• data is accessible in the volume.