Skip to content
This repository has been archived by the owner on Oct 22, 2024. It is now read-only.

Commit

Permalink
pmem-csi: avoid false negative liveness probe
Browse files Browse the repository at this point in the history 
During a long-running DeleteVolume for a 200GB volume the LVM lock is held and
the liveness probe for metrics data timed out after a minute, killing the
container, because metrics retrieval must take the LVM lock.

As operation duration is non-deterministic, the solution is to probe only
metrics data which can be served immediately when the process is up and
running. This is less precise (= doesn't really check any functionality), but
that's an inherent problem of liveness probes: they simply cannot cover the
full functionality.

(cherry picked from commit 3e92fe3)
  • Loading branch information
@pohly
pohly committed Nov 10, 2021
1 parent 0cc7c13 commit 16dc613
Show file tree
Hide file tree
Showing 29 changed files with 166 additions and 130 deletions.
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.19/direct/pmem-csi.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -401,7 +401,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -480,7 +480,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -500,7 +500,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.19/direct/testing/pmem-csi.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -402,7 +402,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -482,7 +482,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -502,7 +502,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.19/lvm/pmem-csi.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -401,7 +401,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -480,7 +480,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -500,7 +500,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.19/lvm/testing/pmem-csi.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -402,7 +402,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -482,7 +482,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -502,7 +502,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.19/pmem-csi-direct-testing.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -402,7 +402,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -482,7 +482,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -502,7 +502,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.19/pmem-csi-direct.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -401,7 +401,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -480,7 +480,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -500,7 +500,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.19/pmem-csi-lvm-testing.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -402,7 +402,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -482,7 +482,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -502,7 +502,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.19/pmem-csi-lvm.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -401,7 +401,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -480,7 +480,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -500,7 +500,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.20/direct/pmem-csi.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -401,7 +401,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -480,7 +480,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -500,7 +500,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.20/direct/testing/pmem-csi.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -402,7 +402,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -482,7 +482,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -502,7 +502,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
8 changes: 4 additions & 4 deletions deploy/kubernetes-1.20/lvm/pmem-csi.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -401,7 +401,7 @@ spec:
startupProbe:
failureThreshold: 60
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down Expand Up @@ -480,7 +480,7 @@ spec:
livenessProbe:
failureThreshold: 6
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 10
Expand All @@ -500,7 +500,7 @@ spec:
startupProbe:
failureThreshold: 300
httpGet:
path: /metrics
path: /metrics/simple
port: metrics
scheme: HTTP
periodSeconds: 1
Expand Down
Loading

0 comments on commit 16dc613

Please sign in to comment.