Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: disable timeout for liveness probe in k8s and keep only k8s timeout #5594

Merged
merged 9 commits into from
Jan 16, 2023

Conversation

alaeddine-13
Copy link
Contributor

As an executor is increasingly stressed by load, the health check latency is affected.
It may eventually exceed the default 1 second timeout and therefore, an Executor might be killed by k8s.
This PR disables the ping timeout in liveness probes, keeping just the k8s timeout

@github-actions github-actions bot added size/XS area/core This issue/PR affects the core codebase component/resource labels Jan 12, 2023
@codecov
Copy link

codecov bot commented Jan 12, 2023

Codecov Report

Merging #5594 (c96f5a2) into master (89e9286) will increase coverage by 1.29%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5594      +/-   ##
==========================================
+ Coverage   86.68%   87.98%   +1.29%     
==========================================
  Files         124      124              
  Lines        9939    10048     +109     
==========================================
+ Hits         8616     8841     +225     
+ Misses       1323     1207     -116     
Flag Coverage Δ
jina 87.98% <ø> (+1.29%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
jina/serve/runtimes/head/request_handling.py 90.78% <0.00%> (-3.33%) ⬇️
jina/serve/streamer.py 93.33% <0.00%> (-3.28%) ⬇️
jina/serve/runtimes/asyncio.py 85.82% <0.00%> (-0.90%) ⬇️
jina/serve/runtimes/worker/__init__.py 95.74% <0.00%> (+0.03%) ⬆️
jina/serve/runtimes/gateway/__init__.py 98.36% <0.00%> (+0.08%) ⬆️
jina/serve/runtimes/head/__init__.py 97.85% <0.00%> (+0.11%) ⬆️
jina/checker.py 85.71% <0.00%> (+0.34%) ⬆️
jina/jaml/__init__.py 95.00% <0.00%> (+0.35%) ⬆️
jina/orchestrate/deployments/config/k8s.py 100.00% <0.00%> (+0.61%) ⬆️
jina/orchestrate/deployments/__init__.py 89.77% <0.00%> (+0.62%) ⬆️
... and 13 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Member

@JoanFM JoanFM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can u add a test that can simulate what we were observing? putting time sleeps and overcharging should be good

@alaeddine-13 alaeddine-13 reopened this Jan 16, 2023
@github-actions github-actions bot added size/S area/cli This issue/PR affects the command line interface labels Jan 16, 2023
@JoanFM JoanFM merged commit 9b50e92 into master Jan 16, 2023
@JoanFM JoanFM deleted the debug-raise-health-check branch January 16, 2023 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cli This issue/PR affects the command line interface area/core This issue/PR affects the core codebase component/resource size/S size/XS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants