Make ironic backoff on neutron api failures #205
Merged
+12
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When networking-generic-switch is used as the ML2 driver for Neutron,
configuring ports may take a very long time. Increasing the API request
timeout does not resolve this, as neutron's queue may grow without bound.
Instead, we have ironic keep a short timeout, and retry on failures,
with exponential backoff. This allows neutron to apply backpressure to
ironic when the queue grows too long.
See https://docs.openstack.org/ironic/xena/configuration/config.html#neutron.status_code_retries
Two configuration parameters are added:
ironic_neutron_status_code_retries
For requests failing with a "retriable" status code, ironic will retry
the request this many times.
ironic_neutron_status_code_retry_delay
Specifies the delay between retries, in seconds.
If not set, retries will use exponential backoff, starting at 0.5s,
and increasing up to 60s.
Only set this value if neutron requests often take 60s or more, in
which case the exponential backoff's maximum will be insufficient to
allow the queue to drain.