Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make ironic backoff on neutron api failures #205

Merged
merged 1 commit into from
Aug 24, 2022

Conversation

msherman64
Copy link
Contributor

When networking-generic-switch is used as the ML2 driver for Neutron,
configuring ports may take a very long time. Increasing the API request
timeout does not resolve this, as neutron's queue may grow without bound.

Instead, we have ironic keep a short timeout, and retry on failures,
with exponential backoff. This allows neutron to apply backpressure to
ironic when the queue grows too long.

See https://docs.openstack.org/ironic/xena/configuration/config.html#neutron.status_code_retries

Two configuration parameters are added:

  • ironic_neutron_status_code_retries
    For requests failing with a "retriable" status code, ironic will retry
    the request this many times.
  • ironic_neutron_status_code_retry_delay
    Specifies the delay between retries, in seconds.
    If not set, retries will use exponential backoff, starting at 0.5s,
    and increasing up to 60s.
    Only set this value if neutron requests often take 60s or more, in
    which case the exponential backoff's maximum will be insufficient to
    allow the queue to drain.

When networking-generic-switch is used as the ML2 driver for Neutron,
configuring ports may take a very long time. Increasing the API request
timeout does not resolve this, as neutron's queue may grow without bound.

Instead, we have ironic keep a short timeout, and retry on failures,
with exponential backoff. This allows neutron to apply backpressure to
ironic when the queue grows too long.

See https://docs.openstack.org/ironic/xena/configuration/config.html#neutron.status_code_retries

Two configuration parameters are added:

* `ironic_neutron_status_code_retries`
  For requests failing with a "retriable" status code, ironic will retry
  the request this many times.
* `ironic_neutron_status_code_retry_delay`
  Specifies the delay between retries, in seconds.
  If not set, retries will use exponential backoff, starting at 0.5s,
  and increasing up to 60s.
  Only set this value if neutron requests often take 60s or more, in
  which case the exponential backoff's maximum will be insufficient to
  allow the queue to drain.
@msherman64 msherman64 merged commit 7772b2e into stable/xena Aug 24, 2022
@msherman64 msherman64 deleted the wip/ironic_backoff branch May 23, 2024 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant