cloud-init hard codes MTU configuration at initial deploy time #3793
Comments
ubuntu-server-builder
added
incomplete
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-12T13:37:40.543454+00:00 Launchpad attachments: cloud-init.tar.gz |
|
Launchpad user Ryan Harper(raharper) wrote on 2020-10-12T16:17:38.198439+00:00 " # curl http://169.254.169.254/openstack/2018-08-27/network_data.json How is cloud-init to know from this network-config.json that DHCP will provide an MTU value? How does it know that it should ignore the provided MTU? If DHCP is providing MTU, should network-config.json then not provide the MTU value? |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-12T18:53:00.349038+00:00 That is an excellent question, I see that the example provided in the Nova documentation [0] provides However, I also see that the reporting of The OpenStack format metadata does provide a separate field that distinguishes between the various types of dynamic and static configuration [2] and I see that cloud-init already makes use of it [3]. So I would suggest that whenever OpenStack eludes to dynamic configuration being in play cloud-init should not write the MTU value into the on-disk configuration but let it be configured by dynamic network configuration protocol. What do you think? 0: https://docs.openstack.org/nova/latest/user/metadata.html#openstack-format-metadata cloud-init/cloudinit/sources/helpers/openstack.py Lines 598 to 609 in 0710450 |
|
Launchpad user Ryan Harper(raharper) wrote on 2020-10-12T19:52:31.962412+00:00
I would argue the opposite. The existing behavior is that the MTU provided If network-data.json MTU value is null, then I think it all of this works the |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-12T21:29:06.069378+00:00 On the flip side the presence of the MTU key in the OpenStack metadata cannot be used as an indicator for intent from either the system or the user that the DHCP server should not be providing the MTU either. Looking at the commit that changed the behaviour in OpenStack the intent of the original code was to always provide the MTU value in the metadata regardless of network type, the fact that it showed as Up until 2017 the default for the OpenStack controlled DHCP server was to always provide an MTU. In 2017 the ability to control this behaviour was removed and from that point onward it always provides an MTU. The user has no way of influencing the contents of the OpenStack network metadata, apart from downgrading to a 5 year old version. I don't see an easy way of overriding cloud-inits default behaviour by adding additional configuration through vendor data either. Perhaps adding a cloud-init config stanza for how the OpenStack source driver should interpret the presence of MTU in the network metadata could be a path to retain compability with anyone relying on the current behaviour and at the same time providing a way forward for everyone else? Meanwhile instances are configured to obtain an address dynamically but stuck with a static value for MTU forever, and not being able to adjust to changes being made to the environment without manual intervention to individual instances. |
|
Launchpad user Ryan Harper(raharper) wrote on 2020-10-12T22:31:53.948163+00:00
Before we continue suggesting that cloud-init should somehow guess If we can guess that an OpenStack which sent an MTU really didn't I'm being pendantic here to make a point. OpenStack is the "Oracle" If the configuration is sub-optimal should not the cloud itself Has there been any attempt to ask in OpenStack upstream why the MTU
This is correct. Network-config cannot be part of user-data or
If the network-data.json provided by the metadata service is not |
|
Launchpad user Richard Harding(rharding) wrote on 2020-10-13T20:15:58.800649+00:00 Frode, can you explain to me the OpenStack operator path here. I'm not familiar with how these adjustments are practically made. You mention "a operator to reduce the MTU available to instances" and "To maximize performance these clouds have configured their instances to use the maximum available MTU without leaving any headroom" but then also say that the MTU controls in OpenStack have been removed? I'd like to understand where the knobs a cloud operator have available to them and then look at how to identify the "source of truth". So far I understand one is coming from the cloud itself, but I'm not sure how, another potential source of truth is a DHCP value provided to the instance. I assume that DHCP knob is in the DHCP server config and not done through a more centralized OpenStack knob that assures common behavior among DHCP and non-DHCP instances? |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-13T21:19:28.565241+00:00 Richard, The "MTU controls in OpenStack have been removed" part pertains to the removal of the operator facing configuration option for the OpenStack DHCP server as to whether or not it should provide information about the network MTU in it's response to clients. Where the removal here means that it is permanently ON, as in you could expect a OpenStack cloud to always provide information about MTU in DHCP packets from the time it was removed. This was introduced in OpenStack Ocata back in 2017 [4]. Prior to this the default value of The act of reducing the MTU on networks is done through the OpenStack Neutron API and/or through migration tools [5]. The effect of reducing the MTU of a network construct in OpenStack result in reducing the MTU for the involved router interfaces as well as the associated DHCP server configuration. There also exist levers to inject configuration into the OpenStack DHCP server to prepare for a migration which is what we use in our recommended migration path [6], you can view the functional test code here [7]. The functional test code does not include a step for reducing MTU and waiting X hours for a DHCP renewal which is why this issue was missed, instead it injects the reduced MTU config prior to launching a first test instance [8]. 4: openstack/neutron@832240a |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-13T22:34:19.473833+00:00 The CI artifact referenced in [6] in the previous comment was removed, the source can be viewed here: https://review.opendev.org/#/c/715132/4/deploy-guide/source/app-ovn.rst |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-14T06:57:19.597570+00:00 Comment #8 made me think about why this works in our functional test, so I went to investigate that. In our functional test we use Bionic images, and sure enough the netplan.yaml [9] written there does NOT include the MTU! The OpenStack Metadata source remains the same [10]. 9: ubuntu@banana-1: This file is generated from information provided by the datasource. Changesto it will not persist across an instance reboot. To disable cloud-init'snetwork configuration capabilities, write a file/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:network: {config: disabled}network: 10: $ curl http://169.254.169.254/openstack/2018-08-27/network_data.json |
|
Launchpad user Richard Harding(rharding) wrote on 2020-10-14T12:12:46.824356+00:00 Hmm, however it should be the same version of cloud-init in either Bionic or Focal tests. Is this a netplan change? That's one side of the coin. In working to understand the "best path forward" I just want to make sure I'm following. The controls in the OpenStack DHCP service are purely a "on/off" switch ( The actual value for the MTU is a Neutron setting and in theory, should be the same then from DHCP network data or by the provided network_data.json information? In this case the pain point is that existing instances won't process the DHCP change of MTU properly because cloud-init has written out the netplan.yaml and even though DHCP comes in with a new setting cloud-init isn't triggered in any fashion to update its understanding of the world and write out a new compatible netplan. The final nail in this coffin is that the setting cloud-init is setting overrides the value for MTU that comes in via DHCP. Do I have that right? If so, a couple of questions then.
|
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-14T22:56:29.074979+00:00
In the bionic image I have: In the focal image I have: I guess it's time for me to ask a question: is it cloud-init that renders /etc/netplan/50-cloud-init.yaml? If so where does netplan fit in when the difference is how that file is rendered and not how it is interpreted. As you can see in #10 the mtu statement is not in the file on bionic, while it is on focal. Since versions appear to be the same my guess would be that there is some internal modelling of how bionic vs. focal should be configured?
Previously you had the
The value for the MTU is a per virtual network setting which is exposed to the end user of the cloud. And yes, the setting set on the virtual network should be the one exposed in the network_data.json. But remember that the operator of the cloud has power to inject options directly into the DHCP server which could mean the DHCP server could advertise a different MTU than the user has chosen for their network. If the operator of the cloud has chosen to do so it is most likely for a very good operational reason. If the end user or operator intends to configure instances with DHCP, DHCP should be authoritative source of truth.
Yes.
This does indeed sound interesting, with regards to a cloud operator possibly not having any access or control over the instances end users run on their cloud having levers to control network configuration in such instances for maintenance/migration purposes in some manner would be valuable. The alternative is forklift and endless nagging of end users to do manual intervention and the support load that comes afterwards when everything breaks because they did not pay attention to the operators requests in time.
I think when the operator and/or end user intends to use network auto configuration (that be DHCP or IPv6 SLAAC) that should be the authoritative source of truth for the instance. Any other path will risk turning a whole estate of instances the cloud operator does not necessarily have access to into door stops whenever the network configuration changes. My conclusion so far is that Bionic guests behaves correctly as detailed in #10, Focal guests behave incorrectly as detailed in the original bug description. |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-15T05:54:41.120814+00:00 I can't let go of why the same version of cloud-init renders different config with the same data source. So a question: are there other data sources in use that we have not yet examined? Information from the hypervisor configuration trickling through the virtio drivers or something like that? I will compare the qemu configuration for the to instances and see if there are any differences. |
|
Launchpad user Ryan Harper(raharper) wrote on 2020-10-15T15:21:50.730790+00:00
There isn't an internal model; cloud-init SRU's master back to The pipeline looks like: cloudinit (fetch network-data.json from OpenStack)
Are you sure the 'mtu' value it was present in the network-data.json Give your JSON from [10], bionic and focal render this the same BIONIC% lxc launch ubuntu-daily:bionic b1 Internal State Read input format 'network_data.json' from 'network-data.json'. 2020-10-15 15:09:17,805 - util.py[DEBUG]: Writing to /root/test/etc/netplan/50-cloud-init.yaml - wb: [644] 503 bytes This file is generated from information provided by the datasource. Changesto it will not persist across an instance reboot. To disable cloud-init'snetwork configuration capabilities, write a file/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:network: {config: disabled}network: FOCAL% lxc launch ubuntu-daily:focal f1 Internal State Read input format 'network_data.json' from 'network-data.json'. 2020-10-15 15:12:57,910 - util.py[DEBUG]: Writing to /root/test/etc/netplan/50-cloud-init.yaml - wb: [644] 503 bytes This file is generated from information provided by the datasource. Changesto it will not persist across an instance reboot. To disable cloud-init'snetwork configuration capabilities, write a file/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:network: {config: disabled}network: |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-15T15:33:25.148769+00:00 Adding some excerpts from cloud-init logs differences between Focal and Bionic, appears to be quite a bit difference in how config is handled. I also went back and compared if there were other differences like image properties or similar stuff that could find their way into qemu config, but found none. If you say cloud-init should be equal on Bionic and Focal, where does this information come from apart from the cloud metadata (which also is equal for both instances)? Focal: grep fa:16:3e:d6:d0:91 /var/log/cloud-init.log2020-10-15 14:48:15,982 - stages.py[DEBUG]: applying net config names for {'version': 1, 'config': [{'type': 'physical', 'mtu': 8942, 'subnets': [{'type': 'dhcp4'}], 'mac_address': 'fa:16:3e:d6:d0:91', 'name': 'ens2'}]} Bionic: |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-15T15:40:18.927217+00:00 Ryan, thanks for those pointers, will check. I also see in #15 that Bionic uses Fallback while Focal uses an actual ds, don't know why though. |
|
Launchpad user Ryan Harper(raharper) wrote on 2020-10-15T15:57:52+00:00 On Thu, Oct 15, 2020 at 10:45 AM Frode Nordahl 1899487@bugs.launchpad.net
Bah, I keep forgetting, that Bionic does NOT read OpenStack metadata That landed right after 18.04 was released. |
|
Launchpad user Launchpad Janitor(janitor) wrote on 2020-12-15T04:17:23.068572+00:00 [Expired for cloud-init because there has been no activity for 60 days.] |
|
Launchpad user Launchpad Janitor(janitor) wrote on 2020-12-15T04:17:24.489964+00:00 [Expired for netplan.io (Ubuntu) because there has been no activity for 60 days.] |
|
Launchpad user Alexander Balderson(asbalderson) wrote on 2021-11-05T20:00:07.919765+00:00 I'm going to re-open this bug after working through Openstack networking migration for OVS to OVN. I ran into an issue where the MTU set in netplan caused all my instances to be lost after the migration. While this can, and should, be documented, I also think that reducing places where instances could be lost should be taken whenever possible. |
|
Launchpad user Brett Holman(holmanb) wrote on 2021-11-09T18:30:52.024484+00:00 Hi Alexander, Thanks for reopening. Sorry to hear about the instance loss. We have a short term fix, but I think we need to request a fix in Openstack for network_data.json as well. Currently cloud-init passes a configured MTU value to the renderers (netplan/systemd-networkd/etc), which in turn treat a configured MTU as overriding DHCP MTU options. There are reasons one might want to do this so I don't think we want to try to change this behavior for all of cloud-init. You can force cloud-init to configure the network on every boot[1], the downside is that this will increase subsequent boot times. To semantically match other datasources, Openstack would only expose MTU settings if it intended for the MTU to override DHCP's MTU option. This would mean that the mtu would only get configured if a link's network type was not dhcp (change here[2] I think). Could you open a ticket with openstack for this? If that would break other use cases, this behavior could possibly be worked around in the cloud-init openstack datasource by ignoring the MTU metadata field, but I would want to see if Openstack can fix this first resorting to that. [1] Override: apply network config on every bootupdates: |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2021-11-10T10:34:52.922632+00:00 Adding upstream OpenStack Nova to the bug to get their perspective on why the OpenStack datasource is exposing MTU when it knows that the network should be configured using DHCP ref Brett's question in #21. |
|
Launchpad user Frode Nordahl(fnordahl) wrote on 2021-12-13T09:07:13.152329+00:00 To add to Bretts comment in #21, the proposed workaround is only effective if you know about this situation beforehand. I.e. if you enabled the override through vendor data before deploying any Focal or newer instances. Another possible workaround is to preform a "cold migration". When Nova stops/starts an instance the domain XML on the hypervisor is re-created. We can take advantage of this behavior, if an instance is stop/started after a network MTU has been lowered the new domain XML will have the new MTU which makes the libvirt driver enforce the MTU in the instance. The instance configuration will still make systemd-networkd attempt to set the hard coded MTU but it will not be allowed to: ubuntu@u: |
|
Launchpad user Artom Lifshitz(notartom) wrote on 2022-08-24T00:22:00.297264+00:00 While Nova indeed exposes the MTU in our metadata, our source of truth for that information is Neutron, via the [As an aside, we've had a long standing issue wherein Neutron allows the MTU to be mutable, but there's no real support for changing the MTU within Nova, necessitating the cold migration work around that Frode has mentioned] As Neutron controls both the DHCP agent and the So since Nova's metadata is just a proxy for Neutron's MTU in this case, I think Brett's question in comment #21 is better asked to the Neutron folks. |
|
Launchpad user Slawek Kaplonski(slaweq) wrote on 2022-09-02T14:02:06.058260+00:00 I read through this bug today and here are my thoughts about it.
I think that nova provides as mtu data from the network_info_cache so new value will be visible in the metadata after some time (few seconds in my tests) after it was changed in Neutron. But that don't solves anything because cloud-init already configured it during boot process and will not check it again. @frode: is my understanding correct? If yes, could the solution be to provide mtu value in the metadata ONLY if all subnets on port don't have dhcp enabled? And provide null otherwise? |
|
Launchpad user Chad Smith(chad.smith) wrote on 2022-09-07T20:06:15.656111+00:00 Thanks @alexander @frode @Artom and @slawek && Brett for weighing in on this long-standing bug and trying to sort long-term maintenance/migration use-cases and how cloud-init could better interact in those changing conditions/settings with neutron.
I think that suggestion would work only for immediate deployment needs at initial deployment time. The other gap or issue is during long-term maintenance what an admin should do when changing MTU in Neutron or migrating OVS -> OVN and needs to trigger a network config refresh from cloud-init.
Correct, cloud-init by default won't perform any network config operations after first boot on OpenStack unless the configuration was added to /etc/cloud/cloud.cfg.d/80-some-file.cfg as Brett mentioned in #21. Per Frode's comment #23: "the proposed workaround is only effective if you know about this situation beforehand." Agreed. But, in the cases where an admin is changing an MTU value through Neutron or migrating from OVS to OVN, I think those use-cases are ones where we now know ahead of time we'll be changing network configuration that may need to be re-written by cloud-init. In these cases where we know our existing VM will soon contain stale MTU data, I think we can prime the cloud-init system in one of two ways: Option 1. Configure cloud-init to regenerate network on every boot and trigger that reboot manually or automatically after MTU settings have changed a. Brett's comment makes sense here. Add /etc/cloud/cloud.cf.g.d/80-openstack-network-per-boot.cfg to the affected VM prior to network migration or MTU change: cat > /etc/cloud/cloud.cfg.d/80-openstack-network-per-boot.cfg <<EOF apply network config on every bootupdates: b. Make MTU change or perform OVS->OVN migration steps Option 2. Configure cloud-init to react to udev hotplug events (v. 21.3 or later) and re-render network on such udev events on the running system without reboot -- Do we know if migration from OVS -> OVN results in add|remove udev events, a VM configured for cloud-init hotplug would automatically read and apply new network_data.json values upon receipt of that udev event? If udev events occur along this migration path, we can: Allow datasource to regenerate network config on any udev hotplug eventsupdates: b. Restart the VM prior to MTU change or OVS->OVN migration which will start systemd cloud-init-hotplug.socket and cloud-init-hotplug.service on the system to listen for udev hotplug events c. Make changes to MTU or OVS-OVN migration to trigger udev events which will re-apply network config If migration or MTU changes don't result in udev events it is possible to manually 'fake' such a hotplug event to force re-rendering of network if still you have console/ssh access to the VM; get a current physical interface name on the systemPHYS_INTERFACE= fake a hotplug add event on a known NIC physical interface triggering cloud-init to crawl and apply of network_data.json metadatasudo cloud-init devel hotplug-hook -s net handle --udevaction add -d ${PHYS_INTERFACE} |
|
Launchpad user Chad Smith(chad.smith) wrote on 2022-09-08T02:21:35.685717+00:00 I guess the third option that could trigger network re-write across migration or MTU changes would be that OpenStack updates the instance-id at http://169.254.169.254/latest/meta-data/instance-id whenever the MTU value changes. By default, cloud-init re-provisions an OpenStack machine with any user-data, vendor-data and network_data.json anytime the instance-id changes. It's a pretty big hammer, and no additional configuration would be needed for default cloud-init behavior. Upon, if cloud-init detects a change in http://169.254.169.254/latest/meta-data/instance-id it would re-provision the system based current network_data.json and user_data provided from openstack at that time. I'd be wary of this full reprovision scenario as it will also involve cloud-init re-running ssh host key generation so SSH automation, and any user creation, password setting based on userdata which may introduce obstacles due to unrecognized SSH host keys when trying to connect to the migrated instance. Or resetting initial user passwords which could have changed over the life of the VM. |
|
Launchpad user OpenStack Infra(hudson-openstack) wrote on 2023-03-06T09:21:13.127123+00:00 This issue was fixed in the openstack/nova 27.0.0.0rc1 release candidate. |
This bug was originally filed in Launchpad as LP: #1899487
Launchpad details
Launchpad user Frode Nordahl(fnordahl) wrote on 2020-10-12T13:37:40.543454+00:00
When using OpenStack cloud provider
cloud-initwill write out /etc/netplan/50-cloud-init.yaml at initial instance boot and not update it on subsequent boots of the instance.The OpenStack metadata service provides information about MTU for the network [0] and cloud-init takes this value and writes it into the netplan configuration [1].
A side effect of configuring the MTU through netplan is that the
systemd-networkd[Link] section [2] gets the MTUBytes value filled and this in turn makessystemd-networkdignore the MTU value provided by DHCP [3][4].During the lifetime of a cloud events occur that will force a operator to reduce the MTU available to instances attached to its overlay networks. This may happen because of software imposed change of tunnel type (GRE -> VXLAN, VXLAN -> GENEVE) or change of topology or encapsulation in the physical network equipment.
To maximize performance these clouds have configured their instances to use the maximum available MTU without leaving any headroom to account for such changes and the only way to move forward is to reduce the available MTU on the instances. We are facing a concrete challenge with this now where we have users wanting to migrate from VXLAN tunnels to GENEVE tunnels with 38 byte header size.
0: # curl http://169.254.169.254/openstack/2018-08-27/network_data.json
{"links": [{"id": "tapa035fb68-01", "vif_id": "a035fb68-010c-42e3-8da7-ea3c36a0d607", "type": "ovs", "mtu": 8942, "ethernet_mac_address": "fa:16:3e:31:26:f7"}], "networks": [{"id": "network0", "type": "ipv4_dhcp", "link": "tapa035fb68-01", "network_id": "b4ef84c0-1235-48a8-aaf7-03fab7ef5367"}], "services": []}
1: # cat /etc/netplan/50-cloud-init.yaml
This file is generated from information provided by the datasource. Changes
to it will not persist across an instance reboot. To disable cloud-init's
network configuration capabilities, write a file
/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
network: {config: disabled}
network:
version: 2
ethernets:
ens2:
dhcp4: true
match:
macaddress: fa:16:3e:31:26:f7
mtu: 8950
set-name: ens2
2: # cat /run/systemd/network/10-netplan-ens2.link
[Match]
MACAddress=fa:16:3e:31:26:f7
[Link]
Name=ens2
WakeOnLan=off
MTUBytes=8950
3: # cat /run/systemd/network/10-netplan-ens2.network
[Match]
MACAddress=fa:16:3e:31:26:f7
Name=ens2
[Link]
MTUBytes=8950
[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6
[DHCP]
RouteMetric=100
UseMTU=true
4: Oct 12 13:30:18 canary-3 systemd-networkd[24084]: /run/systemd/network/10-netplan-ens2.network: MTUBytes= in [Link] section and UseMTU= in [DHCP] section are set. Disabling UseMTU=.
The text was updated successfully, but these errors were encountered: