LINUX: multiple IP addresses on QETH device override routing information on host system #230
Comments
|
In case the QETH device is used as an uplink port to a VSWITCH, is the IP address at the QETH statement in the Hercules configuration file really needed? What is the need of defining this address? Because it is a tunX device??? |
|
It would be nice to see the Hercules config statements that define the LCS and the QETH devices. It would also be useful if you could issue ip addr as well as ip route commands at each step. |
|
The Hercules config statements and all ip route results are shown above. |
$ ip address
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen
1000
link/ether 00:e0:81:b5:14:d1 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.190/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::2e0:81ff:feb5:14d1/64 scope link
valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:e0:81:b5:14:d0 brd ff:ff:ff:ff:ff:ff
67: tap0: mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether 7a:67:2f:7e:e0:71 brd ff:ff:ff:ff:ff:ff
inet6 fe80::7867:2fff:fe7e:e071/64 scope link
valid_lft forever preferred_lft forever
68: tun1: mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500 link/none
inet 10.10.0.190 peer 10.0.0.192/24 scope global tun1
valid_lft forever preferred_lft forever
inet6 fe80::cfab:d32c:b5c:27f/64 scope link flags 800
valid_lft forever preferred_lft forever
|
|
During some rare occasions the following message are issued by Hercules and z/VM: This is NOT happening all the time. I couldn't find the reason why these messages are issued yet. |
|
The LCS is taken down, but its IP address is not being unregistered with QETH. When the LCS is brought up again (the ... because it is already registered, QETH has returned IPA_RC_IP_ADDR_ALREADY_USED (0xE00A), prompting the z/VM |
|
Regarding the ip address output you provided: Was that after the z/VM VSWITCH was up? Or after the z/OS QETH was up? Or after the z/OS LCS was up? Is this route problem reproducible? Does it happen all of the time, or only occasionally? QETH itself does nothing with routes. QETH simply says use this IP address with this interface and the Linux kernel sets up the appropriate route table entries. This is why I asked for both the ip address and ip route output at each stage: To try and understand what might be confusing the Linux kernel. (Unfortunately I don't have openSUSE or z/VM 6.4 or z/OS 2.3, and I run Hercules with no privileges, so can't reproduce any of your environment.) |
|
The addresses of the interfaces do not change. Therefore I added the addresses when all interfaces are active. All the addresses are unregistered when the QETH stops and all routes are removed from the host as well. LCS starts and the MAC address of it is shown in the log. When QETH starts, the configured address of the IP stack is pingable. Once z/OS discovers it has another interface, it then registers that address also on QETH, and in the same moment routing is overwritten on the host (with the network 10.0.0.0/24 on tun1, which is the subnet of LCS, and the rest of the private LAN overriding the route to 10.10.0.0/24 of QETH) . |
|
I'm sure your description of what happens is what happens, but we need detailed information to try and understand what happens. For example, your ip address output contains: inet 10.10.0.190 peer 10.0.0.192/24 Which of the three steps (VSWITCH start? z/OS QETH start? Or z/OS LCS start?) sets the prefix length? The z/OS QETH start should have set the peer address to 10.10.0.192. Did it? Would you please provide the full ip address and ip route output after each of the three steps, so that we can see what effect each step is having on the tun interface? Thanks. |
Fish-Git
added
(Unknown)
|
Find hereby all the IP ROUTE and IP ADDRESS results during the various stages. |
|
Rule one of the Linux kernel is that if an entity tells it to do something stupid, the entity knows what it is doing. In this case the entity (i.e. Hercules) is telling the kernel to use an IP address from subnet B with a tun interface defined on subnet A. The fact that it messes up networking? Refer to rule one! I was never keen on adding VIPA support to QETH because I could foresee that what is now happening would happen. *nix tun interfaces should only ever have two IP addresses, the local (i.e. *nix host) and the peer (i.e. the Hercules guest), which should both be in the same subnet. If the peer wishes to use additional IP addresses, *nix host routing should be used to route the additional address via the peer address. z/OS Comms Server considering all home address to be VIPA is, for Hercules, a complete pain in the arse. In the real world it doesn't matter, everyone out there using real z iron is only using OSA. (Though I don't think it's only a z/OS problem, I have noticed that on my z/VM 6.1 system the default gateway IP address (with is on a CTCI interface) is being registered with the layer3 VSWITCH. I have no idea why!) There are various long(er) term solutions:
In the short term I'm afraid the only solution is to either:
|
|
Would it not be easier to simulate OSA layer 3 devices if the Hercules implementation would use a *nix TAP device for it ? I don't know what caused the decision to use a TAP devices only for layer 2, and a TUN device for layer 3, as I joined the Hercules community too late to be aware of the original reasoning for this. I'd be interested to learn about the reasons for that. For a Windows host, I presume the Fish solution may effectively already be making no real TUN/TAP difference for supporting OSA layer 2 or 3. (Whilst of course handling the different L2/L3 packet header formats correctly.) How does zPDT support L2 vs L3, or does it perhaps have the same limitations as Hercules ? Cheers, Peter |
IMHO, yes, it would. QETH would have to wrap the layer3 IP packets in Ethernet frames, and would have to handle IPv4 ARP/RARP and IPv6 Neighbor Discovery, but its all doable.
When I became involved the decision had already been made. I've always assumed that Jan simply thought 'layer2 = Ethernet = tap, layer3 = IP = tun'. I'm sure that at the time he had more important matters to worry about, like actually getting QETH to work at all.
Yes, presumably. I've always assumed WIN-CTCI deals with Ethernet frames from/to the host, but I've never felt the need to read winpcap documentation to find out.
I don't know how it's done, I've never had access to a zPDT. I wouldn't have thought zPDT uses tuntap to get traffic to/from the hosts network interface(s), but... |
|
Thanks Ian for confirming my thoughts. So we know what we could work on when we get bored this coming winter … Cheers, |
I do run Hercules non-privileged nevertheless it changes the IP configuration. Isn't it possible that the additional route is added by Hercules when it is encountered that there is another subnet address registered on the interface? Regarding zPDT: zPDT networking is quite complex. Below is a snapshot from command 'find_io' which shows the network configuration in zPDT and the corresponding definitions in the configuration file of zPDT. zPDT can connect directly to an Ethernet adapter (see F0 and F1 below). Side effect of this implementation is that the host IP address cannot be reached from z/OS when using the same ethX adapter. Therefore it is recommended to use a separate ethX adapter for zPDT. In my case I use eth1 (F1) for connections of the systems on zPDT. I use F00-F02 as uplink port for vswitch VMINTR01. All z/VM guests connect then to this vswitch. The /dev/tap0 - /dev/tap7 are used for PTP connections. With this connection the host can be reached. IP addresses are always of type 10.1.x.1 where x is the number of /dev/tap interface. FIND_IO on "ibmsys1" zPDT: At the zPDT configuration it is defined like here: On z/VM: query vswitch vmintr01 VSWITCH SYSTEM VMINTR01 Type: QDIO Connected: 5 Maxconn: INFINITE PERSISTENT RESTRICTED NONROUTER Accounting: OFF USERBASED LOCAL VLAN Unaware MAC address: 02-00-00-00-00-02 MAC Protection: OFF IPTimeout: 5 QueueStorage: 8 Isolation Status: OFF VEPA Status: OFF Uplink Port: State: Ready PMTUD setting: EXTERNAL PMTUD value: 8992 Trace Pages: 8 RDEV: 0F00.P00 VDEV: 0603 Controller: DTCVSW1 ACTIVE Ready; T=0.01/0.01 13:04:26 query f00-f02 OSA 0F00 ATTACHED TO DTCVSW1 0603 DEVTYPE OSA CHPID F1 OSD OSA 0F01 ATTACHED TO DTCVSW1 0604 DEVTYPE OSA CHPID F1 OSD OSA 0F02 ATTACHED TO DTCVSW1 0605 DEVTYPE OSA CHPID F1 OSD Ready; T=0.01/0.01 13:04:37 |
The network configuration is done by module hercifc, which has elevated privileges otherwise it would not be able to change the configuration.
No, it isn't. QETH (or more specifically hercifc at QETHs request) only issues ioctl SIOCSIFDSTADDR requests to add the addresses registered on the interface. QETH does not issue routing related ioctl requests. It is the kernel itself that manipulates the routing table as a result of the ioctl SIOCSIFDSTADDR request(s). LCS is the only Hercules interface type that issues routing related ioctl requests. Hmm, looks like zPDT is using tap bridged to a host ethernet interface? |
|
I don't know really. There is no additional /dev/tap device showing up on an ip link or ip address command. It is a sort of miracle how this is done! The implementation of zPDT acts really as an OSA with VIPA support and so on. When LCS can do routing related ioctl requests why not the QETH driver (or probably hercifc??) |
Indeed!
QETH could do routing related ioctl requests, the code exists already in hercifc for use by LCS. |
|
Then probably the use for QETH could be enabled? LCS registers a host route while QETH a network route. |
|
I don't intend to do anything about this problem. As I said earlier there are a couple of solutions to avoid this problem, either:
|
mcisho
removed
(Unknown)
Fish-Git
added
the
(*WON'T FIX*)
z/VM VSWITCH:
z/OS network interfaces:
1.) LCS (F00-F01) IP: 10.0.0.192/24 (= same subnet as all other systems on the private LAN)
2.) OSD (EC00-EC02) IP: 10.10.0.192/24 (different subnet)
z/OS 2.3 registers the LCS address 10.0.0.192 also on the QETH, and the routing of the LINUX host is also changed. As a result, it defines the subnet 10.0.0.0/24 (the subnet of the host) to tun1, which is the interface for the QETH VSWITCH, and 10.10.0.0/24 addresses do NOT work since the route 10.10.0.0/24 to tun1 ends up being replaced). I have to manually fix the routing by adding manually 10.10.0.0/24 dev tun1 route.
In the Hercules system log it looks like
You see that there is a small time gap before the 10.0.0.192 address is registered on QETH device.
Routing information at the time z/VM has started: the tun1 device is the interface for QETH EC00-EC02:
LINUX host:
At this moment the z/OS guest is not running and also both, the LCS and the QETH address cannot be pinged. This works as expected.
Now when z/OS TCPIP stack is started and both devices LCS/QETH are activated by TCPIP this happens at host site:
LINUX host:
On Hercules:
LINUX host:
In the meantime the QETH address is pingable. Works as expected.
Then on Hercules:
LINUX host:
In this situation 10.10.0.192 is not reachable any more because traffic 10.10.0.0/24 goes to the default route. Therefore 10.10.0.192 is not reachable any more.
This is the problem.
I think the 10.10.0.0/24 dev tun1 route should NOT be deleted and the 10.0.0.0/24 should be ADDED additionally.
@ivan-w Ivan,
It is NOT the case that those interfaces have the SAME IP addresses. Each has its own subnet 10.0.0.0/24 LCS , 10.10.0.0/24 QETH. Only the last octet is the same.
The text was updated successfully, but these errors were encountered: