[BUG] Salt minion StreamClosedError when Salt Master IP changes

[nick-stocks]

Description
When a salt master's IP changes, a minion will lose its connection and fail continuously until the salt-minion service is restarted. Only then will it pick up the new IP

Seems to be a regression of issue 61482
Also reviewed PR 61577

Setup
Any salt minion/master setup with 3005.1 on both minion & master

Steps to Reproduce the behavior
Create a salt minion / master pair running 3005.1. Stop the salt-master, change its IP and start it back up.

Below error will appear continuously until the salt-minion has been restarted

2023-02-07 10:54:34,672 [salt.transport.tcp:625 ][WARNING ][498518] TCP Message Client encountered an exception while connecting to 192.168.0.5:4506: StreamClosedError('Stream is closed',), will reconnect in 1 seconds 

Expected behavior
After x timeout the salt minion should do a DNS lookup to validate the IP hasn't changed

Versions Report

[root@ip-192-168-0-13 bin]# salt-call --versions-report
Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: 1.11.5
      cherrypy: Not Installed
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Oct 20 2022, 09:31:56)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rhel 8.7 Ootpa
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-425.10.1.el8_7.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.7 Ootpa

Additional context
Have tried many different combinations of salt-minion config. None change the behaviour

Below constant

master_alive_interval: 15
random_startup_delay: 5
ping_interval: 1

Below tested for -1, 1 and 15

master_tries: -1

Below tested for both true and false values

auth_safemode

Also tested with and without the below

tcp_keepalive: True
tcp_keepalive_idle: 60
tcp_keepalive_cnt: 5
tcp_keepalive_intvl: 60

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[fraserOrr]

I'm seeing the same issue on the same version of salt with the DNS not resolving to a new IP after some machine cycling. I have the ping set, but even with the auth_safe mode im not seeing the minion restart at any point, which should force the new DNS resolution.