Setting `ping_interval` in salt-minion's config (version 2019.2.1) prevents it from starting #54776

javierbertoli · 2019-09-26T17:36:54Z

Description of Issue

Due to errors in the the minion's function remove_periodic_callback(), when setting the parameter ping_interval in the minion configuration (ie, to check against a DDNS master, as described here), the minion fails to start and floods the logs with

[CRITICAL] Unexpected error while connecting to salt
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1027, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 2733, in tune_in
    self.remove_periodic_callbback('ping', ping_master)
AttributeError: 'Minion' object has no attribute 'remove_periodic_callbback'

At first it seems to be caused by a typo in remove_periodic_callbback() (note the 2 bb) but when fixing it, the error changes to

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1035, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 2741, in tune_in
    self.remove_periodic_callback('ping', ping_master)

and does not start either.

Steps to Reproduce Issue

Install the salt-minion v2019.2.1 packages
Add the following config parameters

cat > /etc/salt/minion.d/test.conf << EOF
auth_tries: 2
master_tries: -1
auth_safemode: false
ping_interval: 5
EOF

Run salt-minion -l debug to get the error.
Remove the ping_interval parameter, repeat step 3 and it will run OK.

Versions Report

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Apr  3 2019, 05:39:12)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist: debian 10.1 
         locale: utf-8
        machine: x86_64
        release: 5.3.0-rc5-amd64
         system: Linux
        version: debian 10.1

The text was updated successfully, but these errors were encountered:

cmcmarrow · 2019-09-26T18:56:03Z

I can confirm that this happens as on Ubuntu as well.

morsik · 2019-09-29T14:31:34Z

Same on CentOS 7.6 with minions deployed via Salt Cloud.

0x647262 · 2019-09-30T19:43:49Z

Master:

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 3.6.8 (default, Aug 20 2019, 17:12:48)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1048-aws
         system: Linux
        version: Ubuntu 18.04 bionic

Minion:

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug 20 2019, 17:12:48)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-64-generic
         system: Linux
        version: Ubuntu 18.04 bionic

As mentioned in the original report, downgrading from to 2019.2.0 or removing the ping_interval configuration option resolves the issue

sastorsl · 2019-10-01T07:14:53Z

Same on Red Hat Enterprise Linux Server 7 / RHEL7

# salt-call -V
Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.31.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Jun 11 2019, 14:33:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.7 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.1.1.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.7 Maipo

Errors in /var/log/salt/minion

2019-10-01 09:15:18,923 [salt.minion      :2313][ERROR   ][2611] This Minion is already running. Not running Minion.tune_in()
2019-10-01 09:15:18,946 [salt.minion      :2692][INFO    ][2611] Minion is ready to receive requests!
2019-10-01 09:15:18,947 [salt.minion      :1049][CRITICAL][2611] Unexpected error while connecting to 10.246.104.52
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1027, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 2733, in tune_in
    self.remove_periodic_callbback('ping', ping_master)
AttributeError: 'Minion' object has no attribute 'remove_periodic_callbback'

NOT having this problem with salt-2019.2.0 on an identical server with redhat 7.6 Maipo running 3.10.0-957.12.2.el7.x86_64.

hedrickbt · 2019-10-01T20:06:41Z

I have created a support ticket for this:
https://help.saltstack.com/hc/en-us/requests/4183

Xopherus · 2019-10-17T15:45:44Z

Has there been a version released with this fix? If not, when can we expect that?

hedrickbt · 2019-10-17T17:04:46Z

@Xopherus ,

The following pull request is being tested. #54777

The plan is for the fix to be included in 2019.2.2. The last update I have from support is 10 days ago. I asked for an update today.

A notice has been put out to let people know to avoid going to 2019.2.1 at https://repo.saltstack.com/

hedrickbt · 2019-10-18T19:12:02Z

Be on the look out for a possible release mid-next-week.

cachedout · 2019-10-21T16:16:41Z

Now that #54777 is merged, shouldn't this be closed?

javierbertoli · 2019-10-21T17:21:18Z

Applied the patch when I submitted #54777 and it has been working for me since then.

hedrickbt · 2019-10-24T20:57:51Z

2019.2.2 has been released with the patch.

javierbertoli mentioned this issue Sep 26, 2019

Fix minion's remove_periodic_callback() #54777

Merged

cmcmarrow added this to the Approved milestone Sep 26, 2019

This was referenced Sep 27, 2019

AttributeError: 'Minion' object has no attribute 'remove_periodic_callbback' #54795

Closed

AttributeError: 'Minion' object has no attribute 'remove_periodic_callbback' #54796

Closed

doesitblend added the ZD The issue is related to a Zendesk customer support ticket. label Oct 1, 2019

mariusvw mentioned this issue Oct 8, 2019

Fixed typo #54926

Closed

garethgreenaway closed this as completed Oct 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setting `ping_interval` in salt-minion's config (version 2019.2.1) prevents it from starting #54776

Setting `ping_interval` in salt-minion's config (version 2019.2.1) prevents it from starting #54776

javierbertoli commented Sep 26, 2019

cmcmarrow commented Sep 26, 2019

morsik commented Sep 29, 2019 •

edited

Loading

0x647262 commented Sep 30, 2019

sastorsl commented Oct 1, 2019 •

edited

Loading

hedrickbt commented Oct 1, 2019

Xopherus commented Oct 17, 2019

hedrickbt commented Oct 17, 2019

hedrickbt commented Oct 18, 2019

cachedout commented Oct 21, 2019

javierbertoli commented Oct 21, 2019

hedrickbt commented Oct 24, 2019

Setting ping_interval in salt-minion's config (version 2019.2.1) prevents it from starting #54776

Setting ping_interval in salt-minion's config (version 2019.2.1) prevents it from starting #54776

Comments

javierbertoli commented Sep 26, 2019

Description of Issue

Steps to Reproduce Issue

Versions Report

cmcmarrow commented Sep 26, 2019

morsik commented Sep 29, 2019 • edited Loading

0x647262 commented Sep 30, 2019

sastorsl commented Oct 1, 2019 • edited Loading

hedrickbt commented Oct 1, 2019

Xopherus commented Oct 17, 2019

hedrickbt commented Oct 17, 2019

hedrickbt commented Oct 18, 2019

cachedout commented Oct 21, 2019

javierbertoli commented Oct 21, 2019

hedrickbt commented Oct 24, 2019

Setting `ping_interval` in salt-minion's config (version 2019.2.1) prevents it from starting #54776

Setting `ping_interval` in salt-minion's config (version 2019.2.1) prevents it from starting #54776

morsik commented Sep 29, 2019 •

edited

Loading

sastorsl commented Oct 1, 2019 •

edited

Loading