send_config_set does not wire in read_timeout for read_until_pattern

[nlspears]

I am attempting to use a script I have always used in the past without issue. I has not been used in a couple of months but now with the updated netmiko it no longer works. Here is the script and the errors I am getting.

from netmiko import ConnectHandler
import getpass
import time, sys, paramiko
import logging

logging.basicConfig(filename='test.log', level=logging.DEBUG)
logger = logging.getLogger("netmiko")

user = input("Enter your SSH username: ")
pword = getpass.getpass()

device = {
'device_type': 'cisco_xe',
#'ip': '192.168.43.10',
'username': user,
'password': pword,
#'secret':'password'
}

ipfile=open("iplist.txt") #This file contains a list of switch ip addresses.
#configfile=open("configvty.txt") #opening the config file with the changes you want to push
#config=configfile.read() ##reads the config file
#configfile.close() #closes the config file
command = [
"ip access-list extended vty",
"permit ip 192.168.208.0 0.0.7.255 any",
"end",
"wr",
]
for line in ipfile:
device['ip']=line.strip()
print("Connecting to Device " + line)
net_connect = ConnectHandler(**device)
print ("Applying Configuration to Device " + line)
output = net_connect.send_config_set(command)
print('Configuration Complete')

Initially, I was using the file "configvty.txt" to read from for the configs. As I am troubleshooting. I have switched to using the command list.

The error I am getting is below:

Traceback (most recent call last):
File "/home/nspears/environments/vtypush.py", line 36, in
output = net_connect.send_config_set(command)
File "/home/nspears/environments/vtyacl_update/lib/python3.10/site-packages/netmiko/base_connection.py", line 2171, in send_config_set
output += self.read_until_pattern(pattern=pattern, re_flags=re.M)
File "/home/nspears/environments/vtyacl_update/lib/python3.10/site-packages/netmiko/base_connection.py", line 651, in read_until_pattern
raise ReadTimeout(msg)
netmiko.exceptions.ReadTimeout:

Pattern not detected: '(?:L70\-MDF\-sw01.$|#.$)' in output.

Things you might try to fix this:

1. Adjust the regex pattern to better identify the terminating string. Note, in
   many situations the pattern is automatically based on the network device's prompt.
2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

[nlspears]

So I have noticed that the configuration is getting sent despite the error, but due to the error it doesn't continue the for loop.

[ktbyers]

@nlspears send_config_set() should only be used for configuration commands and Netmiko will automatically handle exiting configuration mode.

So your code should change to this (as far as command and your for loop):

command = [
  "ip access-list extended vty",
  "permit ip 192.168.208.0 0.0.7.255 any",
]

for line in ipfile:
     device['ip']=line.strip()
     net_connect = ConnectHandler(**device)
     output = net_connect.send_config_set(command)
     output += net_connect.save_config())
     net_connect.discnnect()

Let me know if that works or not.

[nlspears]

Sorry for late response. Will test today and reply. May I ask why the initial statement regarding send_config_set? I have used it in this script in the past to refer to the commented out portion using the configvty.txt file.

[nlspears]

Ok I tried your suggestions and received the same error.

[ktbyers]

Can you post the code you are sending and the full exception stack?

Also can you post your log file here i.e. the test.log?

To answer your question here: not been used in a couple of months but now with the updated netmiko it no longer works.

Netmiko's send_config_set() is intended only for configuration commands and automatically handles entering/exiting configuration mode.

Consequently, sending things like end and wr mem are not expected to work in send_config_set (the end might cause a problem due to changing the prompt state and the wr mem might not work due to potentially prompting for additional information and also be meaningfully slower than normal config commands). Hard to tell exactly why without really diving into it.

So at a high level, it was just lucky that what you sent in earlier versions of Netmiko worked.

[nlspears]

"To answer your question here: not been used in a couple of months but now with the updated netmiko it no longer works."

Hi Kirk,
What I meant here is not the actual config in the text file. You are correct. My past configs did not include end and wr mem. Here is an example of code that used to work, but now errors.

##imports python modules needed to work
from netmiko import ConnectHandler
import time, sys, getpass, paramiko

#I then define a network device dictionary consisting of a device_type, ip, username, and password.
user = input("Enter your SSH username: ")
pword = getpass.getpass()


device = {
    'device_type': 'cisco_ios',
    #'ip': '192.168.43.10',
    'username': user,
    'password': pword,
    #'secret':'password'
}
ipfile=open("iplist.txt") #This file contains a list of switch ip addresses.
configfile=open("configfile.txt") #opening the config file with the changes you want to push
configset=configfile.read() ##reads the config file
configfile.close() #closes the config file

for line in ipfile:
    device['ip']=line.strip()
    print("Connecting to Device " + line)
    net_connect = ConnectHandler(**device)
    #net_connect.enable()
    #switches.open()
    time.sleep(2)
    output = net_connect.send_config_set(configset)
    print ("Applying Configuration to Device " + line)
    print (output)

[ktbyers]

Okay, can you simplify this to a single device that is failing and what the configset variable actually includes?

Also please included the full exception stack trace corresponding to the failing (updated) code.

[sohalevi]

Hi! We encountered the same problem, please find below the details about it:

Netmiko version: 4.1.2
DUT: Cisco ASR1002-HX, IOS: asr1000-universalk9.17.09.02a.SPA.bin

Script Used:

from datetime import datetime
from netmiko import ConnectHandler
from pprint import pprint

device = {
    "device_type": "cisco_ios",
    "host": "192.168.1.1",
    "username": "username",
    "password": "password",
    "read_timeout_override": 90,
}

with ConnectHandler(**device) as net_connect:
    print(net_connect.find_prompt())
    try:
        start_time = datetime.now()
        output = net_connect.send_config_set(config, exit_config_mode=False, cmd_verify=False)
    finally:
        end_time = datetime.now()
        print(f"Exec time: {end_time - start_time}")

output.split('\n')
pprint(output)

OUTPUT:

[INFO    ]  Connected (version 2.0, client Cisco-1.25)
[INFO    ]  Auth banner: b'\r\n======================================================================\r\nMY Banner\r\n======================================================================\r\n'
[INFO    ]  Authentication (password) successful!
LABRouter#
Exec time: 0:00:04.647975


('configure terminal\n'
 'Enter configuration commands, one per line.  End with CNTL/Z.\n'
 'LABRouter(config)#!\n'
 'LABRouter(config)#crypto key generate rsa general-keys label CANAME '
 'modulus 2048\n'
 '% You already have RSA keys defined named CANAME.\n'
 '% They will be replaced.\n'
 '\n'
 '% The key modulus size is 2048 bits\n'
 '% Generating 2048 bit RSA keys, keys will be non-exportable...\n'
 '[OK] (elapsed time was 1 seconds)\n'
 '\n'
 'LABRouter(config)#\n'
 'LABRouter(config)#\n'
 'LABRouter(config)#!\n'
 'LABRouter(config)#!\n'
 'LABRouter(config)#ip host ldap.url 100 1.1.1.2\n'
 'LABRouter(config)#!\n'
 'LABRouter(config)#crypto pki trustpoint CANAME\n'
 'LABRouter(ca-trustpoint)# enrollment retry count 12\n'
 'LABRouter(ca-trustpoint)# enrollment retry period 5\n'
 'LABRouter(ca-trustpoint)# enrollment mode ra\n'
 'LABRouter(ca-trustpoint)# enrollment url http://1.1.1.1:80\n'
 'LABRouter(ca-trustpoint)# usage ike\n'
 'LABRouter(ca-trustpoint)# serial-number\n'
 'LABRouter(ca-trustpoint)# ip-address loopback1\n'
 'LABRouter(ca-trustpoint)# revocation-check none\n'
 'LABRouter(ca-trustpoint)# rsakeypair CANAME 2048\n'
 'LABRouter(ca-trustpoint)# auto-enroll 90 regenerate\n'
 'LABRouter(ca-trustpoint)# fqdn LABRouter.MyDomain.com\n'
 'LABRouter(ca-trustpoint)# subject-name '
 'CN=LABRouter,OU=ou_1,OU=ou_2,OU=ou_3,OU=ou_4,O=MyDomain,C=Count\n'
 'LABRouter(ca-trustpoint)# fingerprint '
 'mylongfingerprintnumber\n'
 'LABRouter(ca-trustpoint)# password mypassword\n'
 'LABRouter(ca-trustpoint)# source interface Loopback1\n'
 'LABRouter(ca-trustpoint)#!\n'
 'LABRouter(ca-trustpoint)#!\n'
 'LABRouter(ca-trustpoint)#crypto pki authenticate CANAME\n')

After crypto pki authenticate command output is not shown in the output variable and there are missing commands after it.

Here the configuration I am applying:

!
crypto key generate rsa general-keys label CANAME modulus 2048


!
!
ip host ldap.url 100 1.1.1.2
!
crypto pki trustpoint CANAME
 enrollment retry count 12
 enrollment retry period 5
 enrollment mode ra
 enrollment url http://1.1.1.1:80
 usage ike
 serial-number
 ip-address loopback1
 revocation-check none
 rsakeypair CANAME 2048
 auto-enroll 90 regenerate
 fqdn LABRouter.MyDomain.com
 subject-name CN=LABRouter,OU=ou_1,OU=sau,OU=ou_2,OU=ou_3,O=MyDomain,C=Count
 fingerprint mylongfingerprintnumber
 password mypassword
 source interface Loopback1
!
!
crypto pki authenticate CANAME
!
!
event manager applet MYEVENTMANAGER
event track 2 state down
! Generate syslog
action 10 syslog priority warnings msg "CUSTOM SYSLOG MSG"
action 20 cli command "enable"
action 30 cli command "show int desc"
!
!
interface GigabitEthernet 1
 ip tcp adjust-mss 1360
!
!
end
! end of Config (ETH) of LABRouter
!

So the event manager part and the part after that got configured on the router, however are not visible in the output variable.

Tested manually, that at the line crypto pki authenticate CANAME is waiting around 20 seconds, until than there es no prompt visible.

I also tried if I do a read_channel after the script is finished: the rest of the missing configurations are shown.

Am I missing a setting with which I could get the whole cli output?
Also tried with send_config_set(read_timeout=60), but I got the same results.

Thank you for looking into it.

[ktbyers]

@sohalevi So when you do this manually:

Tested manually, that at the line crypto pki authenticate CANAME is waiting around 20 seconds, until than there es no prompt visible.

it sits there for around 20 seconds and then the prompt appears?

[sohalevi]

Hi, sorry for not answering your question.
It sits there for 30 secs (I just measured it to be more accurate), and after that messages appear without the prompt at the biginning, but after them the prompt appears in a new line.

[jchristopher327]

I'm hitting this issue as well it seems on 4.1.2

When configuring the boot statement on an Arista switch, the prompt comes back after ~13 seconds. The read_timeout in the task (using Nornir) doesn't seem to change it.

Basic code:

            # Set the boot config.
            task.run(
                name="Set boot config",
                task=netmiko_send_config,
                config_commands="boot system flash:/" + task.host["image"],
                read_timeout=60,
            )

Exception raised comes from the read_until_pattern function.

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/nornir/core/task.py", line 99, in start
    r = self.task(self, **self.params)
  File "/usr/local/lib/python3.9/site-packages/nornir_netmiko/tasks/netmiko_send_config.py", line 38, in netmiko_send_config
    result = net_connect.send_config_set(config_commands=config_commands, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 2206, in send_config_set
    output += self.read_until_pattern(pattern=pattern, re_flags=re.M)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 672, in read_until_pattern
    raise ReadTimeout(msg)
netmiko.exceptions.ReadTimeout:

Pattern not detected: '(?:switch.*$|#.*$)' in output.

Things you might try to fix this:
1. Adjust the regex pattern to better identify the terminating string. Note, in
many situations the pattern is automatically based on the network device's prompt.
2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

[jchristopher327]

Looking over the code, it looks like the send_config_set function sets the read_timeout to 15 if it's None or assigns whatever value was passed to the function. When the read_until_pattern function is called (I'm guessing here), the read_timeout is not passed to it and the function has it set to 10.

[ktbyers]

@jchristopher327 @sohalevi

Okay, it looks like there is a mismatch in the expected behavior of read_timeout and send_config_set (i.e. you are expecting it to do something it doesn't actually do). This definitely could be a design error on my part.

From the code for send_config_set()

        :param read_timeout: Absolute timer to send to read_channel_timing. Should be rarely needed.

https://github.com/ktbyers/netmiko/blame/develop/netmiko/base_connection.py#L2151

It is intentionally not used for the read_until_pattern call.

Can you try using the following arguments to send_config_set

cmd_verify=False
read_timeout=60

I am going to leave this issue open and change the title (as the behavior should probably be changed).

[jchristopher327]

Hi Kirk.

I thought I had tested your suggestion earlier but noticed I was setting cmd_verify=False in the wrong place. This now works.

            # Set the boot config.
            task.run(
                name="Set boot config",
                task=netmiko_send_config,
                config_commands="boot system flash:/" + task.host["image"],
                read_timeout=60,
                cmd_verify=False,
            )

[jchristopher327]

I spoke too soon. I just realized I tested while having read_timeout_override: 300 in groups.yml which allows it to work. If I remove it, then I get the same traceback.

[ktbyers]

Did you try setting read_timeout=300 here (and remove the read_timeout_override)?

            # Set the boot config.
            task.run(
                name="Set boot config",
                task=netmiko_send_config,
                config_commands="boot system flash:/" + task.host["image"],
                read_timeout=300,
                cmd_verify=False,
            )

[jchristopher327]

No I hadn't tried that but just did with the same result.

            # Set the boot config.
            task.run(
                name="Set boot config",
                task=netmiko_send_config,
                config_commands="boot system flash:/" + task.host["image"],
                read_timeout=300,
                cmd_verify=False,
            )

eos:
  platform: "eos"
  port: 443
  connection_options:
    netmiko:
      port: 22
      extras:
  data:
    replace_config: false

traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/nornir/core/task.py", line 99, in start
    r = self.task(self, **self.params)
  File "/usr/local/lib/python3.9/site-packages/nornir_netmiko/tasks/netmiko_send_config.py", line 38, in netmiko_send_config
    result = net_connect.send_config_set(config_commands=config_commands, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 2214, in send_config_set
    output += self.exit_config_mode()
  File "/usr/local/lib/python3.9/site-packages/netmiko/cisco_base_connection.py", line 55, in exit_config_mode
    return super().exit_config_mode(exit_config=exit_config, pattern=pattern)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 2024, in exit_config_mode
    if self.check_config_mode():
  File "/usr/local/lib/python3.9/site-packages/netmiko/arista/arista.py", line 46, in check_config_mode
    output = self.read_until_pattern(pattern=pattern)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 672, in read_until_pattern
    raise ReadTimeout(msg)
netmiko.exceptions.ReadTimeout:

Pattern not detected: '[>\\#]' in output.

Things you might try to fix this:
1. Adjust the regex pattern to better identify the terminating string. Note, in
many situations the pattern is automatically based on the network device's prompt.
2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

[ktbyers]

@jchristopher327 Okay, that is a different failure than you were getting earlier. This one is failing on exit_config_mode().

You might need to look at the Netmiko session_log to get more details about why it is failing.

eos:
  platform: "eos"
  port: 443
  connection_options:
    netmiko:
      port: 22
      extras:
          # Note, this assumes you are only testing against a single device
          # (as the group is sharing the same file).
          session_log: "output.txt"
  data:
    replace_config: false

[jchristopher327]

Unfortunately not much to go on. The output of the session log just ends with the boot config command.

# output omitted

switch#verify /md5 flash:EOS-4.27.9M.swi
verify /md5 (flash:EOS-4.27.9M.swi) = e553642aca96f34a54065c637059994a
switch#
switch#
switch#configure terminal
switch(config)#
switch(config)#boot system flash:/EOS-4.27.9M.swi

[ktbyers]

Okay, let me look at it some more.

[ktbyers]

@jchristopher327 Can you test this proposed fix?

#3156

You will need to use these settings:

            # Set the boot config.
            task.run(
                name="Set boot config",
                task=netmiko_send_config,
                config_commands="boot system flash:/" + task.host["image"],
                read_timeout=N,         # Where N is some time in seconds with enough margin so that 'boot' command completes
                cmd_verify=True,        # This is the default
            )

[jchristopher327]

Hey Kirk,

I tried installing the fix but started running into dependency issues that I think boil down to nornir-netmiko pinning textfsm==1.1.2. I noticed that napalm had reverted the pin some time ago.

Let me know if I should open a PR to on the nornir-netmiko repo with textfsm^1.1.2 so I can test this out.

[ktbyers]

Yeah that pinning should go away. Do you want to do a PR on that?

We should just remove that dependency in nornir_netmiko as such (i.e. remove the textfsm dependency). Netmiko will install it.

[jchristopher327]

Here's the PR in nornir-netmiko ktbyers/nornir_netmiko#55

[ktbyers]

@jchristopher327 Okay, that is merged so you would need the develop version of nornir_netmiko. We can push a new release of nornir_netmiko once we verify the fix in the PR works for Netmiko works.

[jchristopher327]

The proposed fix worked. Here's a rundown of the things I did on my end to test. Let me know if there's anything else you'd like me to test.

used the develop branch of napalm since 4.0.0 has textfsm<=1.1.2 as a dependency

-napalm==4.0.0
+git+https://github.com/napalm-automation/napalm.git@develop

use the netmiko fix

-netmiko==4.1.2
+git+https://github.com/ktbyers/netmiko.git@send_config_set_read_timeout

use nornir_netmiko without the textfsm dependency (fyi, my PR was merged into master not develop)

-nornir-netmiko==0.2.0
+git+https://github.com/ktbyers/nornir_netmiko.git@master

code changes

--- a/inventory/groups.yml
+++ b/inventory/groups.yml
@@ -146,7 +146,6 @@ eos:
     netmiko:
       port: 22
       extras:
-        read_timeout_override: 300
   data:
     replace_config: false
     image: "EOS-4.27.9M.swi"

--- a/eos_upgrade.py
+++ b/eos_upgrade.py
@@ -92,6 +92,7 @@ def software_validate(task):
                 task=netmiko_send_config,
                 config_commands="boot system flash:/" + task.host["image"],
                 read_timeout=30,  # this is currently broken. updated an existing netmiko issue (2774)
+                cmd_verify=True,
             )

script result

---- Set boot config ** changed : True ----------------------------------------- INFO
configure terminal
switch(config)#boot system flash:/EOS-4.27.9M.swi
switch(config)#end
switch#
---- Commit configuration ** changed : False ----------------------------------- INFO
Copy completed successfully.

[ktbyers]

This was fixed with #3156