I started my IT career at an ISP help desk back in the early 2000s. I remember watching the network engineers typing in what seemed like cryptic commands into a text terminal. Like magic, the network devices would then bend to their will and behave in the way they intended. In time, I got to learn and embrace these magic commands that I could type into the terminal. As network engineers, these CLI-based commands are like secret codes we share with each other in this world we call network engineering. Manually typing in the command was just something we all had to do to get the job done, no harm, no foul.

However, it was right around the year 2014 when we started to see the industry coming to a consensus about the clear need to move away from manual, human-driven CLIs toward an automatic, computer-centric automation API. Make no mistake, we still need to directly communicate with the device when making network designs, bringing up an initial proof of concept, and deploying the topology for the first time. However, once the network is deployed, the network management requirement is now to consistently make the same changes reliably across all network devices.

These changes need to be error-free, and the engineers need to repeat the steps without being distracted or feeling tired. This requirement sounds like an ideal job for computers and our favorite programming language, Python.

Of course, if the network devices can only be managed with the command line, the main challenge becomes how we can replicate the previous manual interactions between the router and the administrator automatically with a computer program. In the command line, the router will output a series of information and will expect the administrator to enter a series of manual commands based on the engineer’s interpretation of the output. For example, in a Cisco Internetwork Operating System (IOS) device, you have to type in enable to get into a privileged mode, and upon receiving the returned prompt with the # sign, you then type in configure terminal in order to go into the configuration mode. The same process can further be expanded into the interface configuration mode and routing protocol configuration mode. This is in sharp contrast to a computer-driven, programmatic mindset. When the computer wants to accomplish a single task, say, put an IP address on an interface, it wants to structurally give all the information to the router at once, and it would expect a single yes or no answer from the router to indicate the success or failure of the task.

The solution, as implemented by both Pexpect and Paramiko, is to treat the interactive process as a child process and watch over the interaction between the child process and the destination device. Based on the returned value, the parent process will decide the subsequent action, if any.

I am sure we are all anxious to get started on using the Python libraries, but first, we will need to construct our network lab in order to have a network to test our code against. We will begin by looking at different ways we can build our network labs.

Before we dive into the Python libraries and frameworks, let’s examine the options of putting together a lab for the benefit of learning. As the old saying goes, “practice makes perfect” – we need an isolated sandbox to safely make mistakes, try out new ways of doing things, and repeat some of the steps to reinforce concepts that were not clear on the first try.

To put together a network lab, we basically have two options: physical devices or virtual devices. Let’s look at the advantages and disadvantages of the respective options.

Physical devices

This option consists of putting together a lab with physical network devices that you can see and touch. If you are lucky enough, you might even be able to construct a lab that is an exact replication of your production environment. The advantages and disadvantages of a physical lab are as follows:

• Advantages: It is an easy transition from lab to production. The topology is easier to understand for managers and fellow engineers who can look at and work on the devices if need be. The comfort level with physical devices is extremely high because of familiarity.
• Disadvantages: It is relatively expensive to pay for devices that will only be used in a lab. Also, physical devices require engineering hours to rack and stack and are not very flexible once constructed.

Virtual devices

Virtual devices are emulations or simulations of actual network devices. They are either provided by the vendors or by the open source community. The advantages and disadvantages of virtual devices are as follows:

• Advantages: Virtual devices are easier to set up, relatively cheap, and can make changes to the topology quickly.
• Disadvantages: They are usually scaled-down versions of their physical counterparts. Sometimes there are feature gaps between the virtual and the physical device.

Of course, deciding on a virtual or physical lab is a personal decision derived from a trade-off between the cost, ease of implementation, and the risk of having a gap between lab and production environments. In some of the places I have worked, the virtual lab was used when doing an initial proof-of-concept, while the physical lab was used when we moved closer to the final design.

In my opinion, as more and more vendors decide to produce virtual appliances, the virtual lab is the way to proceed in a learning environment. The feature gap of the virtual appliance is relatively small and specifically documented, especially when the virtual instance is provided by the vendor. The cost of the virtual appliance is relatively small compared to buying physical devices. The time to build using virtual devices is much shorter because they are just software programs.

For this book, I will use a combination of physical and virtual devices for concept demonstration, with a preference for virtual devices. For the examples we will see, the differences should be transparent. If there are any known differences between the virtual and physical devices pertaining to our objectives, I will make sure to list them.

For the code examples in the book, I will try to make the network topology as simple as possible while still being able to demonstrate the concept at hand. Each virtual network usually consists of not more than a few nodes, and we will reuse the same virtual network for multiple labs if possible.

For the examples in this book, I will utilize Cisco Modeling Labs, https://www.cisco.com/c/en/us/products/cloud-systems-management/modeling-labs/index.html, as well as other virtual platforms, such as Arista vEOS. As we will see in the next section, Cisco provides CML in both a paid version and a free, hosted version on Cisco DevNet (https://developer.cisco.com/site/devnet/) based on availability. The use of CML is optional. You can use any lab devices you have, but it might make it easier to follow along with the book examples. Also worth noting is that Cisco has strict software license requirements for device images, so by purchasing or using the free hosted CML, you will be less likely to violate their software license requirements.

Cisco modeling labs

I remember when I first started to study for my Cisco Certified Internetwork Expert (CCIE) lab exam, I purchased some used Cisco equipment from eBay to study with. Even with a used equipment discount, each router and switch still cost hundreds of US dollars. To save money, I purchased some really outdated Cisco routers from the 1980s (search for Cisco AGS routers in your favorite search engine for a good chuckle), which significantly lacked features and horsepower, even for lab standards. As much as it made for an interesting conversation with family members when I turned them on (they were really loud), putting the physical devices together was not fun. They were heavy and clunky, and it was a pain to connect all the cables, and to introduce link failure, I would literally have to unplug a cable.

Fast-forward a few years. Dynamips was created, and I fell in love with how easy it was to create different network scenarios. This was especially important when trying to learn a new concept. All I needed was the IOS images from Cisco and a few carefully constructed topology files, and I could easily build a virtual network to test my knowledge on. I had a whole folder of network topologies, pre-saved configurations, and different versions of images, as called for by different scenarios. The addition of a GNS3 frontend gave the whole setup a beautiful GUI facelift. With GNS3, you can just click and drop your links and devices; you can even print out the network topology for your manager or client right out of the GNS3 design panel. The only disadvantage of GNS3 was the tool not being officially blessed by Cisco, and the perceived lack of credibility because of it.

In 2015, the Cisco community decided to fulfill this need by releasing the Cisco Virtual Internet Routing Lab (VIRL), https://learningnetwork.cisco.com/s/virl. This quickly became my go-to tool as the network lab when developing, learning, and practicing network automation code.

A few years after the introduction of VIRL, Cisco released Cisco Modeling Labs (CML), https://developer.cisco.com/modeling-labs/. It is a great network simulation platform with an easy-to-use HTML UI and a comprehensive API.

At the time of writing, the single-user license for CML is 199 USD (keep in mind that there is a free, hosted version on Cisco DevNet). In my opinion, the CML platform offers a few advantages over other alternatives and the cost is a bargain:

• Ease of use: As mentioned, all the images for IOSv, IOS-XRv, NX-OSv, ASAv, and other images are included in a single download.
• Official: CML is a widely used tool internally at Cisco and within the network engineering community. In fact, CML is used extensively for the new Cisco DevNet Expert Lab exam. Because of its popularity, bugs get fixed quickly, new features are carefully documented, and useful knowledge is widely shared among its users.
• Third-party KVM images integration: CML permits users to upload third-party VM images, such as Windows VM, that are not bundled by default.
• Others: The CML tool offers many other features, such as dashboard list view, multiuser grouping, Ansible integration, and pyATS integration.

We will not use all of the CML features in this book, but it is nice to know the tool is so feature-rich and is constantly being updated. Again, I want to stress the importance of having a lab to follow along for the book examples but it does not need to be Cisco CML. The code examples provided in this book should work across any lab device, as long as it runs the same software type and version.

CML tips

The CML website (https://developer.cisco.com/modeling-labs/) and documentation (https://developer.cisco.com/docs/modeling-labs/) offer lots of guidance and information, from installation to usage. The lab topology will be included in the respective chapters in the book’s GitHub repository (https://github.com/PacktPublishing/Mastering-Python-Networking-Fourth-Edition). The lab images can be directly imported to the lab via the Import button:

Figure 2.1: CML Console Image Lab Image

For the labs, each of the devices will have its management interface connected to an unmanaged switch, which in turn connects to an external connection for access:

Figure 2.2: Unmanaged Switch for Management Interface Access

You will need to change the IP address of the management interface to fit your own lab’s schema. For example, in the 2_DC_Topology.yaml file in Chapter 2, the IP address of lax-edg-r1 GigabitEthernet0/0 0 is 192.168.2.51. You will need to change this IP address according to your own lab.

If you are using virtual lab software other than CML, you can open the topology file with any text editor (such as Sublime Text, shown below) and see each of the devices’ configurations. You can then copy and paste the configuration into your own lab devices:

Figure 2.3: Topology File Viewed with Text Editor

We talked about Cisco DevNet briefly earlier in this section. Let us explore more about DevNet in the next section.

Cisco DevNet

Cisco DevNet (https://developer.cisco.com/site/devnet/) is the premier, all-in-one website when it comes to network automation resources at Cisco. It is free to sign up and provides free remote labs, free video courses, guided learning tracks, documentation, and much more.

The Cisco DevNet Sandbox (https://developer.cisco.com/site/sandbox/) is a great alternative if you do not already have a lab at your own disposal or want to try out new technologies. Some of the labs are always on, while others you need to reserve. The lab availability will depend on usage.

GNS3 and others

There are a few other virtual labs that I have used and would recommend. GNS3 is one of them:

Figure 2.5: GNS3 Website

As mentioned previously, GNS3 is what a lot of us use to study for certification tests and to practice for labs. The tool has really grown up from the early days of being the simple frontend for Dynamips into a viable commercial product. GNS3 is vendor-neutral, which can be helpful if we want to build a multi-vendor lab. This is typically done either by making a clone of the image (such as Arista vEOS) or by directly launching the network device image via other hypervisors (such as KVM).

Another multi-vendor network emulation environment that has gotten a lot of great reviews is the Emulated Virtual Environment Next Generation (Eve-NG): http://www.eve-ng.net/. I personally do not have enough experience with the tool, but many of my colleagues and friends in the industry use it for their network labs. If you are familiar with containers, containerlab (https://containerlab.dev/) can also be an alternative for you.

There are also other standalone virtualized platforms, such as Arista vEOS (https://www.arista.com/en/cg-veos-router/veos-router-overview), Juniper vMX (https://www.juniper.net/us/en/products/routers/mx-series/vmx-virtual-router-software.html), and Nokia SR-Linux (https://www.nokia.com/networks/data-center/service-router-linux-NOS/), which you can use as standalone virtual appliances during testing.

They are great complementary tools for testing platform-specific features. Many of them are offered as paid products on public cloud provider marketplaces for easier access.

Now that we have built our network lab, we can start to experiment with Python libraries that can help with management and automation. We will begin with enabling the Python virtual environment. Then we will install and use the Pexpect library for some examples.

Let us start by using the Python virtual environment. The Python virtual environment allows us to manage separate package installations for different projects by creating a “virtual” isolated Python installation and installing packages into that virtual installation. By using a virtual environment, we do not need to worry about breaking the packages installed globally or from other virtual environments. We will start by installing the python3.10-venv package, then create the virtual environment itself:

$ sudo apt update
$ sudo apt install python3.10-venv
$ python3 -m venv venv 
$ source venv/bin/activate
(venv) $
(venv) $ deactivate

From the output, we saw that we use the venv module from our installation, create a virtual environment called “venv” and then activate it. While the virtual environment is activated, you will see the (venv) label in front of your hostname, indicating that you are in that virtual environment. When finished, you can use the deactivate command to exit the virtual environment. If interested, you can learn more about Python virtual environments here: https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#installing-virtualenv.

Once we have activated the virtual environment, we can move on to install the Pexpect library.

Pexpect is a pure Python module for spawning child applications, controlling them, and responding to expected patterns in their output. Pexpect works like Don Libes’ Expect. Pexpect allows our script to spawn a child application and control it as if a human were typing commands; more information can be found on Pexpect’s documentation page: https://pexpect.readthedocs.io/en/stable/index.html.

Similar to the original Tool Command Language (TCL) Expect module by Don Libes, Pexpect launches, or spawns, another process and watches over it in order to control the interaction. The Expect tool was originally developed to automate interactive processes such as FTP, Telnet, and rlogin, and was later expanded to include network automation. Unlike the original Expect, Pexpect is entirely written in Python, which does not require TCL or C extensions to be compiled. This allows us to use the familiar Python syntax and its rich standard library in our code.

Pexpect installation

The Pexpect installation process is straightforward:

(venv) $ pip install pexpect

Let’s do a quick test to make sure the package is usable; make sure we start the Python interactive shell from the virtual environment:

(venv) $ python
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pexpect
>>> dir(pexpect)
['EOF', 'ExceptionPexpect', 'Expecter', 'PY3', 'TIMEOUT', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__revision__', '__spec__', '__version__', 'exceptions', 'expect', 'is_executable_file', 'pty_spawn', 'run', 'runu', 'searcher_re', 'searcher_string', 'spawn', 'spawnbase',
 'spawnu', 'split_command_line', 'sys', 'utils', 'which']
    >>> exit()

Pexpect overview

For this chapter, we will use the 2_DC_Topology and work on the two IOSv devices, lax-edg-r1 and lax-edg-r2:

Figure 2.6: lax-edg-r1 and lax-edg-r2

The devices will each have a management address in the 192.16.2.x/24 range. In the example, lax-edg-r1 will have 192.168.2.51 and lax-edg-r2 will have 192.168.2.52 as the management IP. If this is the first time the device is powered up, it will need to generate an RSA key for SSH:

lax-edg-r2(config)#crypto key generate rsa

For older IOSv software images, we might also need to add the following lines to the ssh configuration (~/.ssh/config) depending on your platform:

Host 192.168.2.51
  HostKeyAlgorithms +ssh-rsa
  KexAlgorithms +diffie-hellman-group-exchange-sha1
Host 192.168.2.52
  HostKeyAlgorithms +ssh-rsa
  KexAlgorithms +diffie-hellman-group-exchange-sha1

With the devices ready, let’s take a look at how you would interact with the router if you were to telnet into the device:

(venv) $ $ telnet 192.168.2.51
Trying 192.168.2.51...
Connected to 192.168.2.51.
Escape character is '^]'.
<skip>
User Access Verification
Username: cisco
Password:

The device configuration uses the username of cisco, and the password is also cisco. Notice that the user is already in the privileged mode because of the privilege assigned in the configuration:

lax-edg-r1#sh run | i cisco
enable password cisco
username cisco privilege 15 secret 5 $1$SXY7$Hk6z8OmtloIzFpyw6as2G.
 password cisco
 password cisco

The auto-config also generated vty access for both telnet and SSH:

line con 0
 password cisco
line aux 0
line vty 0 4
 exec-timeout 720 0
 password cisco
 login local
 transport input telnet ssh

Let’s see a Pexpect example using the Python interactive shell:

>>> import pexpect
>>> child = pexpect.spawn('telnet 192.168.2.51')
>>> child.expect('Username')
0
>>> child.sendline('cisco')
6
>>> child.expect('Password')
0
>>> child.sendline('cisco')
6
>>> child.expect('lax-edg-r1#')
0
>>> child.sendline('show version | i V')
19
>>> child.before
b": \r\n************************************************************************\r\n* IOSv is strictly limited to use for evaluation, demonstration and IOS  *\r\n* education. IOSv is provided as-is and is not supported by Cisco's      *\r\n* Technical Advisory Center. Any use or disclosure, in whole or in part, *\r\n* of the IOSv Software or Documentation to any third party for any       *\r\n* purposes is expressly prohibited except as otherwise authorized by     *\r\n* Cisco in writing.                                                      *\r\n***********************************************************************\r\n"
>>> child.sendline('exit')
5
>>> exit()

In the previous interactive example, Pexpect spawns off a child process and watches over it in an interactive fashion. There are two important methods shown in the example, expect() and sendline(). The expect() line indicates the string in the Pexpect process looks for when the returned string is considered done. This is the expected pattern. In our example, we knew the router had sent us all the information when the hostname prompt (lax-edg-r1#) was returned. The sendline() method indicates which words should be sent to the remote device as the command. There is also a method called send(), but sendline() includes a linefeed, which is similar to pressing the Enter key at the end of the words you sent in the previous telnet session. From the router’s perspective, it is just as if someone typed in the text from a Terminal. In other words, we are tricking the routers into thinking they are interfacing with a human being when they are actually communicating with a computer.

The before and after properties will be set to the text printed by the child application. The before properties will be set to the text printed by the child application up to the expected pattern. The after string will contain the text that was matched by the expected pattern. In our case, the before text will be set to the output between the two expected matches (lax-edg-r1#), including the show version command. The after text is the router hostname prompt:

>>> child.sendline('show version | i V')
19
>>> child.expect('lax-edg-r1#')
0
>>> child.before
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.6(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 9Y0KJ2ZL98EQQVUED5T2Q\r\n'
>>> child.after
b'iosv-1#'

If you are wondering about the b' in front of the return, it is a Python byte string (https://docs.python.org/3.10/library/stdtypes.html).

What would happen if you expected the wrong term? For example, if we typed in username with the lowercase “u” instead of Username after spawning the child application, the Pexpect process would look for a string of username from the child process. In that case, the Pexpect process would just hang because the word username would never be returned by the router. The session would eventually time out, or we could manually exit out via Ctrl + C.

The expect() method waits for the child application to return a given string, so in the previous example, if you wanted to accommodate both lowercase and uppercase u, you could use the following term:

>>> child.expect('[Uu]sername')

The square bracket serves as an or operation that tells the child application to expect a lowercase or uppercase “u” followed by sername as the string. What we are telling the process is that we will accept either Username or username as the expected string. For more information on these different types of matching using a regular expression, go to: https://docs.python.org/3.10/library/re.html.

The expect() method can also contain a list of options instead of just a single string; these options can also be regular expressions themselves. Going back to the previous example, we can use the following list of options to accommodate the two different possible strings:

>>> child.expect(['Username', 'username'])

Generally speaking, use the regular expression for a single expect string when we can fit the different letters in a regular expression, whereas use the possible options if we need to catch completely different responses from the device, such as a password rejection. For example, if we use several different passwords for our login, we want to catch % Login invalid as well as the device prompt.

One important difference between Pexpect regular expressions and Python regular expressions is that Pexpect matching is non-greedy, which means they will match as little as possible when using special characters. Because Pexpect performs regular expressions on a stream, it cannot look ahead, as the child process generating the stream may not be finished. This means the special dollar sign character $ typically matching the end of the line is useless because .+ will always return no characters, and the .* pattern will match as little as possible. In general, just keep this in mind and be as specific as you can be on the expect match strings.

Let’s consider the following scenario:

>>> child.sendline('show run | i hostname')
22
>>> child.expect('lax-edg-r1')
0
>>> child.before
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.6(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 9Y0KJ2ZL98EQQVUED5T2Q\r\n'
>>>

Hmm... Something is not quite right here. Compare it to the Terminal output before; the output you expect would be hostname lax-edg-r1:

iosv-1#sh run | i hostname
hostname lax-edg-r1

Taking a closer look at the expected string will reveal the mistake. In this case, we were missing the hash (#) sign behind the lax-edg-r1 hostname. Therefore, the child application treated the second part of the return string as the expected string:

>>> child.sendline('show run | i hostname')
22
>>> child.expect('lax-edg-r1#')
0
>>> child.before
b'#show run | i hostname\r\nhostname lax-edg-r1\r\n'

You can see a pattern emerging from the usage of Pexpect after a few examples. The user maps out the sequence of interactions between the Pexpect process and the child application. With some Python variables and loops, we can start to construct a useful program that will help us gather information and make changes to network devices.

Our first Pexpect program

Our first program, chapter2_1.py, extends what we did in the last section with some additional code:

#!/usr/bin/env python
import pexpect
devices = {'iosv-1': {'prompt': 'lax-edg-r1#', 'ip': '192.168.2.51'},
           'iosv-2': {'prompt': 'lax-edg-r2#', 'ip': '192.168.2.52'}}
username = 'cisco'
password = 'cisco'
for device in devices.keys():
    device_prompt = devices[device]['prompt']
    child = pexpect.spawn('telnet ' + devices[device]['ip'])
    child.expect('Username:')
    child.sendline(username)
    child.expect('Password:')
    child.sendline(password)
    child.expect(device_prompt)
    child.sendline('show version | i V')
    child.expect(device_prompt)
    print(child.before)
    child.sendline('exit')

We used a nested dictionary in line 5:

devices = {'iosv-1': {'prompt': 'lax-edg-r1#', 'ip': '192.168.2.51'},
           'iosv-2': {'prompt': 'lax-edg-r2#', 'ip': '192.168.2.52'}}

The nested dictionary allows us to refer to the same device (such as lax-edg-r1) with the appropriate IP address and prompt symbol. We can then use those values for the expect() method later on in the loop.

The output prints out the show version | i V output on the screen for each of the devices:

$ python chapter2_1.py 
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 98U40DKV403INHIULHYHB\r\n'
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\n'

Now that we have seen a basic example of Pexpect, let us go deeper into more features of the library.

More Pexpect Features

In this section, we will look at more Pexpect features that might come in handy when certain situations arise.

If you have a slow or fast link to your remote device, the default expect() method timeout is 30 seconds, which can be increased or decreased via the timeout argument:

>>> child.expect('Username', timeout=5)

You can choose to pass the command back to the user using the interact() method. This is useful when you just want to automate certain parts of the initial task:

>>> child.sendline('show version | i V')
19
>>> child.expect('lax-edg-r1#')
0
>>> child.before
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-
-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 98U40DKV403INHIULHYHB\r\n'
>>> child.interact()
show version | i V
Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)
Processor board ID 98U40DKV403INHIULHYHB
lax-edg-r1#exit
Connection closed by foreign host.
>>>

You can get a lot of information about the child.spawn object by printing it out in string format:

>>> str(child)
"<pexpect.pty_spawn.spawn object at 0x7f068a9bf370>\ncommand: /usr/bin/telnet\nargs: ['/usr/bin/telnet', '192.168.2.51']\nbuffer (last 100 chars): b''\nbefore (last 100 chars): b'TERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\\r\\nProcessor board ID 98U40DKV403INHIULHYHB\\r\\n'\nafter: b'lax-edg-r1#'\nmatch: <re.Match object; span=(165, 176), match=b'lax-edg-r1#'>\nmatch_index: 0\nexitstatus: 1\nflag_eof: False\npid: 25510\nchild_fd: 5\nclosed: False\ntimeout: 30\ndelimiter: <class 'pexpect.exceptions.EOF'>\nlogfile: None\nlogfile_read: None\nlogfile_send: None\nmaxread: 2000\nignorecase: False\nsearchwindowsize: None\ndelaybeforesend: 0.05\ndelayafterclose: 0.1\ndelayafterterminate: 0.1"
>>>

The most useful debug tool for Pexpect is to log the output in a file:

>>> child = pexpect.spawn('telnet 192.168.2.51')
>>> child.logfile = open('debug', 'wb')

For more information on Pexpect features, check out: https://pexpect.readthedocs.io/en/stable/api/index.html

We have been working with Telnet so far in our examples, which leaves our communication in clear text during the session. In modern networks, we typically use secure shell (SSH) for management. In the next section, we will take a look at Pexpect with SSH.

Pexpect and SSH

Pexpect has a subclass called pxssh, which specializes in setting up SSH connections. The class adds methods for login, logout, and various tricky things to handle the different situations in the ssh login process. The procedures are mostly the same, with the exception of login() and logout():

>>> from pexpect import pxssh
>>> child = pxssh.pxssh()
>>> child.login('192.168.2.51', 'cisco', 'cisco', auto_prompt_reset=False) 
True
>>> child.sendline('show version | i V')
19
>>> child.expect('lax-edg-r1#')
0
>>> child.before
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 98U40DKV403INHIULHYHB\r\n'
>>> child.logout()
>>>

Notice the auto_prompt_reset=False argument in the login() method. By default, pxssh uses the shell prompt to synchronize the output. But since it uses the PS1 option for most of bash-shell or c-shell, they will error out on Cisco or other network devices.

Pexpect complete example

As the final step, let’s put everything you have learned so far about Pexpect into a script. Putting code into a script makes it easier to use in a production environment, as well as easier to share with your colleagues. We will write our second script, chapter2_2.py:

#!/usr/bin/env python
import getpass
from pexpect import pxssh
devices = {'lax-edg-r1': {'prompt': 'lax-edg-r1#', 'ip': '192.168.2.51'},
           'lax-edg-r2': {'prompt': 'lax-edg-r2#', 'ip': '192.168.2.52'}}
commands = ['term length 0', 'show version', 'show run']
username = input('Username: ')
password = getpass.getpass('Password: ')
# Starts the loop for devices
for device in devices.keys():
    outputFileName = device + '_output.txt'
    device_prompt = devices[device]['prompt']
    child = pxssh.pxssh()
    child.login(devices[device]['ip'], username.strip(), password.strip(), auto_prompt_reset=False)
    # Starts the loop for commands and write to output
    with open(outputFileName, 'wb') as f:
        for command in commands:
            child.sendline(command)
            child.expect(device_prompt)
            f.write(child.before)
    child.logout()

The script further expands from our first Pexpect program with the following additional features:

• It uses SSH instead of Telnet.
• It supports multiple commands instead of just one by making the commands into a list (line 8) and loops through the commands (starting at line 20).
• It prompts the user for their username and password instead of hardcoding them in the script for better security posture.
• It writes the output in two files, lax-edg-r1_output.txt and lax-edg-r2_output.txt.

After the code is executed, we should see the two output files in the same directory. Besides Pexpect, Paramiko is another popular Python library used to handle interactive sessions.

Paramiko is a Python implementation of the SSHv2 protocol. Just like the pxssh subclass of Pexpect, Paramiko simplifies the SSHv2 interaction between the host and the remote device. Unlike pxssh, Paramiko focuses only on SSHv2 with no Telnet support. It also provides both client and server operations.

Paramiko is the low-level SSH client behind the high-level automation framework Ansible for its network modules. We will cover Ansible in Chapter 4, The Python Automation Framework – Ansible. Let’s take a look at the Paramiko library.

Installation of Paramiko

Installing Paramiko is pretty straightforward with Python pip. However, there is a hard dependency on the cryptography library. The library provides low-level, C-based encryption algorithms for the SSH protocol.

The installation instruction for Windows, macOS, and other flavors of Linux can be found at: https://cryptography.io/en/latest/installation/.

We will show the Paramiko installation steps for our Ubuntu 22.04 virtual machine:

sudo apt-get install build-essential libssl-dev libffi-dev python3-dev 
pip install cryptography
pip install paramiko

Let us test the library’s usage by importing it with the Python interpreter:

$ python
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> exit()

Now we are ready to take a look at Paramiko in the next section.

Paramiko overview

Let’s look at a quick Paramiko example using the Python 3 interactive shell:

>>> import paramiko, time
>>> connection = paramiko.SSHClient()
>>> connection.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> connection.connect('192.168.2.51', username='cisco', password='cisco', look_for_keys=False, allow_agent=False)
>>> new_connection = connection.invoke_shell()
>>> output = new_connection.recv(5000)
>>> print(output) b"\r\n*************************************************************************\
r\n* IOSv is strictly limited to use for evaluation, demonstration and IOS  *\r\n* education. IOSv is provided as-is and is not supported by Cisco's      *\r\n* Technical Advisory Center. Any use or disclosure, in whole or in part, *\r\n* of the IOSv Software or Documentation to any third party for any       *\r\n* purposes is expressly prohibited except as otherwise authorized by     *\r\n* Cisco in writing.                                                      *\r\n***********************************************************************\r\nlax-edg-r1#"
>>> new_connection.send("show version | i V\n")
19
>>> time.sleep(3)
>>> output = new_connection.recv(5000)
>>> print(output)
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 98U40DKV403INHIULHYHB\r\nlax-edg-r1#'
>>> new_connection.close()
>>>

The time.sleep() function inserts a time delay to ensure all the outputs were captured. This is particularly useful on a slower network connection or a busy device. This command is not required but is recommended depending on your situation.

Even if we are seeing the Paramiko operation for the first time, the beauty of Python and its clear syntax means that we can make a pretty good educated guess at what the program is trying to do:

>>> import paramiko
>>> connection = paramiko.SSHClient()
>>> connection.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> connection.connect('192.168.2.51', username='cisco', password='cisco',
look_for_keys=False, allow_agent=False)

The first four lines create an instance of the SSHClient class from Paramiko. The next line sets the policy that the client should use regarding keys; in this case, lax-edg-r1 might not be in either the system host keys or the application’s keys. In our scenario, we will automatically add the key to the application’s HostKeys object. At this point, if you log on to the router, you will see all the login sessions from Paramiko.

The next few lines invoke a new interactive shell from the connection and a repeatable pattern of sending a command and retrieving the output. Finally, we close the connection.

Some readers who have used Paramiko before might be familiar with the exec_command() method instead of invoking a shell. Why do we need to invoke an interactive shell instead of using exec_command() directly? Unfortunately, exec_command() on Cisco IOS only allows a single command. Consider the following example with exec_command() for the connection:

>>> connection.connect('192.168.2.51', username='cisco', password='cisco', look_for_keys=False, allow_agent=False)
>>> stdin, stdout, stderr = connection.exec_command('show version | i V\n')
>>> stdout.read()
b'Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)rnProcessor board ID 98U40DKV403INHIULHYHBrn'
>>>

Everything works great; however, if you look at the number of sessions on the Cisco device, you will notice that the connection is dropped by the Cisco device without you closing the connection. Because the SSH session is no longer active, exec_command() will return an error if you want to send more commands to the remote device:

>>> stdin, stdout, stderr = connection.exec_command('show version | i V\n') 
Traceback (most recent call last):
<skip>
raise SSHException('SSH session not active') paramiko.ssh_exception.SSHException: SSH session not active
>>>

In the previous example, the new_connection.recv() command displayed what was in the buffer and implicitly cleared it out for us. What would happen if you did not clear out the received buffer? The output would just keep on filling up the buffer and would overwrite it:

>>> new_connection.send("show version | i V\n")
 19
>>> new_connection.send("show version | i V\n") 
19
>>> new_connection.send("show version | i V\n") 
19
>>> new_connection.recv(5000)
b'show version | i VrnCisco IOS Software, IOSv Software (VIOS- ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)rnProcessor
board ID 98U40DKV403INHIULHYHBrnlax-edg-r1#show version | i VrnCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)rnProcessor board ID 98U40DKV403INHIULHYHBrnlax-edg-r1#show version | i VrnCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)rnProcessor board ID 98U40DKV403INHIULHYHBrnlax-edg-r1#'
>>>

For consistency of the deterministic output, we will retrieve the output from the buffer each time we execute a command.

First Paramiko program

Our first program will use the same general structure as the Pexpect program we have put together. We will loop over a list of devices and commands while using Paramiko instead of Pexpect. This will give us a good compare and contrast of the differences between Paramiko and Pexpect.

If you have not done so already, you can download the code, chapter2_3.py, from the book’s GitHub repository at https://github.com/PacktPublishing/Mastering-Python-Networking-Fourth-Edition. I will list the notable differences here:

devices = {'lax-edg-r1': {'ip': '192.168.2.51'},
           'lax-edg-r2': {'ip': '192.168.2.52'}}

We no longer need to match the device prompt using Paramiko; therefore, the device dictionary can be simplified:

commands = ['show version', 'show run']

There is no sendline equivalent in Paramiko; instead, we manually include the newline break in each of the commands:

def clear_buffer(connection):
    if connection.recv_ready():
        return connection.recv(max_buffer)

We include a new method to clear the buffer for sending commands, such as terminal length 0 or enable, because we do not need the output for those commands. We simply want to clear the buffer and get to the execution prompt. This function will later be used in the loop, such as in line 25 of the script:

output = clear_buffer(new_connection)

The rest of the program should be pretty self-explanatory, similar to what we have seen in this chapter. The last thing I would like to point out is that since this is an interactive program, we place a buffer and wait for the command to be finished on the remote device before retrieving the output:

time.sleep(5)

After we clear the buffer, we will wait five seconds between the execution of commands. This will give the device adequate time to respond if it is busy.

More Paramiko features

We will look at Paramiko a bit later in Chapter 4, The Python Automation Framework – Ansible, when we discuss Ansible, as Paramiko is the underlying transport for many of the network modules. In this section, we will take a look at some of the other features of Paramiko.

Paramiko for servers

Paramiko can be used to manage servers through SSHv2 as well. Let’s look at an example of how we can use Paramiko to manage servers. We will use key-based authentication for the SSHv2 session.

In this example, I used another Ubuntu virtual machine on the same hypervisor as the destination server. You can also use a server on the CML simulator or an instance in one of the public cloud providers, such as Amazon AWS EC2.

We will generate a public-private key pair for our Paramiko host:

ssh-keygen -t rsa

This command, by default, will generate a public key named id_rsa.pub, as the public key under the user home directory ~/.ssh along with a private key named id_rsa. Treat the private key with the same attention as you would for private passwords that you do not want to share with anybody else.

You can think of the public key as a business card that identifies who you are. Using the private and public keys, the message will be encrypted by your private key locally and decrypted by the remote host using the public key. We should copy the public key to the remote host. In production, we can do this via out-of-band using a USB drive; in our lab, we can simply copy the public key to the remote host’s ~/.ssh/authorized_keys file. Open up a Terminal window for the remote server so you can paste in the public key.

Copy the content of ~/.ssh/id_rsa.pub on your management host with Paramiko:

$ cat ~/.ssh/id_rsa.pub 
ssh-rsa <your public key>

Then, paste it to the remote host under the user directory; in this case, I am using echou for both sides:

<Remote Host>$ vim ~/.ssh/authorized_keys
ssh-rsa <your public key>

You are now ready to use Paramiko to manage the remote host. Notice in this example that we will use the private key for authentication as well as the exec_command() method for sending commands:

>>> import paramiko
>>> key = paramiko.RSAKey.from_private_key_file('/home/echou/.ssh/id_rsa')
>>> client = paramiko.SSHClient()
>>> client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> client.connect('192.168.199.182', username='echou', pkey=key)
>>> stdin, stdout, stderr = client.exec_command('ls -l')
>>> stdout.read()
b'total 44ndrwxr-xr-x 2 echou echou 4096 Jan 7 10:14 Desktopndrwxr-xr-x 2
echou echou 4096 Jan 7 10:14 Documentsndrwxr-xr-x 2 echou echou 4096 Jan 7
10:14 Downloadsn-rw-r--r-- 1 echou echou 8980 Jan 7 10:03
examples.desktopndrwxr-xr-x 2 echou echou 4096 Jan 7 10:14 Musicndrwxr-xr-x
echou echou 4096 Jan 7 10:14 Picturesndrwxr-xr-x 2 echou echou 4096 Jan 7 10:14 Publicndrwxr-xr-x 2 echou echou 4096 Jan 7 10:14 Templatesndrwxr-xr-x
2 echou echou 4096 Jan 7 10:14 Videosn'
>>> stdin, stdout, stderr = client.exec_command('pwd')
>>> stdout.read()
b'/home/echou'
>>> client.close()
>>>

Notice that in the server example, we do not need to create an interactive session to execute multiple commands. You can now turn off password-based authentication in your remote host’s SSHv2 configuration for more secure key-based authentication with automation enabled.

Why do we want to know about using private keys as authentication methods? More and more network devices, such as Cumulus and Vyatta switches, are moving toward using Linux shell and public-private key authentication as a security mechanism. For some operations, we will use a combination of SSH session and key-based authentication for authentication.

More Paramiko examples

In this section, let’s make the Paramiko program more reusable. There is one downside of our existing script: we need to open up the script every time we want to add or delete a host, or whenever we need to change the commands we want to execute on the remote host.

This is due to the fact that both the host and command information are statically entered inside the script. Hardcoding the host and command has a higher chance of making mistakes when making a change. By making both the host and command files read in as parameters for the script, we can make the script more flexible. Users (and future us) can simply modify these text files when you need to make host or command changes.

We have incorporated the change in the script named chapter2_4.py.

Instead of hardcoding the commands, we broke the commands into a separate commands.txt file. Up to this point, we have been using show commands; in this example, we will make configuration changes. In particular, we will change the logging buffer size to 30000 bytes:

$ cat commands.txt 
config t
logging buffered 30000 
end
copy run start

The device’s information is written into a devices.json file. We chose JSON format for the device’s information because JSON data types can be easily translated into Python dictionary data types:

$ cat devices.json 
{
    "lax-edg-r1": {
        "ip": "192.168.2.51"
    },
    "lax-edg-r2": {
        "ip": "192.168.2.52"
    }
}

In the script, we made the following changes:

with open('devices.json', 'r') as f:
    devices = json.load(f)
with open('commands.txt', 'r') as f:
    commands = f.readlines()

Here is an abbreviated output from the script execution:

$ python chapter2_4.py 
Username: cisco
Password: 
b'terminal length 0\r\nlax-edg-r1#config t\r\nEnter configuration commands, one per line.  End with CNTL/Z.\r\nlax-edg-r1(config)#'
b'logging buffered 30000\r\nlax-edg-r1(config)#'
b'end\r\nlax-edg-r1#'
b'copy run start'
<skip>

Do a quick check to make sure the change has taken place in both running-config and startup-config:

lax-edg-r1#sh run | i logging
logging buffered 30000

The Paramiko library is a general-purpose library intended for working with interactive command-line programs. For network management, there is another library, Netmiko, a fork from Paramiko, that is purpose-built for network device management. We will take a look at it in the upcoming section.

Paramiko is a great library to do low-level interactions with Cisco IOS and other vendor devices. But as you have noticed from previous examples, we are repeating many of the same steps between lax-edg-r1 and lax-edg-r2 for device login and execution. Once we start to develop more automation commands, we also start to repeat ourselves in capturing terminal outputs and formatting them into a usable format. Wouldn’t it be great if somebody could write a Python library that simplifies these low-level steps and share it with other network engineers?

Ever since 2014, Kirk Byers (https://github.com/ktbyers) has been working on open-source initiatives to simplify the management of network devices. In this section, we will take a look at an example of the Netmiko (https://github.com/ktbyers/netmiko) library that he created.

First, we will install the netmiko library using pip:

(venv) $ pip install netmiko

We can use the example published on Kirk’s website, https://pynet.twb-tech.com/blog/automation/netmiko.html, and apply it to our labs. We will start by importing the library and its ConnectHandler class. Then we will define our device parameter as a Python dictionary and pass it to the ConnectHandler. Notice that we are defining a device_type of cisco_ios in the device parameter:

>>> from netmiko import ConnectHandler
>>> net_connect = ConnectHandler(
...     device_type="cisco_ios",
...     host="192.168.2.51",
...     username="cisco",
...     password="cisco",
... )

This is where the simplification begins. Notice that the library automatically determines the device prompt as well as formatting the returned output from the show command:

>>> net_connect.find_prompt()
'lax-edg-r1#'
>>> output = net_connect.send_command('show ip int brief')
>>> print(output)
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         192.168.2.51    YES NVRAM  up                    up      
GigabitEthernet0/1         10.0.0.1        YES NVRAM  up                    up      
Loopback0                  192.168.0.10    YES NVRAM  up                    up    

Let’s see another example for the second Cisco IOS device in our lab and send a configuration command instead of a show command. Note that the command attribute is a list that can contain multiple commands:

>>> net_connect_2 = ConnectHandler(
...     device_type="cisco_ios",
...     host="192.168.2.52",
...     username="cisco",
...     password="cisco",
... )
>>> output = net_connect_2.send_config_set(['logging buffered 19999'])
>>> print(output)
configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
lax-edg-r2(config)#logging buffered 19999
lax-edg-r2(config)#end
lax-edg-r2#
>>> exit()

How cool is that? Netmiko automatically took care of the nitty-gritty stuff for us, allowing us to focus on the commands themselves. The netmiko library is a great time saver and is used by many network engineers. In the next section, we will take a look at the Nornir (https://github.com/nornir-automation/nornir) framework, which also aims to simplify low-level interactions.

Nornir (https://nornir.readthedocs.io/en/latest/) is a pure Python automation framework intended to be used directly from Python. We will start with installing nornir in our environment:

(venv)$ pip install nornir nornir_utils nornir_netmiko

Nornir expects us to define an inventory file, hosts.yaml, consisting of the device information in a YAML format. The information specified in this file is no different than what we have previously defined using the Python dictionary in the Netmiko example:

---
lax-edg-r1:
    hostname: '192.168.2.51'
    port: 22
    username: 'cisco'
    password: 'cisco'
    platform: 'cisco_ios'
lax-edg-r2:
    hostname: '192.168.2.52'
    port: 22
    username: 'cisco'
    password: 'cisco'
    platform: 'cisco_ios'

We can use the netmiko plugin from the nornir library to interact with our device, as illustrated in the chapter2_5.py file:

#!/usr/bin/env python
from nornir import InitNornir
from nornir_utils.plugins.functions import print_result
from nornir_netmiko import netmiko_send_command
nr = InitNornir()
result = nr.run(
    task=netmiko_send_command,
    command_string="show arp"
)
print_result(result)

The execution output is shown as follows:

(venv) $ python chapter2_5.py 
netmiko_send_command************************************************************
* lax-edg-r1 ** changed : False ************************************************
vvvv netmiko_send_command ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.0.0.1                -   5254.001e.e911  ARPA   GigabitEthernet0/1
Internet  10.0.0.2               17   fa16.3e00.0001  ARPA   GigabitEthernet0/1
^^^^ END netmiko_send_command ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* lax-edg-r2 ** changed : False ************************************************
vvvv netmiko_send_command ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.0.128.1             17   fa16.3e00.0002  ARPA   GigabitEthernet0/1
Internet  10.0.128.2              -   5254.0014.e052  ARPA   GigabitEthernet0/1
^^^^ END netmiko_send_command ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We have taken a pretty huge leap forward in this chapter in automating our network using Python. However, some of the methods we have used feel like workarounds for automation. We attempted to trick the remote devices into thinking they were interacting with a human on the other end. Even if we use libraries such as Netmiko or the Nornir framework, the underlying approach remains the same. Just because somebody else has done the work to help abstract the grunt work of the low-level interaction, we are still susceptible to the downsides of dealing with CLI-only devices.

Looking ahead, let us discuss some of the downsides of Pexpect and Paramiko compared to other tools in preparation for our discussion on API-driven methods in the next chapters.

Downsides of Pexpect and Paramiko compared to other tools

The biggest downside of our current method for automating CLI-only devices is that the remote devices do not return structured data. They return data that is ideal for fitting on a terminal to be interpreted by a human, not by a computer program. The human eye can easily interpret a space, while a computer only sees a return character.

We will take a look at a better way in the upcoming chapter. As a prelude to Chapter 3, APIs and Intent-Driven Networking, let’s discuss the idea of idempotency.

Idempotent network device interaction

The term idempotency has different meanings, depending on its context. But in this book’s context, the term means that when a client makes the same call to a remote device, the result should always be the same. I believe we can all agree that this is necessary. Imagine a scenario where each time you execute the same script, you get a different result back. I find that scenario very scary. How can you trust your script if that is the case? It would render our automation effort useless because we need to be prepared to handle different returns.

Since Pexpect and Paramiko are blasting out a series of commands interactively, the chance of having a non-idempotent interaction is higher. Going back to the fact that the return results needed to be screen scraped for useful elements, the risk of difference is much higher. Something on the remote end might have changed between the time we wrote the script and the time when the script is executed for the 100th time. For example, if the vendor makes a screen output change between releases without us updating the script, the script might break our network.

If we need to rely on the script for production, we need the script to be idempotent as much as possible.

Bad automation speeds up bad things

Bad automation allows you to poke yourself in the eye a lot faster, it is as simple as that. Computers are much faster at executing tasks than human engineers. If we had the same set of operating procedures executed by a human versus a script, the script would finish faster than humans, sometimes without the benefit of having a solid feedback loop between steps. The internet is full of horror stories of when someone pressed the Enter key and immediately regretted it.

We need to minimize the chances of bad automation scripts screwing things up. We all make mistakes; carefully testing your script before any production work and having a small blast radius are two keys to making sure you can catch your mistake before it comes back and bites you. No tool or human can eliminate mistakes completely, but we can strive to minimize the mistakes. As we have seen, as great as some of the libraries we have used in this chapter are, the underlying CLI-based method is inherently faulty and error-prone. We will introduce the API-driven method in the next chapter, which addresses some of the CLI-driven management deficiencies.

In this chapter, we covered low-level ways to communicate directly with network devices. Without a way to programmatically communicate and make changes to network devices, there is no automation. We looked at several libraries in Python that allow us to manage devices that were meant to be managed by the CLI. Although useful, it is easy to see how the process can be somewhat fragile. This is mostly due to the fact that the network gear in question was meant to be managed by human beings and not computers.

In Chapter 3, APIs and Intent-Driven Networking, we will look at network devices supporting API and intent-driven networking.

Join our book community

To join our community for this book – where you can share feedback, ask questions to the author, and learn about new releases – follow the QR code below:

https://packt.link/networkautomationcommunity