Before you start reading this book, you should know the basics of Python programming, including its basic syntax, variable types, data types, tuples, lists, dictionaries, functions, strings, and methods.

We will work with Python version 3.10, available at https://www.python.org/downloads.

The examples and source code for this chapter are available in the GitHub repository at https://github.com/PacktPublishing/Python-for-Security-and-Networking.

Check out the following video to see the Code in Action: https://packt.link/Chapter01.

In this section, we will review different types of data structures, including lists, tuples, and dictionaries. We will see methods and operations for managing these data structures and practical examples where we review the main use cases.

Python lists

Lists in Python are equivalent to structures such as dynamic vectors in programming languages such as C and C++. We can express literals by enclosing their elements between a pair of brackets and separating them with commas. The first element of a list has index 0.

Lists in Python are, used to store sets of related items of the same or different types. Also, a list is a mutable data structure which allows the list content can be modified after it has been created.

To create a list in Python, simply enclose a comma-separated sequence of elements in square brackets []. For example, creating a list with response codes would be done as follows:

>>> responses = [200,400,403,500]

Indexes are used to access an element of a list. An index is an integer that indicates the position of an element in a list. The first element of a list always starts at index 0.

>>> responses[0]
200
>>> responses[1]
400

If an attempt is made to access an index that is outside the range of the list, the interpreter will throw the IndexError exception. Similarly, if an index that is not an integer is used, the TypeError exception will be thrown:

>>> responses[4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range

Consider the following example: a programmer can create a list using the append() method by adding objects, printing the objects, and then sorting them before printing again. We describe a list of protocols in the following example, and use the key methods of a Python list, such as add, index, and remove:

>>> protocolList = []
>>> protocolList.append("FTP")
>>> protocolList.append("SSH")
>>> protocolList.append("SMTP")
>>> protocolList.append("HTTP")
>>> print(protocolList)
['FTP','SSH','SMTP','HTTP']
>>> protocolList.sort()
>>> print(protocolList)
['FTP','HTTP','SMTP','SSH']
>>> type(protocolList)
<type 'list'>
>>> len(protocolList)
4

To access specific positions, we can use the index() method, and to delete an element, we can use the remove() method:

>>> position = protocolList.index('SSH')
>>> print("SSH position"+str(position))
SSH position 3
>>> protocolList.remove("SSH")
>>> print(protocolList)
['FTP','HTTP','SMTP']
>>> count = len(protocolList)
>>> print("Protocol elements "+str(count))
Protocol elements 3

To print out the whole protocol list, use the following instructions. This will loop through all the elements and print them:

>>> for protocol in protocolList:
...     print(protocol)
...
FTP
HTTP
SMTP

Lists also provide methods that help manipulate the values within them and allow us to store more than one variable within them and provide a better way to sort object arrays in Python. These are the techniques commonly used to manage lists:

• .append(value): Appends an element at the end of the list
• .count('x'): Gets the number of 'x' elements in the list
• .index('x'): Returns the index of 'x' in the list
• .insert('y','x'): Inserts 'x' at location 'y'
• .pop(): Returns the last element and removes it from the list
• .remove('x'): Removes the first 'x' from the list
• .reverse(): Reverses the elements in the list
• .sort(): Sorts the list in ascending order

The indexing operator allows access to an element and is expressed syntactically by adding its index in brackets to the list, list [index]. You can change the value of a chosen element in the list using the index between brackets:

protocolList [4] = 'SSH'
print("New list content: ", protocols)

Also, you can copy the value of a specific position to another position in the list:

protocolList [1] = protocolList [4]
print("New list content:", protocols)

The value inside the brackets that selects one element of the list is called an index, while the operation of selecting an element from the list is known as indexing.

Adding elements to a list

Lists are mutable sequences that can be modified, which means items can be added, updated, or removed. To add one or more elements, we can use the extend() method. Also, we can use the insert() method to insert an element in a specific index location. We can add elements to a list by means of the following methods:

• list.append(value): This method allows an element to be inserted at the end of the list. It takes its argument’s value and puts it at the end of the list that owns the method. The list’s length then increases by one.
• list.extend(values): This method allows inserting many elements at the end of the list.
• list.insert(location, value): The insert() method is a bit smarter since it can add a new element at any place in the list, not just at the end. It takes as arguments first the required location of the element to be inserted and then the element to be inserted.

In the following example we are using these methods to add elements to the response code list.

>>> responses.append(503)
>>> responses
[200, 400, 403, 500, 503]
>>> responses.extend([504,505])
>>> responses
[200, 400, 403, 500, 503, 504, 505]
>>> responses.insert(6,300)
>>> responses
[201, 400, 403, 500, 503, 504, 300, 505]

Reversing a list

Another interesting operation that we perform in lists is the one that offers the possibility of getting elements in a reverse way in the list through the reverse() method:

>>> protocolList.reverse()
>>> print(protocolList)
['SMTP','HTTP','FTP']

Another way to do the same operation is to use the -1 index. This quick and easy technique shows how you can access all the elements of a list in reverse order:

>>> protocolList[::-1]
>>> print(protocolList)
['SMTP','HTTP','FTP']

Searching elements in a list

In this example, we can see the code for finding the location of a given element inside a list. We use the range function to get elements inside protocolList and we compare each element with the element to find. When both elements are equal, we break the loop and return the element. To find out if an element is contained in a list, we can use the membership operator in.

>>> 'HTTPS' in protocolList
False
>>> 'HTTP' in protocolList
True

You can find the following code in the search_element_list.py file:

protocolList = ["FTP", "HTTP", "SNMP", "SSH"]
element_to_find = "SSH"
for i in range(len(protocolList)):
    if element_to_find in protocolList[i]:
        print("Element found at index", i)
        break

Now that you know how to add, reverse, and search for elements in a list, let’s move on to learning about tuples in Python.

Python tuples

Like lists, the tuple class in Python is a data structure that can store elements of different types.

Along with the list and range classes, it is one of the sequence types in Python, with the particularity that they are immutable. This means its content cannot be modified after it has been created.

In general, to create a tuple in Python, you simply define a sequence of elements separated by commas. Indices are used to access an element of a tuple. An index is an integer indicating the position of an element in a tuple. The first element of a tuple always starts at index 0.

>>> tuple=("FTP","SSH","HTTP","SNMP")
>>> tuple[0]
'FTP'

If an attempt is made to access an index that is outside the range of the tuple, the interpreter will throw the IndexError exception. Similarly, if an index that is not an integer is used, the TypeError exception will be thrown:

>>> tuple[5]
Traceback (most recent call last):
  File "<input>", line 1, in <module>
IndexError: tuple index out of range

As with lists and all sequential types, it is permissible to use negative indices to access the elements of a tuple. In this case, the index -1 refers to the last element of the sequence, -2 to the penultimate, and so on:

>>> tuple[-1]
'SNMP'
>>> tuple[-2]
'HTTP'

When trying to modify a tuple, we see how we get an error since tuples are immutable objects:

>>> tuple[0]="FTP"
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Now that you know the basic data structures for working with Python, let’s move on to learning about Python dictionaries in order to organize information in the key-value format.

Python dictionaries

The Python dictionary data structure is probably the most important in the entire language and allows us to associate values with keys. Python’s dict class is a map type that maps keys to values. Unlike sequential types (list, tuple, range, or str), which are indexed by a numeric index, dictionaries are indexed by keys. Among the main features of the dictionaries, we can highlight:

• It is a mutable type, that is, its content can be modified after it has been created.
• It is a type that reserves the order in which key-value pairs are inserted.

In Python there are several ways to create a dictionary. The simplest is to enclose a sequence of comma-separated key:value pairs in curly braces {}. In this example we will define the service name as the key and the port number as the value.

>>> services = {"FTP":21, "SSH":22, "SMTP":25, "HTTP":80}

Another way to create a dictionary is using the dict class:

>>> dict(services)
{'FTP': 21, 'SSH': 22, 'SMTP': 25, 'HTTP': 80}
>>> type(services)
<class 'dict'>

Accessing an element of a dictionary is one of the main operations for which this type of data exists. Access to a value is done by indexing the key. To do this, simply enclose the key in square brackets. If the key does not exist, the KeyError exception will be thrown.

>>> services['FTP']
21

The dict class also offers the get (key[, default value]) method. This method returns the value corresponding to the key used as the first parameter. If the key does not exist, it does not throw any errors, but returns the second argument by default. If this argument is not supplied, the value None is returned.

>>> services.get('SSH')
22

If the key does not exist, it does not throw any errors, but returns the second argument by default.

>>> services.get('gopher', "service not found")
'service not found'

If this argument is not supplied, the value None is returned.

>>> type(services.get('gopher'))
<class 'NoneType'>

Using the update method, we can combine two distinct dictionaries into one. In addition, the update method will merge existing elements if they conflict:

>>> services = {"FTP":21, "SSH":22, "SMTP":25, "HTTP":80}
>>> services2 = {"FTP":21, "SSH":22, "SMTP":25, "LDAP":389}
>>> services.update(services2)
>>> services
{'FTP': 21, 'SSH': 22, 'SMTP': 25, 'HTTP': 80, 'LDAP': 389}

The first value is the key, and the second the key value. We can use any unchangeable value as a key. We can use numbers, sequences, Booleans, or tuples, but not lists or dictionaries, since they are mutable.

The main difference between dictionaries and lists or tuples is that values contained in a dictionary are accessed by their name and not by their index. You may also use this operator to reassign values, as in the lists and tuples:

>>> services["HTTP"] = 8080
>>> services
{'FTP': 21, 'SSH': 22, 'SMTP': 25, 'HTTP': 8080, 'LDAP': 389}

This means that a dictionary is a set of key-value pairs with the following conditions:

• Each key must be unique: That means it is not possible to have more than one key of the same value.
• A key may be a number or a string.
• A dictionary is not a list: A list contains a set of numbered values, while a dictionary holds pairs of values.
• The len() function: This works for dictionaries and returns the number of key-value elements in the dictionary.

The dict class implements three methods, since they return an iterable data type, known as view objects. These objects provide a view of the keys and values of type dict_values contained in the dictionary, and if the dictionary changes, these objects are instantly updated. The methods are as follows:

• items(): Returns a view of (key, value) pairs from the dictionary.
• keys(): Returns a view of the keys in the dictionary.
• values(): Returns a view of the values in the dictionary.

>>> services.items()
dict_items([('FTP', 21), ('SSH', 22), ('SMTP', 25), ('HTTP', 8080), ('LDAP', 389)])
>>> services.keys()
dict_keys(['FTP', 'SSH', 'SMTP', 'HTTP', 'LDAP'])
>>> services.values()
dict_values([21, 22, 25, 8080, 389])

You might want to iterate over a dictionary and extract and display all the key-value pairs with a for loop:

>>> for key,value in services.items():
...     print(key,value)
... 
FTP 21
SSH 22
SMTP 25
HTTP 8080
LDAP 389

The dict class is mutable, so elements can be added, modified, and/or removed after an object of this type has been created. To add a new item to an existing dictionary, use the assignment operator =. To the left of the operator appears the dictionary object with the new key in square brackets [] and to the right the value associated with said key.

>>> services['HTTPS'] = 443
>>> services
{'FTP': 21, 'SSH': 22, 'SMTP': 25, 'HTTP': 8080, 'LDAP': 389, 'HTTPS': 443}

Now that you know the main data structures for working with Python, let’s move on to learning how to structure our Python code with functions and classes.

Remove an item from a dictionary in Python

In Python there are several ways to remove an element from a dictionary. They are the following:

• pop(key [, default value]): If the key is in the dictionary, it removes the element and return its value; if not, it returns the default value. If the default value is not provided and the key is not in the dictionary, the KeyError exception is raised.
• popitem(): Removes the last key:value pair from the dictionary and returns it. If the dictionary is empty, the KeyError exception is raised.
• del d[key]: Deletes the key:value pair. If the key does not exist, the KeyError exception is thrown.
• clear(): Clears all key:value pairs from the dictionary.

In the following instructions we are removing the elements of the services dictionary using the previous methods:

>>> services = {'FTP': 21, 'SSH': 22, 'SMTP': 25, 'HTTP': 8080, 'LDAP': 389, 'HTTPS': 443} 
>>> services.pop('HTTPS')
443
>>> services
{'FTP': 21, 'SSH': 22, 'SMTP': 25, 'HTTP': 8080, 'LDAP': 389}
>>> services.popitem()
('LDAP', 389)
>>> services
{'FTP': 21, 'SSH': 22, 'SMTP': 25, 'HTTP': 8080}
>>> del services['HTTP']
>>> services
{'FTP': 21, 'SSH': 22, 'SMTP': 25}
>>> services.clear()
>>> services
{}

In this section, we will review Python functions, classes, and objects in Python scripts. We will review some examples for declaring and using in our script code.

Python functions

A function is a block of code that performs a specific task when the function is invoked. You can use functions to make your code reusable, better organized, and more readable. Functions can have parameters and return values. There are at least four basic types of functions in Python:

• Built-in functions: These are an integral part of Python. You can see a complete list of Python’s built-in functions at https://docs.python.org/3/library/functions.html.
• Functions that come from pre-installed modules.
• User-defined functions: These are written by developers in their own code, and they use them freely in Python.
• The lambda function: This allows us to create anonymous functions that are built using expressions such as product = lambda x,y : x * y, where lambda is a Python keyword and x and y are the function parameters.

In Python, functions include reusable code-ordered blocks. This allows a developer to write a block of code to perform a single action. Although Python offers several built-in features, a developer may build user-defined functionality.

Python functions are defined using the def keyword with the function name, followed by the function parameters. The function’s body is composed of Python statements to be executed. You have the option to return a value to the function caller at the end of the function, or if you do not assign a return value, it will return the None value by default.

For instance, we can define a function that returns True if the item value is found in the dictionary and False otherwise. You can find the following code in the my_function.py file:

def contains(dictionary,item):
    for key,value in dictionary.items():
        if value == item:
            return True
    return False 
dictionary = {1:100,2:200,3:300}
print(contains(dictionary,200))
print(contains(dictionary,300))
print(contains(dictionary,350))

Two important factors make parameters special:

• Parameters only exist within the functions in which they were described, and the only place where the parameter can be specified is in the space between a pair of parentheses in the def state.
• Assigning a value to the parameter is done at the time of the function’s invocation by specifying the corresponding argument.

Python classes

Python is an object-oriented language that allows you to create classes from descriptions and instantiate them. The functions specified inside the class are instance methods, also known as member functions.

Python’s way of constructing objects is via the class keyword. A Python object is an assembly of methods, variables, and properties. Lots of objects can be generated with the same class description. Here is a simple example of a protocol object definition.

You can find the following code in the protocol.py file:

class Protocol(object):
    def __init__(self, name, number,description):
        self.name = name
        self.number = number
        self.description = description
    def getProtocolInfo(self):
        return self.name+ " "+str(self.number)+ " "+self.description

The init method is a special method that acts as a constructor method to perform the necessary initialization operation. The method’s first parameter is a special keyword, and we use the self-identifier for the current object reference. The self keyword is a reference to the object itself and provides a way for its attributes and methods to access it.

An object is a set of requirements and qualities assigned to a specific class. Classes form a hierarchy, which means that an object belonging to a specific class belongs to all the superclasses at the same time.

To build an object, write the class name followed by any parameter needed in parentheses. These are the parameters that will be transferred to the init method, which is the process that is called when the class is instantiated:

>>> https_protocol= Protocol("HTTPS", 443, "Hypertext Transfer Protocol Secure")

Now that we have created our object, we can access its attributes and methods through the object.attribute and object.method() syntax:

>>> protocol_http.getProtocolInfo()
HTTPS 443 Hypertext Transfer Protocol Secure

In summary, object programming is the art of defining and expanding classes. A class is a model of a very specific part of reality, reflecting properties and methods found in the real world. The new class may add new properties and new methods, and therefore may be more useful in specific applications.

Python inheritance

In the previous code, we can see a method with the name __init__, which represents the class constructor. If a class has a constructor, it is invoked automatically and implicitly when the object of the class is instantiated. This method allows us to initialize the internal state of an object when we create an object of a class.

Python inheritance is an important concept in object-oriented programming languages. This feature means creating a new class that inherits all the functionality of the parent class and allows the new class to add additional functionality to the base functionality.

In object-oriented terminology, when class “X” is inherited by class “Y”, “X” is called a Super Class or Base Class and “Y” is called a Subclass or Derived Class. One more fact to keep in mind is that only the fields and methods that are not private are accessible by the Derived Class. Private fields and methods are only accessible by the class itself.

Single inheritance occurs when a child class inherits the attributes and methods of a single parent class. The following is an example of simple inheritance in Python where we have a base class and a child class that inherits from the parent class. Note the presence of the __init__ method in both classes , which allows you to initialize the properties of the class as an object constructor.

You can find the following code in the Inheritance_simple.py file.

class BaseClass:
    def __init__(self, property):
        self.property = property
    def message(self):
        print('Welcome to Base Class')
    def message_base_class(self):
        print('This is a message from Base Class')
 
class ChildClass(BaseClass):
    def __init__(self, property):
        BaseClass.__init__(self, property)
    def message(self):
        print('Welcome to ChildClass')
        print('This is inherited from BaseClass')

In our main program we declare two objects, one of each class, and we call the methods defined in each of the classes. Also, taking advantage of the inheritance features, we call the method of the parent class using an object of the child class.

if __name__ == '__main__':
    base_obj = BaseClass('property')
    base_obj.message()
    child_obj = ChildClass('property')
    child_obj.message()
    child_obj.message_base_class()

Two built-in functions, isinstance() and issubclass(), are used to check inheritances. One of the methods that we can use to check if a class is a subclass of another is through the issubclass() method. This method allows us to check if a subclass is a child of a superclass and returns the Boolean True or False depending on the result.

>>> print(issubclass(ChildClass, BaseClass))
>>> True
>>> print(issubclass(BaseClass, ChildClass))
>>> False

In the same way, the isinstance() method allows you to check if an object is an instance of a class. This method returns True if the object is the instance of the class that is passed as the second parameter. The syntax of this special method is isinstance(Object,Class).

>>> print(isinstance(base_obj, BaseClass))
>>> True
>>> print(isinstance(child_obj, ChildClass))
>>> True
>>> print(isinstance(child_obj, BaseClass))
>>> True

Multiple inheritance occurs when a child class inherits attributes and methods from more than one parent class. We could separate both main classes with a comma when creating the secondary class. In the following example we are implementing multiple inheritance where the child class is inheriting from the MainClass and MainClass2 classes.

You can find the following code in the Inheritance_multiple.py file.

class MainClass:
    def message_main(self):
        print('Welcome to Main Class')
class MainClass2:
    def message_main2(self):
        print('Welcome to Main Class2')
class ChildClass(MainClass,MainClass2):
    def message(self):
        print('Welcome to ChildClass')
        print('This is inherited from MainClass and MainClass2')

Our main program creates an object of the Child class, on which we could access both methods of the parent classes.

if __name__ == '__main__':
    child_obj = ChildClass()
    child_obj.message()
    child_obj.message_main()
    child_obj.message_main2()

Python also supports multilevel inheritance, which allows the child class to have inheritance below it. That means the base class is the parent class of all sub-classes and inheritance goes from parent to child. In this way, child classes can access properties and methods from parent classes, but parent classes cannot access the properties of the child class.

In the following example we are implementing multilevel inheritance where the child class is inheriting from the MainClass and we add another level of inheritance with the ChildDerived class, which is inheriting from the Child class. You can find the following code in the Inheritance_multilevel.py file.

class MainClass:
    def message_main(self):
        print('Welcome to Main Class')
class Child(MainClass):
    def message_child(self):
        print('Welcome to Child Class')
        print('This is inherited from Main')
class ChildDerived(Child):
    def message_derived(self):
        print('Welcome to Derived Class')
        print('This is inherited from Child')

In the previous code we first create a main class and then create a child class that is inherited from Main and create another class derived from the child class. We see how the child_derived_obj object is an instance of each of the classes that are part of the hierarchy. In multilevel inheritance, the features of the base class and the derived class are inherited into the new derived class. In our main program we declare a child-derived object and we call the methods defined in each of the classes.

if __name__ == '__main__':
    child_derived_obj = ChildDerived()
    child_derived_obj.message_main()
    child_derived_obj.message_child()
    child_derived_obj.message_derived()
    print(issubclass(ChildDerived, Child))
    print(issubclass(ChildDerived, MainClass))
    print(issubclass(Child, MainClass))
    print(issubclass(MainClass, ChildDerived))
    print(isinstance(child_derived_obj, Child))
    print(isinstance(child_derived_obj, MainClass))
    print(isinstance(child_derived_obj, ChildDerived))

When executing the previous script, we see how from the ChildDerived class we can call the methods from the Child and Main classes. Also, with the issubclass() and isinstance() methods we can check whether the child_derived_obj object is a subclass and instance of the higher classes within the management hierarchy.

Advantages of Python inheritance

One of the main advantages is code reuse, allowing us to establish a relationship between classes, avoiding the need to re-declare certain methods or attributes.

Classes allow us to build objects on top of a collection of abstractly defined attributes and methods. And the ability to inherit will allow us to create larger and more capable child classes by inheriting multiple attributes and methods from others as well as more specific controlling the same for a single class.

The following are some benefits of using inheritance in Python’s object-oriented programming:

• Python inheritance provides code reusability, readability, and scalability.
• Reduce code repetition. You can define all the methods and attributes in the parent class that are accessible by the child classes.
• By dividing the code into multiple classes, identifying bugs in applications is easier.

When working with files it is important to be able to move through the filesystem, determine the type of file, and open a file in the different modes offered by the operating system.

Reading and writing files in Python

Now we are going to review the methods for reading and writing files. These are the methods we can use on a file object for different operations:

• file.open(name_file,mode): Opens a file with a specific mode.
• file.write(string): Writes a string in a file.
• file.read([bufsize]): Reads up to bufsize, the number of bytes from the file. If run without the buffer size option, it will read the entire file.
• file.readline([bufsize]): Reads one line from the file.
• file.close(): Closes the file and destroys the file object.

The open() function is usually used with two parameters (the file with which we are going to work and the access mode) and it returns a file-type object. When opening a file with a certain access mode with the open() function, a file object is returned.

The opening modes can be r (read), w (write), and a (append). We can combine the previous modes with others depending on the file type. We can also use the b (binary), t (text), and + (open reading and writing) modes. For example, you can add a + to your option, which allows read/write operations with the same object:

>>> f = open("file.txt","w")
>>> type(f)
<class '_io.TextIOWrapper'>
>>> f.close()

The following properties of the file object can be accessed:

• closed: Returns True if the file has been closed. Otherwise, False.
• mode: Returns the opening mode.
• name: Returns the name of the file
• encoding: Returns the character encoding of a text file

In the following example, we are using these properties to get information about the file.

You can find the following code in the read_file_properties.py file.

file_descryptor = open("read_file_properties.py", "r+") 
print("Content: "+file_descryptor.read())
print("Name: "+file_descryptor.name)
print("Mode: "+file_descryptor.mode)
print("Encoding: "+str(file_descryptor.encoding))
file_descryptor.close()

When reading a file, the readlines() method reads all the lines of the file and joins them in a list sequence. This method is very useful if you want to read the entire file at once:

>>> allLines = file.readlines()

The alternative is to read the file line by line, for which we can use the readline() method. In this way, we can use the file object as an iterator if we want to read all the lines of a file one by one:

>>> with open("file.txt","r") as file:
...    for line in file:
...        print(line)

In the following example, we are using the readlines() method to process the file and get counts of the lines and characters in this file.

You can find the following code in the count_lines_chars.py file.

try:
    countlines = countchars = 0
    file = open('count_lines_chars.py', 'r')
    lines = file.readlines()
    for line in lines:
        countlines += 1
        for char in line:
            countchars += 1
    file.close()
    print("Characters in file:", countchars)
    print("Lines in file:", countlines)
except IOError as error:
    print("I/O error occurred:", str(error))

If the file we are reading is not available in the same directory, then it will throw an I/O exception with the following error message:

I/O error occurred: [Errno 2] No such file or directory: 'newfile.txt'

Writing text files is possible using the write() method and it expects just one argument that represents a string that will be transferred to an open file. You can find the following code in the write_lines.py file:

try:
    myfile = open('newfile.txt', 'wt') 
    for i in range(10):
        myfile.write("line #" + str(i+1) + "\n")
    myfile.close()
except IOError as error:
    print("I/O error occurred: ", str(error.errno))

In the previous code, we can see how a new file called newfile.txt is created. The open mode wt means that the file is created in write mode and text format.

There are multiple ways to open and create files in Python, but the safest way is by using the with keyword, in which case we are using the Context Manager approach. When we are using the open statement, Python delegates to the developer the responsibility for closing the file, and this practice can provoke errors since developers sometimes forget to close it.

Developers can use the with statement to handle this situation in a safely way. The with statement automatically closes the file even if an exception is raised. Using this approach, we have the advantage that the file is closed automatically, and we don’t need to call the close() method.

You can find the following code in the creating_file.py file:

def main():
        with open('test.txt', 'w') as file:
                file.write("this is a test file")
if __name__ == '__main__':
        main()

The previous code uses the context manager to open a file and returns the file as an object. We then call file.write("this is a test file"), which writes it into the created file. The with statement then handles closing the file for us in this case, so we don’t have to think about it.

At this point we have reviewed the section on working with files in Python. The main advantage of using these methods is that they provide an easy way by which you can automate the process of managing files in the operating system.

In the next section, we’ll review how to manage exceptions in Python scripts. We’ll review the main exceptions we can find in Python for inclusion in our scripts.

Learn and understand exceptions management in Python

Each time your code executes in an unintended way Python stops your program, and it creates a special kind of data, called an exception. An exception or runtime error occurs during program execution. Exceptions are errors that Python detects during execution of the program. If the interpreter experiences an unusual circumstance, such as attempting to divide a number by 0 or attempting to access a file that does not exist, an exception is created or thrown, telling the user that there is a problem.

When an exception is not handled correctly, the execution flow is interrupted, and the console shows the information associated with the exception so that the reader can solve the problem with the information returned by the exception. Exceptions can be handled so that the program does not terminate.

Let’s look at some examples of exceptions:

>>> 4/0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero
>>> a+4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> "4"+4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't convert 'int' object to str implicit

In the previous examples, we can see the exception traceback, which consists of a list of the calls that caused the exception. As we see in the stack trace, the error was caused by executing an operation that is not permitted in Python.

In Python, we can use a try/except block to resolve situations related to exception handling. Now, the program tries to run the division by zero. When the error happens, the exceptions manager captures the error and prints a message that is relevant to the exception:

>>> try:
...     print("10/0=",str(10/0))
... except Exception as exception:
...     print("Error =",str(exception))
... 
Error = division by zero

The try keyword begins a block of the code that may or may not be performing correctly. Next, Python tries to perform some operations; if it fails, an exception is raised, and Python starts to look for a solution.

At this point, the except keyword starts a piece of code that will be executed if anything inside the try block goes wrong – if an exception is raised inside a previous try block, it will fail here, so the code located after the except keyword should provide an adequate reaction to the raised exception. The following code raises an exception related to accessing an element that does not exist in the list:

>>> try:
...     list=[]
...     element=list[0]
... except Exception as exception:
...     print("Exception=",str(exception))
... 
Exception= list index out of range

In the previous code the exception is produced when trying to access the first element of an empty list.

In the following example, we join all these functionalities with exception management when we are working with files. If the file is not found in the filesystem, an exception of the IOError type is thrown, which we can capture thanks to our try..except block. You can find the following code in the read_file_exception.py file:

try:
    file_handle = open("myfile.txt", "r")
except IOError as exception:
    print("Exception IOError: Unable to read from myfile ", exception)
except Exception as exception:
    print("Exception: ", exception)
else:
    print("File read successfully")
    file_handle.close()

In the preceding code, we manage an exception when opening a file in read mode and if the file does not exist it will throw the message "Exception IOError: Unable to read from myfile [Errno 2] No such file or directory: 'myfile.txt'".

Python 3 defines 63 built-in exceptions, and all of them form a tree-shaped hierarchy. Some of the built-in exceptions are more general (they include other exceptions), while others are completely concrete. We can say that the closer to the root an exception is located, the more general (abstract) it is.

Some of the exceptions available by default are listed here (the class from which they are derived is in parentheses):

• BaseException: The class from which all exceptions inherit.
• Exception (BaseException): An exception is a special case of a more general class named BaseException.
• ZeroDivisionError (ArithmeticError): An exception raised when the second argument of a division is 0. This is a special case of a more general exception class named ArithmeticError.
• EnvironmentError (StandardError): This is a parent class of errors related to input/output.
• IOError (EnvironmentError): This is an error in an input/output operation.
• OSError (EnvironmentError): This is an error in a system call.
• ImportError (StandardError): The module or the module element that you wanted to import was not found.

All the built-in Python exceptions form a hierarchy of classes. The following script dumps all predefined exception classes in the form of a tree-like printout.

You can find the following code in the get_exceptions_tree.py file:

def printExceptionsTree(ExceptionClass, level = 0):
    if level > 1:
        print("   |" * (level - 1), end="")
    if level > 0:
        print("   +---", end="")
    print(ExceptionClass.__name__)
    for subclass in ExceptionClass.__subclasses__():
        printExceptionsTree(subclass, level+1)
printExceptionsTree(BaseException)

As a tree is a perfect example of a recursive data structure, a recursion seems to be the best tool to traverse through it. The printExceptionsTree() function takes two arguments:

• A point inside the tree from which we start traversing the tree
• A level to build a simplified drawing of the tree’s branches

This could be a partial output of the previous script:

BaseException
     +---Exception
     |     +---TypeError
     |     +---StopAsyncIteration
     |     +---StopIteration
     |     +---ImportError
     |     |     +---ModuleNotFoundError
     |     |     +---ZipImportError
     |     +---OSError
     |     |     +---ConnectionError
     |     |     |     +---BrokenPipeError
     |     |     |     +---ConnectionAbortedError
     |     |     |     +---ConnectionRefusedError
     |     |     |     +---ConnectionResetError
     |     |     +---BlockingIOError
     |     |     +---ChildProcessError
     |     |     +---FileExistsError
     |     |     +---FileNotFoundError
     |     |     +---IsADirectoryError
     |     |     +---NotADirectoryError
     |     |     +---InterruptedError
     |     |     +---PermissionError
     |     |     +---ProcessLookupError
     |     |     +---TimeoutError
     |     |     +---UnsupportedOperation
     |     |     +---herror
     |     |     +---gaierror
     |     |     +---timeout
     |     |     +---Error
     |     |     |     +---SameFileError
     |     |     +---SpecialFileError
     |     |     +---ExecError
     |     |     +---ReadError

In the output of the previous script, we can see the root of Python’s exception classes is the BaseException class (this is a superclass of all the other exceptions). For each of the encountered classes, it performs the following set of operations:

• Print its name, taken from the __name__ property.
• Iterate through the list of subclasses delivered by the __subclasses__() method, an recursively invoke the printExceptionsTree() function, incrementing the nesting level, respectively.

Now that you know the functions, classes, objects and exceptions for working with Python, let’s move on to learning how to manage modules and packages. Also, we will review the use of some modules for managing parameters, including argparse and optarse.

In this section, you will learn how Python provides modules that are built in an extensible way and offers the possibility to developers to create their own modules.

What is a module in Python?

A module is a collection of functions, classes, and variables that we can use for implementing and application. There is a large collection of modules available with the standard Python distribution. Modules have a dual purpose among which we can highlight:

• Break a program with many lines of code into smaller parts.
• Extract a set of definitions that you use frequently in your programs to be reused. This prevents, for example, having to copy functions from one program to another.

A module can be specified as a file containing definitions and declarations from Python. The file must have a .py extension and its name corresponds to the name of the module. We can start by defining a simple module in a .py file. We’ll define a simple message(name) function inside the my_functions.py file that will print "Hi,{name}.This is my first module".

You can find the following code in the my_functions.py file inside the first_module folder:

def message(name):
    print(f"Hi {name}.This is my first module")

Within our main.py file, we can then import this file as a module and use the message(name) method. You can find the following code in the main.py file:

import my_functions
def main():
    my_functions.message("Python")
if __name__ == '__main__':
    main()

When a module is imported, its content is implicitly executed by Python. You already know that a module can contain instructions and definitions. Usually, the statements are used to initialize the module and are only executed the first time the module name appears in an import statement.

That’s all we need in order to define a very simple Python module within our Python scripts.

How to import modules in Python

To use the definitions of a module in the interpreter or in another module, you must first import it. To do this, the import keyword is used. Once a module has been imported, its definitions can be accessed via the dot . operator.

We can import one or several names of a module as follows. This allows us to directly access the names defined in the module without having to use the dot . operator.

>>> from my_functions import message
>>> message('python')

We can also use the * operator to import all the functions of the module.

>>> from my_functions import *
>>> message('python')

Accessing any element of the imported module is done through the namespace, followed by a dot (.) and the name of the element to be obtained. In Python, a namespace is the name that has been indicated after the word import, that is, the path (namespace) of the module.

It is also possible to abbreviate namespaces by means of an alias. To do this, during the import, the keyword as is assigned followed by the alias with which we will refer to that imported namespace in the future. In this way, we can redefine the name that will be used within a module using the as reserved word:

>>> from my_functions import message as my_message 
>>> my_message('python')
Hi python. This is my first module

Getting information from modules

We can get more information about methods and other entities from a specific module using the dir() method. This method returns a list with all the definitions (variables, functions, classes, …) contained in a module. For example, if we execute this method using the my_functions module we created earlier, we will get the following result:

>>> dir(my_functions)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'message']

The dir() method returns an alphabetically sorted list containing all entities’ names available in the module identified by any name passed to the function as an argument. For example, you can run the following code to print the names of all entities within the sys module. We can obtain the list of built - in modules with the following instructions:

>>> import sys
>>> sys.builtin_module_names
('_abc', '_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 'xxsubtype')
>>> dir(sys)
['__breakpointhook__', '__displayhook__', '__doc__', '__excepthook__', '__interactivehook__', '__loader__', '__name__', '__package__', '__spec__', '__stderr__', '__stdin__', '__stdout__', '__unraisablehook__', '_base_executable', '_clear_type_cache', '_current_frames',...]

The other modules that we can import are saved in files, which are in the paths indicated in the interpreter:

>>> sys.path
['', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']

In the previous code, we are using the dir() method to get all name entities from the sys module.

Difference between a Python module and a Python package

In the same way that we group functions and other definitions into modules, Python packages allow you to organize and structure the different modules that make up a program in a hierarchical way. Also, packages make it possible for multiple modules with the same name to exist and not cause errors.

A package is simply a directory that contains other packages and modules. Also, in Python, for a directory to be considered a package, it must include a module called __init__.py. In most cases, this file will be empty; however, it can be used to initialize package-related code. Among the main differences between a module and a package, we can highlight the following:

• Module: Each of the .py files that we create is called a module. The elements created in a module (functions, classes, …) can be imported to be used in another module. The name we are going to use to import a module is the name of the file.
• Package: A package is a folder that contains .py files and contains a file called __init__.py. This file does not need to contain any instructions. The packages, at the same time, can also contain other sub-packages.

Managing parameters in Python

Often in Python, scripts that are used on the command line as arguments are used to give users options when they run a certain command. To develop this task, one of the options is to use the argparse module, which comes installed by default when you install Python.

One of the interesting choices is that the type of parameter can be indicated using the type attribute. For example, if we want to treat a certain parameter as if it were an integer, then we might do so as follows:

parser.add_argument("-param", dest="param", type="int")

Another thing that could help us to have a more readable code is to declare a class that acts as a global object for the parameters. For example, if we wanted to pass several parameters at the same time to a function, we could use the above mentioned global object, which is the one that contains the global execution parameters.

You can find the following code in the params_global_argparse.py file:

import argparse
class Parameters:
    """Global parameters"""
    def __init__(self, **kwargs):
        self.param1 = kwargs.get("param1")
        self.param2 = kwargs.get("param2")
def view_parameters(input_parameters):
    print(input_parameters.param1)
    print(input_parameters.param2)
parser = argparse.ArgumentParser(description='Testing parameters')
parser.add_argument("-p1", dest="param1", help="parameter1")
parser.add_argument("-p2", dest="param2", help="parameter2")
params = parser.parse_args()
input_parameters = Parameters(param1=params.param1,param2=params.param2)
view_parameters(input_parameters)

In the previous script, we are using the argparse module to obtain parameters and we encapsulate these parameters in an object with the Parameters class.

For more information, you can check out the official website: https://docs.python.org/3/library/argparse.html.

In the following example, we are using the argparse module to manage those parameters that we could use to perform a port scan, such as the IP address, ports, and verbosity level. You can find the following code in the params_port_scanning.py file:

import argparse
if __name__ == "__main__":
    description = """ Uses cases:
        +  Basic scan:
            -target 127.0.0.1
        + Specific port:
            -target 127.0.0.1 -port 21
        + Port list:
            -target 127.0.0.1 -port 21,22
        + Only show open ports
            -target 127.0.0.1 --open True """
    parser = argparse.ArgumentParser(description='Port scanning', epilog=description,
                                     formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("-target", metavar='TARGET', dest="target", help="target to scan",required=True)
    parser.add_argument("-ports", dest="ports", 
                        help="Please, specify the target port(s) separated by comma[80,8080 by default]",
                        default = "80,8080")
    parser.add_argument('-v', dest='verbosity', default=0, action="count",
                        help="verbosity level: -v, -vv, -vvv.")
    parser.add_argument("--open", dest="only_open", action="store_true",
                        help="only display open ports", default=False)

Having set the necessary parameters using the add_argument() method, we could then access the values of these arguments using the parser module’s parse_args() method. Later, we could access the parameters using the params variable.

    params = parser.parse_args()
    print("Target:" + params.target)
    print("Verbosity:" + str(params.verbosity))
    print("Only open:" + str(params.only_open))
    portlist = params.ports.split(',')
    for port in portlist:
        print("Port:" + port)

Running the script above with the -h option shows the arguments it accepts and some execution use cases.

$ python params_port_scanning.py -h
usage: params_port_scan_complete.py [-h] -target TARGET [-ports PORTS] [-v] [--open]
Port scanning
optional arguments:
  -h, --help      show this help message and exit
  -target TARGET  target to scan
  -ports PORTS    Please, specify the target port(s) separated by comma[80,8080 by default]
  -v              verbosity level: -v, -vv, -vvv.
  --open          only display open ports
Uses cases:
        +  Basic scan:
             -target 127.0.0.1
        + Specific port:
             -target 127.0.0.1 -port 21
        + Port list:
             -target 127.0.0.1 -port 21,22
        + Only show open ports
             -target 127.0.0.1 --open True

When running the above script without any parameters, we get an error message stating the target argument is required.

$ python params_port_scanning.py
usage: params_port_scanning.py [-h] -target TARGET [-ports PORTS] [-v] [--open]
params_port_scanning.py: error: the following arguments are required: -target

When running the above script with the target argument, we get default values for the rest of parameters. For example, default values are 0 for verbosity and 80 and 8080 for ports.

$ python params_port_scanning.py -target localhost
Params:Namespace(only_open=False, ports='80,8080', target='localhost', verbosity=0)
Target:localhost
Verbosity:0
Only open:False
Port:80
Port:8080

When running the above script with the target, ports, and verbosity arguments, we get new values for these parameters.

$ python params_port_scanning.py -target localhost -ports 22,23 -vv
Params:Namespace(only_open=False, ports='22,23', target='localhost', verbosity=2)
Target:localhost
Verbosity:2
Only open:False
Port:22
Port:23

Managing parameters with OptionParser

Python provides a class called OptionParser for managing command-line arguments. OptionParser is part of the optparse module, which is provided by the standard library. OptionParser allows you to do a range of very useful things with command-line arguments:

• Specify a default if a certain argument is not provided.
• It supports both argument flags (either present or not) and arguments with values.
• It supports different formats of passing arguments.

Let’s use OptionParser to manage parameters in the same way we have seen before with the argparse module. In the code provided here, command-line arguments are used to pass in variables.

You can find the following code in the params_global_optparser.py file:

from optparse import OptionParser
class Parameters:
    """Global parameters"""
    def __init__(self, **kwargs):
        self.param1 = kwargs.get("param1")
        self.param2 = kwargs.get("param2")
 
def view_parameters(input_parameters):
    print(input_parameters.param1)
    print(input_parameters.param2)
parser = OptionParser()
parser.add_option("--p1", dest="param1", help="parameter1")
parser.add_option("--p2", dest="param2", help="parameter2")
(options, args) = parser.parse_args()
input_parameters = Parameters(param1=options.param1,param2=options.param2)
view_parameters(input_parameters)

The previous script demonstrates the use of the OptionParser class. It provides a simple interface for command-line arguments, allowing you to define certain properties for each command-line option. It also allows you to specify default values. If certain arguments are not provided, it allows you to throw specific errors.

For more information, you can check out the official website: https://docs.python.org/3/library/optparse.html.

Now that you know how Python manages modules and packages, let’s move on to learning how to manage dependencies and create a virtual environment with the virtualenv utility.

In this section, you will be able to identify how to manage dependencies and the execution environment with pip and virtualenv.

Managing dependencies in a Python project

If our project has dependencies with other libraries, the goal will be to have a file where we have such dependencies, so that our module is built and distributed as quickly as possible. For this function, we can create a file called requirements.txt, which contains all the dependencies the module requires.

To install all the dependencies, we can use the following command with the pip utility:

$ pip -r requirements.txt

Here, pip is the Python package and dependency manager and requirements.txt is the file where all the dependencies of the project are saved.

Install Python modules

Python has an active community of developers and users who develop both standard Python modules, as well as modules and packages developed by third parties. The Python Package Index, or PyPI (https://pypi.org), is the official software package repository for third-party applications in the Python programming language.

To install a new python Package, you have the following alternatives:

• Use the one that is packaged depending on the operating system and distribution you are using. For example, using apt-cache show <package>
• Install pip on your computer and, as a superuser, install the Python package that interests us. This solution can give us many problems, since we can break the dependencies between the versions of our Python packages installed on the system and some package may stop working.
• Use virtual environments: It is a mechanism that allows you to manage Python programs and packages without having administration permissions, that is, any user without privileges can have one or more “isolated spaces” where they can install different versions of Python programs and packages. To create the virtual environments, we can use the virtualenv program or the venv module.

Generating the requirements.txt file

We also have the ability to create the requirements.txt file from the project source code. For this task, we can use the pipreqs module, whose code can be downloaded from the GitHub repository at https://github.com/bndr/pipreqs.

In this way, the module can be installed either with the pip install pipreqs command or through the GitHub code repository using the python setup.py install command.

For more information about the module, you can refer to the official PyPI page https://pypi.org/project/pipreqs/.

To generate the requirements.txt file, you could execute the following command:

$ pipreqs <path_project>

Working with virtual environments

When operating with Python, it’s strongly recommended that you use virtual environments. A virtual environment provides a separate environment for installing Python modules and an isolated copy of the Python executable file and associated files.

You can have as many virtual environments as you need, which means that you can have multiple module configurations configured, and you can easily switch between them.

Configuring virtualenv

When you install a Python module on your local computer without having to use a virtual environment, you install it on the operating system globally. Typically, this installation requires a user root administrator, and the Python module is configured for each user and project.

The best approach at this point is to create a Python virtual environment if you need to work on many Python projects, or if you are working with several projects that are sharing some modules.

virtualenv is a Python module that enables you to build isolated, virtual environments. Essentially, you must create a folder that contains all the executable files and modules needed for a project. You can install virtualenv as follows:

1. Type in the following command:

   $ sudo pip install virtualenv
   

2. To create a new virtual environment, create a new folder and enter the folder from the command line:

   $ cd your_new_folder
   $ virtualenv name-of-virtual-environment
   $ source bin/activate
   

3. Once it is active, you will have a clean environment of modules and libraries, and you will have to download the dependencies of the project so that they are copied in this directory using the following command:

   (venv) > pip install -r requirements.txt
   

   Executing this command will initiate a folder with the name indicated in your current working directory with all the executable files of Python and the pip module, which allows you to install different packages in your virtual environment.

   IMPORTANT NOTE

   If you are working with Python 3.3+, virtualenv is included in stdlib. You can get an installation update for virtualenv in the Python documentation: https://docs.python.org/3/library/venv.html.

4. virtualenv is like a sandbox where all the dependencies of the project will be installed when you are working, and all modules and dependencies are kept separate. If users have the same version of Python installed on their machine, the same code will work in the virtual environment without requiring any changes.

Now that you know how to install your own virtual environment, let’s move on to review development environments for Python scripting, including Python IDLE and PyCharm.

In this section, we will review PyCharm and Python IDLE as development environments for Python scripting.

Setting up a development environment

In order to rapidly develop and debug Python applications, it is necessary to use an Integrated Development Environment (IDE). If you want to try different options, we recommend you check out the list that is on the official Python site, where you can see the tools according to your operating systems and needs:

https://wiki.python.org/moin/IntegratedDevelopmentEnvironments

Out of all the environments, the following two are the ones we will look at:

• Python IDLE: https://docs.python.org/3/library/idle.html
• PyCharm: http://www.jetbrains.com/pycharm

Debugging with Python IDLE

Python IDLE is the default IDE that is installed when you install Python in your operating system. Python IDLE allows you to debug your script and see errors and exceptions in the Python shell console:

Figure 1.1: Running a script in the Python shell

In the preceding screenshot, we can see the output in the Python shell and the exception is related to File not found.

PyCharm

PyCharm (https://www.jetbrains.com/pycharm) is a multi-platform tool that we can find for many operating systems, such as Windows, Linux, and macOS X. There are two versions of PyCharm, community and technical, with variations in functionality relating to web framework integration and support for databases. The main advantages of this development environment are as follows:

• Autocomplete, syntax highlighter, analysis tool, and refactoring
• Integration with web frameworks, such as Django and Flask
• An advanced debugger
• Connection with version control systems, such as Git, CVS, and SVN

In the following screenshot, we can see how to configure virtualenv in PyCharm:

Figure 1.2: Configuring virtualenv in PyCharm

In the preceding screenshot, we are setting the configuration related to establishing a new environment for the project using Virtualenv.

Debugging with PyCharm

In this example, we are debugging a Python script that is applying simple inheritance. An interesting topic is the possibility of adding a breakpoint to our script. In the following screenshot, we are setting a breakpoint in the __init__ method of the class ChildClass:

Figure 1.3: Setting a breakpoint in PyCharm

With the View Breakpoint option, we can see the breakpoint established in the script:

Figure 1.4: Viewing breakpoints in PyCharm

In the following screenshot, we can visualize the values of the parameters that contain the values we are debugging:

Figure 1.5: Debugging variables in PyCharm

In this way, we can know the state of each of the variables at runtime, as well as modify their values to change the logic of our script.

In this chapter, we learned how to install Python on the Windows and Linux operating systems. We reviewed the main data structures and collections, such as lists, tuples, and dictionaries. We also reviewed functions, managing exceptions, and how to create classes and objects, as well as the use of attributes and special methods. Then we looked at development environments and a methodology to introduce into programming with Python. Finally, we reviewed the main development environments, PyCharm and Python IDLE, for script development in Python.

In the next chapter, we will explore programming system packages for working with operating systems and filesystems, threads, and concurrency.

As we conclude, here is a list of questions for you to test your knowledge regarding this chapter’s material. You will find the answers in the Assessments section of the Appendix:

1. Which data structure in Python allows us to associate values with keys?
2. What are the methods we can use to add elements to a list?
3. What is the approach that we can follow in Python to handle files and manage exceptions in an easy and secure way?
4. What is the Python parent class for errors related to input/output?
5. What are the Python modules that enable you to build virtual environments?

In these links, you will find more information about the aforementioned tools and the official Python documentation for some of the modules we have analyzed:

• Python 3.10 version library: https://docs.python.org/3.10/library
• Virtualenv documentation: https://virtualenv.pypa.io/en/latest/
• Python Integrated Development Environments: https://wiki.python.org/moin/IntegratedDevelopmentEnvironments

Join our community on Discord

Join our community’s Discord space for discussions with the author and other readers:

https://packt.link/SecNet