Leveraging exception matching rules

The try statement lets us capture an exception. When an exception is raised, we have a number of choices for handling it:

• Ignore it: If we do nothing, the program stops. We can do this in two ways—don't use a try statement in the first place, or don't have a matching except clause in the try statement.
• Log it: We can write a message and use a raise statement to let the exception propagate after writing to a log; generally, this will stop the program.
• Recover from it: We can write an except clause to do some recovery action to undo any effects of the partially completed try clause.
• Silence it: If we do nothing (that is, use the pass statement), then processing is resumed after the try statement. This silences the exception.
• Rewrite it: We can raise a different exception. The original exception becomes a context for the newly raised exception.

What about nested contexts? In this case, an exception could be ignored by an inner try but handled by an outer context. The basic set of options for each try context is the same. The overall behavior of the software depends on the nested definitions.

Our design of a try statement depends on the way that Python exceptions form a class hierarchy. For details, see Section 5.4, Python Standard Library. For example, ZeroDivisionError is also an ArithmeticError and an Exception. For another example, FileNotFoundError is also an OSError as well as an Exception.

This hierarchy can lead to confusion if we're trying to handle detailed exceptions as well as generic exceptions.

Getting ready

Let's say we're going to make use of the shutil module to copy a file from one place to another. Most of the exceptions that might be raised indicate a problem too serious to work around. However, in the specific event of FileNotFoundError, we'd like to attempt a recovery action.

Here's a rough outline of what we'd like to do:

>>> from pathlib import Path
>>> import shutil
>>> import os
>>> source_dir = Path.cwd()/"data"
>>> target_dir = Path.cwd()/"backup"
>>> for source_path in source_dir.glob('**/*.csv'):
...     source_name = source_path.relative_to(source_dir)
...     target_path = target_dir/source_name
...     shutil.copy(source_path, target_path)

We have two directory paths, source_dir and target_dir. We've used the glob() method to locate all of the directories under source_dir that have *.csv files.

The expression source_path.relative_to(source_dir) gives us the tail end of the filename, the portion after the directory. We use this to build a new, similar path under the target_dir directory. This assures that a file named wc1.csv in the source_dir directory will have a similar name in the target_dir directory.

The problems arise with handling exceptions raised by the shutil.copy() function. We need a try statement so that we can recover from certain kinds of errors. We'll see this kind of error if we try to run this:

FileNotFoundError: [Errno 2] No such file or directory: '/Users/slott/Documents/Writing/Python/Python Cookbook 2e/Modern-Python-Cookbook-Second-Edition/backup/wc1.csv'

This happens when the backup directory hasn't been created. It will also happen when there are subdirectories inside the source_dir directory tree that don't also exist in the target_dir tree. How do we create a try statement that handles these exceptions and creates the missing directories?

How to do it...

1. Write the code we want to use indented in the try block:

   >>>     try:
   ...         shutil.copy(source_path, target_path)
   

2. Include the most specific exception classes first. In this case, we have a meaningful response to the specific FileNotFoundError.
3. Include any more general exceptions later. In this case, we'll report any generic OSError that's encountered. This leads to the following:

   >>>     try:
   ...         target = shutil.copy(source_path, target_path)
   ...     except FileNotFoundError:
   ...         target_path.parent.mkdir(exist_ok=True, parents=True)
   ...         target = shutil.copy(source_path, target_path)
   ...     except OSError as ex:
   ...         print(f"Copy {source_path} to {target_path} error {ex}")
   

We've matched exceptions with the most specific first and the more generic after that.

We handled FileNotFoundError by creating the missing directories. Then we did copy() again, knowing it would now work properly.

We logged any other exceptions of the class OSError. For example, if there's a permission problem, that error will be written to a log and the next file will be tried. Our objective is to try and copy all of the files. Any files that cause problems will be logged, but the copying process will continue.

How it works...

Python's matching rules for exceptions are intended to be simple:

• Process except clauses in order.
• Match the actual exception against the exception class (or tuple of exception classes). A match means that the actual exception object (or any of the base classes of the exception object) is of the given class in the except clause.

These rules show why we put the most specific exception classes first and the more general exception classes last. A generic exception class like Exception will match almost every kind of exception. We don't want this first, because no other clauses will be checked. We must always put generic exceptions last.

There's an even more generic class, the BaseException class. There's no good reason to ever handle exceptions of this class. If we do, we will be catching SystemExit and KeyboardInterrupt exceptions; this interferes with the ability to kill a misbehaving application. We only use the BaseException class as a superclass when defining new exception classes that exist outside the normal exception hierarchy.

There's more...

Our example includes a nested context in which a second exception can be raised. Consider this except clause:

...     except FileNotFoundError:
...         target_path.parent.mkdir(exist_ok=True, parents=True)
...         target = shutil.copy(source_path, target_path)

If the mkdir() method or shutil.copy() functions raise an exception while handling the FileNotFoundError exception, it won't be handled. Any exceptions raised within an except clause can crash the program as a whole. Handling this can involve nested try statements.

We can rewrite the exception clause to include a nested try during recovery:

...     try:
...         target = shutil.copy(source_path, target_path)
...     except FileNotFoundError:
...         try:
...             target_path.parent.mkdir(exist_ok=True, parents=True)
...             target = shutil.copy(source_path, target_path)
...         except OSError as ex2:
...             print(f"{target_path.parent} problem: {ex2}")
...     except OSError as ex:
...         print(f"Copy {source_path} to {target_path} error {ex}")

In this example, a nested context writes one message for OSError. In the outer context, a slightly different error message is used to log the error. In both cases, processing can continue. The distinct error messages make it slightly easier to debug the problems.

See also

• In the Avoiding a potential problem with an except: clause recipe, we look at some additional considerations when designing exception handling statements.