How to create and reference module packages

We have talked about modules and packages, using the terms interchangeably. However, there is a difference between a module and a package: packages are actually collections of modules and they include a __init__.py file, which can just be an empty file.

The dot-nomenclature used in modules to access specific functions or variables is also used in packages. This time, dotted names allow multiple modules within a package to be accessed without having name conflicts; each package creates its own namespace, and all the modules have their own namespaces.

When packages contain sub-packages (as in the following example), importing modules can be done with either absolute or relative paths. For example, to import the sepia.py module, one could import it with an absolute path: from video.effects.specialFX import sepia.

How to do it...

1. When making a package, follow the normal filesystem hierarchy in terms of directory structure; that is, modules that relate to each other should be placed in their own directory.
2. A possible package for a video file handler is shown in package_tree.py:

      video/                  # Top-level package
          __init__.py         # Top-level initialization
          formats/            # Sub-package for file formats
              __init__.py     # Package-level initialization
              avi_in.py
              avi_out.py
              mpg2_in.py
              mpg2_out.py
              webm_in.py
              webm_out.py
          effects/             # Sub-package for video effects
              specialFX/       # Sub-package for special effects
                  __init__.py
                  sepia.py
                  mosaic.py
                  old_movie.py
                  glass.py
                  pencil.py
                  tv.py
              transform/        # Sub-package for transform effects
                  __init__.py
                  flip.py
                  skew.py
                  rotate.py
                  mirror.py
                  wave.py
                  broken_glass.py
              draw/              # Sub-package for draw effects
                  __init__.py
                  rectangle.py
                  ellipse.py
                  border.py
                  line.py
                  polygon.py

3. But, what happens if you were already in the specialFX/ directory and wanted to import from another package? Use relative paths to walk the directory and import using dots, just like changing directories on the command-line:

      from . import mosaic
      from .. import transform
      from .. draw import rectangle

How it works...

In this example, the whole video package comprises two sub-packages, video formats and video effects, with video effects having several sub-packages of its own. Within each package, each .py file is a separate module. During module importation, Python looks for packages on sys.path.

The inclusion of the __init__.py files is necessary so Python will treat the directories as packages. This prevents directories with common names from shadowing Python modules further along the search path. They also allow calling modules as stand-alone programs via the -m option, when calling Python programs.

Initialization files are normally empty but can contain initialization code for the package. They can also contain an __all__ list, which is a Python list of modules that should be imported whenever from <package> import * is used.

The reason for __all__ is for the developer to explicitly indicate which files should be imported. This is to prevent excessive delay from importing all modules within a package that aren't necessarily needed for other developers. It also limits the chance of undesired side-effects when a module is inadvertently imported. The catch is, the developer needs to update the __all__ list every time the package is updated.

Relative imports are based on the name of the current module. As the main module for a program always has the name "__main__", any modules that will be the main module of an application must use absolute imports.

To be honest, it is generally safer to use absolute imports just to make sure you know exactly what you're importing; with most development environments nowadays providing suggestions for paths, it is just as easy to write out the auto-populated path as it is to use relative paths.

There's more...

If __all__ is not defined in __init__.py, then import * only imports the modules within the specified package, not all sub-packages or their modules. For example, from video.formats import * only imports the video formats; the modules in the effects/ directory will not be included.

This is a best practice for Python programmers: as the Zen of Python (https://www.python.org/dev/peps/pep-0020/) states, explicit is better than implicit. Thus, importing a specific sub-module from a package is a good thing, whereas import * is frowned upon because of the possibility of variable name conflicts.

Packages have the __path__ attribute, which is rarely used. This attribute is a list that has the name of the directory where the package's __init__.py file is located. This location is accessed before the rest of the code for the file is run.

Modifying the package path affects future searches for modules and sub-packages within the package. This is useful when it is necessary to extend the number of modules found during a package search.