Implementing virtual Python environments

As touched on previously, Python virtual environments create separate Python environments, much like virtual machines allow multiple but separate operating systems. Python virtual environments are particularly useful when installing multiple
instances of the same module.

For example, assume you are working on a project that requires version 1.2 of a particular library module for legacy support. Now assume you download a Python program that uses version 2.2 of the same library. If you install everything in the default global location on your hard drive, for example, /usr/lib/python3.6/site-packages, the new program will install the updated library into the same location, overwriting the legacy software. Since you were using an old library for legacy support, there's a good chance that the updated library will break your application.

Also, on shared systems (especially if you don't have admin rights), there is a strong possibility that you simply can't install modules on the system, at least not in the default global site-packages directory. You may luck out and be able to install software for your account but, if you can't, you have to either request permission to install it or go without.

This is where virtual Python environments come into play. Each environment has its own installation directories and there is no sharing of libraries between environments. This means that each version of a module within an environment stays the same, even if you update global libraries. It also means you can have multiple versions of modules installed on your computer at the same time without having conflicts.

Virtual environments have their own shells as well, allowing access to an OS shell that is independent of any other environment or the underlying operating system. This recipe also shows how to spawn a new Python shell from pipenv. Doing this ensures all commands will have access to the installed packages within the virtual environment.

Getting ready

The old way to manage virtual environments was with the venv tool. To install it, use the command sudo apt install python3-venv.

To manage virtual environments in a modern way, the pipenv module (https://docs.pipenv.org/) was developed; it automatically creates and manages virtual environments for projects, as well as adding and removing packages from Pipfile when you install/uninstall packages. It can be installed using pip install pipenv.

Pipfile is an alternative to requirements.txt, which is used to specify exact versions of modules to include in a program. Pipfile actually comprises two separate files: Pipfile and (optionally) Pipfile.lock. Pipfile is simply a listing of the source location of imported modules, the module names themselves (defaulting to the most recent version), and any development packages that are required. pipfile.py, below, is an example of a Pipfile from the Pipenv site (https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"


[dev-packages]
pytest = "*"

Pipfile.lock takes the Pipfile and sets actual version numbers to all the packages, as well as identifying specific hashes for those files. Hashed values are beneficial to minimize security risks; that is, if a particular module version has a vulnerability, its hash value allows it to be easily identified, rather than having to search by version name or some other method. pipfile_lock.py, below, is an example of a Pipfile.lock file from the Pipenv site (https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock):

{
  "_meta": {
    "hash": {
      "sha256": "8d14434df45e0ef884d6c3f6e8048ba72335637a8631cc44792f52fd20b6f97a"
    },
    "host-environment-markers": {
      "implementation_name": "cpython",
      "implementation_version": "3.6.1",
      "os_name": "posix",
      "platform_machine": "x86_64",
      "platform_python_implementation": "CPython",
      "platform_release": "16.7.0",
      "platform_system": "Darwin",
      "platform_version": "Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64",
      "python_full_version": "3.6.1",
      "python_version": "3.6",
      "sys_platform": "darwin"
    },
    "pipfile-spec": 5,
    "requires": {},
    "sources": [
      {
        "name": "pypi",
        "url": "https://pypi.python.org/simple",
        "verify_ssl": true
      }
    ]
  },
  "default": {
    "certifi": {
      "hashes": [
        "sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704",
        "sha256:40523d2efb60523e113b44602298f0960e900388cf3bb6043f645cf57ea9e3f5"
      ],
      "version": "==2017.7.27.1"
    },
    "chardet": {
      "hashes": [
         "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
         "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
      ],
      "version": "==3.0.4"
    },
***further entries truncated***

How to do it...

1. The original, normal way to create a virtual environment comprises three separate steps. First, the virtual environment is created:

      >>> python3 -m venv <dir_name>

2. Next, the virtual environment is activated so it can be used:

      >>> source <dir_name>/bin/activate

3. Finally, pip is used to install the necessary module:

      >>> pip install <module>

4. To make this process easier, pipenv combines the pip and venv calls, so first we have to move to the desired directory where the virtual environment will be placed:

      >>> cd <project_name>

5. Next, we simply call pipenv to create the environment and install the desired module:

      >>> pipenv install <module>

6. Use pipenv to call the shell command and wait for the shell to be created. Observe that a virtual environment has been created and the command prompt is now activated within the environment. The following screenshot includes the commands from the previous steps, for clarity:

How it works...

The preceding pipenv example shows the developer changing to the desired directory for the project, and then invoking pipenv to simultaneously create the virtual environment, activate it, and install the desired module.

In addition to creating the virtual environment, once you have created your Python program, you can run the program using pipenv as well:

>>> pipenv run python3 <program_name>.py

Doing this ensures all installed packages in the virtual environment are available to your program, thus reducing the likelihood of unexpected errors.

When launching a pipenv shell, a new virtual environment is created, with indications of where the environment is created in the file system. In this case, two environment executables are created, referencing both the Python 3.6 command and the default Python command. (Depending on the systems, these may actually reference different versions of Python. For example, the default Python command may call the Python 2.7 environment instead of Python 3.6.)

There's more...

On a side note, the -m option indicates that Python is to run the module as a stand-alone script, that is, its contents will be ran within the __main__ namespace. Doing this means you don't have to know the full path to the module, as Python will look for the script in sys.path. In other words, for modules that you would normally import into another Python file can be run directly from the command line.

In the example of running pipenv, the command takes advantage of the fact that Python allows the -m option to run a module directly or allow it to be imported; in this case, pipenv imports venv to create the virtual environment as part of the creation process.