Operating system-specific binaries

Python programs are normally provided in source code or wheel files. However, there are times when a developer wants to provide OS-specific files, such as a Windows .exe, for ease of installation. Python has a number of options for developers to create stand-alone executable files.

py2exe (https://pypi.python.org/pypi/py2exe/) is one option for creating Windows-specific files. Unfortunately, it is difficult to tell how maintained this project is, as the last release on https://pypi.python.org/pypi/py2exe/0.9.2.2 was in 2014, while http://www.py2exe.org references a release from 2008. It also appears to be only available for Python 3.4 and older versions. However, if you believe this program may be useful, it does convert Python scripts into Windows executables without requiring the installation of Python.

py2app (https://py2app.readthedocs.io/en/latest/) is the primary tool for creating stand-alone Mac bundles. This tool is still maintained at https://bitbucket.org/ronaldoussoren/py2app, and the latest release came out in January 2018. Building is much like with py2exe, but there are several library dependencies required, listed at https://py2app.readthedocs.io/en/latest/dependencies.html.

There are more cross-platform tools for making OS-specific executable programs than there are for specific operating systems. This is good, as many developers use Linux as their development environment and may not have access to a Windows or Mac machine.

For developers who don't want to set up multiple operating systems themselves, there are several online services that allow you to rent operating systems online. For example, http://virtualmacosx.com allows you access to a hosted Mac environment, while there are multiple options for Windows hosting, from Amazon Web Services to regular web hosts.

For those desiring local control of binary execution, cx_Freeze (https://anthony-tuininga.github.io/cx_Freeze/) is one of the more popular executable creation programs for Python. It only works with Python 2.7 or newer, but that shouldn't be a problem for most developers. However, if you want to use it with Python 2 code, you will have to use cx_Freeze version 5; starting with version 6, support for Python 2 code has been dropped.

The modules created by cx_Freeze are stored in ZIP files. Packages, by default, are stored in the file system but can be included in the same ZIP files, if desired.

PyInstaller (http://www.pyinstaller.org) has, as its main goal, compatibility with third-party packages, requiring no user intervention to make external packages work during binary creation. It is available for Python 2.7 and newer versions.

PyInstaller provides multiple ways to package your Python code: as a single directory (containing the executable as well as all necessary modules), as a single file (self-contained and requiring no external dependencies), or in custom mode.

The majority of third-party packages will work with PyInstaller with no additional configuration required. Conveniently, a list, located at https://github.com/pyinstaller/pyinstaller/wiki/Supported-Packages, is provided for packages known to work with PyInstaller; if there are any limitations, for example, only working on Windows, these are noted as well.

Cython (http://cython.org) is actually a superset of Python, designed to give C-like performance to Python code. This is done by allowing types to be added to the Python code; whereas Python is normally dynamically typed, Cython allows static typing of variables. The resulting code is compiled into C code, which can be executed by the normal Python interpreter as normal, but at the speed of compiled C code.

While normally used to create extensions for Python, or to speed up Python processing, using the --embed flag with the cpython command will create a C file, which can then be compiled to a normal application file.

Naturally, this takes more knowledge of using gcc or your compiler of choice, as you have to know how to import the Python headers during compilation, and which other directories need to be included. As such, Cython isn't recommended for developers unfamiliar with C code, but it can be a powerful way to make full-featured applications by utilizing both Python and C languages.

Nuitka (http://nuitka.net) is a relatively new Python compiler program. It is compatible with Python 2.6 and later, but also requires gcc or another C compiler. The latest version, 0.5.29, is beta-ware, but the author claims it is able to compile every Python construct currently available without a problem.

Nuitka functions much like Cython, in that it uses a C compiler to convert Python code into C code, and make executable files. Entire programs can be compiled, with the modules embedded in the file, but individual modules can be compiled by themselves, if desired.

By default, the resulting binary requires Python to be installed, plus the necessary C extension modules. However, it is possible to create true stand-alone executables by using the --stand-alone flag.

How to do it...

1. Write your Python program.
2. To create a Windows .exe file, create a setup.py file to tell the libraries what you want to do. This is mainly importing the setup() function from the Distutils library, importing py2exe, and then calling setup and telling it what type of application it is making, for example, a console, and what the main Python file is. py2exe_setup.py, following, is an example from the documentation of a setup.py file:

      from distutils.core import setup
      import py2exe
      setup(console=['hello.py'])

3. Run the setup script by calling python setup.py py2exe. This creates two directories: build/ and dist/. The dist/ directory is where the new files are placed, while build/ is used for temporary files during the creation process.
4. Test the application by moving to the dist/ directory and running the .exe file located there.
5. To make a macOS .app file, create the setup.py file. Any icons or data files required for the application need to be included during this step.
6. Clean up the build/ and dist/ directories to ensure there are no files that may be accidentally included.
7. Use Alias mode to build the application in-place, that is, not ready for distribution. This allows you to test the program before bundling for delivery.
8. Test the application and verify it works correctly in alias mode.
9. Clean up the build/ and dist/ directories again.
10. Run python setup.py py2app to create the distributable .app file.
11. For cross-platform files, the easiest way to use cx_Freeze is to use the cxfreeze script:

      cxfreeze <program>.py --target-dir=<directory>

Other options are available for this command, such as compressing the bytecode, setting an initialization script, or even excluding modules.

If more functionality is required, a distutils setup script can be created. The command cxfreeze-quickstart can be used to generate a simple setup script; the cx_Freeze documentation provides an example setup.py file (cxfreeze_setup.py):

      import sys
      from cx_Freeze import setup, Executable

      # Dependencies are automatically detected, but it might need fine tuning.
      build_exe_options = {"packages": ["os"], "excludes": ["tkinter"]}

      # GUI applications require a different base on Windows (the default is for 
      # console application).
      base = None
      if sys.platform == "win32":
          base = "Win32GUI"

      setup(  name = "guifoo",
              version = "0.1",
              description = "My GUI application!",
              options = {"build_exe": build_exe_options},
              executables = [Executable("guifoo.py", base=base)])

To run the setup script, run the command: python setup.py build. This will create the directory build/, which contains the subdirectory exe.xxx, where xxx is the platform-specific executable binary indicator:

• • For developers who need even more control, or are looking at creating C scripts for extending or embedding Python, manually working with the classes and modules within the cx_Freeze program is possible.

12. If using PyInstaller, its use is like most other Python programs,  and is a simple command:

      pyinstaller <program>.py

This generates the binary bundle in the dist/ subdirectory. Naturally, there many other options available when running this command:

• • Optionally, UPX (https://upx.github.io/) can be used to compress the executable files and libraries. When used, UPX compresses the files and wraps them in a self-decompressing file. When executed, the UPX wrapper decompresses the enclosed files and the resulting binary is executed normally.
  • To create multiple Python environments for a single operating system, it is recommended you to create virtual Python environments for each Python version to be generated. Then, install PyInstaller in each environment and build the binary within each environment.
  • Like cx_Freeze, to create binaries for different operating systems, the other OSes must be available and PyInstaller used on each one.
  • Create your Python file; save it with the extension .pyx. For example, helloworld.pyx.

13. When working with Cython, create a setup.py file that looks similar to cython_setup.py from http://docs.cython.org/en/latest/src/tutorial/cython_tutorial.html#the-basics-of-cython:

      from distutils.core import setup
      from Cython.Build import cythonize

      setup(
          ext_modules = cythonize("helloworld.pyx")
      )

14. Create the Cython file by running the following:

      $ python setup.py build_ext --inplace

15. This creates a file in the local directory: helloworld.so on *nix and helloworld.pyd on Windows.
16. To use the binary, simply import it into Python as normal.
17. If your Python program doesn't require additional C libraries or a special build configuration, you can use the pyximport library. The install() function from this library allows loading .pyx files directly when imported, rather than having to rerun setup.py every time the code changes.

18. To compile a program using Nuitka with all modules embedded, use the following command:

      nuitka --recurse-all <program>.py

19. To compile a single module, use the following command:

      nuitka --module <module>.py

20. To compile an entire package and embed all modules, the previous commands are combined into a similar format:

      nuitka --module <package> --recurse-directory=<package>

21. To make a truly cross-platform binary, use the option --standalone, copy the <program>.dist directory to the destination system, and then run the .exe file inside that directory.

There's more...

Depending on a user's system configuration, you may need to provide the Microsoft Visual C runtime DLL. The py2exe documentation provides different files to choose from, depending on the version of Python you are working with.

In addition, py2exe does not create the installation builder, that is, installation wizard. While it may not be necessary for your application, Windows users generally expect a wizard to be available when running an .exe file. A number of free, open-source, and proprietary installation builders are available.

One benefit of building Mac binaries is that they are simple to pack for distribution; once the .app file is generated, right-click on the file and choose Create Archive. After that, your application is ready to be shipped out.

A common problem with cx_Freeze is that the program doesn't automatically detect a file that needs to be copied. This frequently occurs if you are dynamically importing modules into your program, for example, a plugin system.

Binaries created by cx_Freeze are generated for the OS it was run on; for instance, to create a Windows .exe file, cx_Freeze has to be used on a Windows computer. Thus, to make a truly cross-platform Python program that is distributed as executable binaries, you must have access to other operating systems. This can be alleviated by using virtual machines, cloud hosts, or simply purchasing the relevant systems.

When PyInstaller is run, it analyzes the supplied Python program and creates a <program>.spec file in the same folder as the Python program. In addition, the build/ subdirectory is placed in the same location.

The build/ directory contains log files and the working files used to actually create the binary. After the executable file is generated, a dist/ directory is placed in the same location as the Python program, and the binary is placed in the dist/ directory.

The executable file generated by Nuitka will have the .exe extension on all platforms. It is still usable on non-Windows OSes, but it is recommended to change the extension to a system-specific one to avoid confusion.

The binary files created with any of the commands previously shown require Python to be installed on the end system, as well as any C extension modules that are used.