Remote processes

So far, we have only executed our scripts on multiple local processors, but we can actually expand this much further. Using the multiprocessing library, it’s actually very easy to execute jobs on remote servers, but the documentation is currently still a bit cryptic. There are actually a few ways of executing processes in a distributed way, but the most obvious one isn’t the easiest one. The multiprocessing.connection module has both the Client and Listener classes, which facilitate secure communication between the clients and servers in a simple way.

Communication is not the same as process management and queue management, however; those features require some extra effort. The multiprocessing library is still a bit bare in this regard, but it’s most certainly possible given a few different processes.

Distributed processing using multiprocessing

We will start with a module containing a few constants that should be shared between all...