Spark distributed training architecture details

The Distributed network training with Spark and DeepLearning4J section in Chapter 7, Training Neural Networks with Spark, explains why it is important to train MNNs in a distributed way across a cluster, and states that DL4J uses a parameter averaging approach to parallel training. This section goes through the architecture details of the distributed training approaches (parameter averaging and gradient sharing, which replaced the parameter averaging approach in DL4J starting from release 1.0.0-beta of the framework). The way DL4J approaches distributed training is transparent to developers, but it is good to have knowledge of it anyway.

Model parallelism and data parallelism

...