Challenges

Now that we have gained an understanding of the Spark architecture, let's prepare for writing scalable analytics by introducing some of the challenges, or gotchas that you might face if you're not careful. Without knowledge of these up-front, you could lose time trying to figure them out on your own!

Algorithmic complexity

As well as the obvious effect of the size of your data, the performance of an analytic is highly dependent on the nature of the problem you're trying to solve. Even some seemingly simple problems, such as a depth first search of a graph, do not have well-defined algorithms that perform efficiently in distributed environments. This being the case, great care should be taken when designing analytics to ensure that they exploit patterns of processing that are readily parallelized. Taking the time to understand the nature of your problem in terms of complexity before you start, can pay off in the long term. In the next section, we'll show you...