The exam consists of 60 questions. The time you’re given to attempt these questions is 120 minutes. This gives you about 2 minutes per question.

To pass the exam, you need to have a score of 70%, which means that you need to answer 42 questions correctly out of 60 for you to pass.

If you are well prepared, this time should be enough for you to answer the questions and also review them before the time finishes.

Next, we will see how the questions are distributed throughout the exam.

Distribution of questions

Exam questions are distributed into the following broad categories. The following table provides a breakdown of questions based on different categories:

Table 1.1: Exam breakdown

Looking at this distribution, you would want to focus on the Spark DataFrame API a lot more in your exam preparation since this section covers around 72% of the exam (about 43 questions). If you can answer these questions correctly, passing the exam will become easier.

But this doesn’t mean that you shouldn’t focus on the Spark architecture areas. Spark architecture questions have varied difficulty, and they can sometimes be confusing. At the same time, they allow you to score easy points as architecture questions are generally straightforward.

Let’s look at some of the other resources available that can help you prepare for this exam.

Resources to prepare for the exam

When you start planning to take the certification exam, the first thing you must do is master Spark concepts. This book will help you with these concepts. Once you’ve done this, it would be useful to do mock exams. There are two mock exams available in this book for you to take advantage of.

In addition, Databricks provides a practice exam, which is very useful for exam preparation. You can find it here: https://files.training.databricks.com/assessments/practice-exams/PracticeExam-DCADAS3-Python.pdf.

Resources available during the exam

During the exam, you will be given access to the Spark documentation. This is done via Webassessor and its interface is a little different than the regular Spark documentation you’ll find on the internet. It would be good for you to familiarize yourself with this interface. You can find the interface at https://www.webassessor.com/zz/DATABRICKS/Python_v2.html. I recommend going through it and trying to find different packages and functions of Spark via this documentation to make yourself comfortable navigating it during the exam.

Next, we will look at how we can register for the exam.

Databricks is the company that has prepared these exams and certifications. Here is the link to register for the exam: https://www.databricks.com/learn/certification/apache-spark-developer-associate.

Next, we will look at some of the prerequisites for the exam.

Prerequisites for the exam

Some prerequisites are needed before you can take the exam so that you can be successful in passing the certification. Some of the major ones are as follows:

• Grasp the fundamentals of Spark architecture, encompassing the principles of Adaptive Query Execution.
• Utilize the Spark DataFrame API proficiently for various data manipulation tasks, such as the following:
  • Performing column operations, such as selection, renaming, and manipulation
  • Executing row operations, including filtering, dropping, sorting, and aggregating data
  • Conducting DataFrame-related tasks, such as joining, reading, writing, and implementing partitioning strategies
  • Demonstrating proficiency in working with user-defined functions (UDFs) and Spark SQL functions
• While not explicitly tested, a functional understanding of either Python or Scala is expected. The examination is available in both programming languages.

Hopefully, by the end of this book, you will be able to fully grasp all these concepts and have done enough practice on your own to be prepared for the exam with full confidence.

Now, let’s discuss what to expect during the online proctored exam.

Online proctored exam

The Spark certification exam is an online proctored exam. What this means is that you will be taking the exam from the comfort of your home, but someone will be proctoring the exam online. I encourage you to understand the procedures and rules of the proctored exam in advance. This will save you a lot of trouble and anxiety at the time of the exam.

To give you an overview, throughout the exam session, the following procedures will be in place:

• Webcam monitoring will be conducted by a Webassessor proctor to ensure exam integrity
• You will need to present a valid form of identification with a photo
• You will need to conduct the exam alone
• Your desk needs to be decluttered and there should be no other electronic devices in the room except the laptop that you’ll need for the exam
• There should not be any posters or charts on the walls of the room that may aid you in the exam
• The proctor will be listening to you during the exam as well, so you’ll want to make sure that you’re sitting in a quiet and comfortable environment
• It is recommended to not use your work laptop for this exam as it requires software to be installed and your antivirus and firewall to be disabled

The proctor’s responsibilities are as follows:

• Overseeing your exam session to maintain exam integrity
• Addressing any queries related to the exam delivery process
• Offering technical assistance if needed
• It’s important to note that the proctor will not offer any form of assistance regarding the exam content

I recommend that you take sufficient time before the exam to set up the environment where you’ll be taking the exam. This will ensure a smooth online exam procedure where you can focus on the questions and not worry about anything else.

Now, let’s talk about the different types of questions that may appear in the exam.

There are different categories of questions that you will find in the exam. They can be broadly divided into theoretical and code questions. We will look at both categories and their respective subcategories in this section.

Theoretical questions

Theoretical questions are the questions where you will be asked about the conceptual understanding of certain topics. Theoretical questions can be subdivided further into different categories. Let’s look at some of these categories, along with example questions taken from previous exams that fall into them.

Explanation questions

Explanation questions are ones where you need to define and explain something. It can also include how something works and what it does. Let’s look at an example.

Which of the following describes a worker node?

1. Worker nodes are the nodes of a cluster that perform computations.
2. Worker nodes are synonymous with executors.
3. Worker nodes always have a one-to-one relationship with executors.
4. Worker nodes are the most granular level of execution in the Spark execution hierarchy.
5. Worker nodes are the coarsest level of execution in the Spark execution hierarchy.

Connection questions

Connections questions are such questions where you need to define how different things are related to each other or how they differ from each other. Let’s look at an example to demonstrate this.

Which of the following describes the relationship between worker nodes and executors?

1. An executor is a Java Virtual Machine (JVM) running on a worker node.
2. A worker node is a JVM running on an executor.
3. There are always more worker nodes than executors.
4. There are always the same number of executors and worker nodes.
5. Executors and worker nodes are not related.

Scenario question

Scenario questions involve defining how things work in different if-else scenarios – for example, “If ______ occurs, then _____ happens.” Moreover, it also includes questions where a statement is incorrect about a scenario. Let’s look at an example to demonstrate this.

If Spark is running in cluster mode, which of the following statements about nodes is incorrect?

1. There is a single worker node that contains the Spark driver and the executors.
2. The Spark driver runs in its own non-worker node without any executors.
3. Each executor is a running JVM inside a worker node.
4. There is always more than one node.
5. There might be more executors than total nodes or more total nodes than executors.

Categorization questions

Categorization questions are such questions where you need to describe categories that something belongs to. Let’s look at an example to demonstrate this.

Which of the following statements accurately describes stages?

1. Tasks within a stage can be simultaneously executed by multiple machines.
2. Various stages within a job can run concurrently.
3. Stages comprise one or more jobs.
4. Stages temporarily store transactions before committing them through actions.

Configuration questions

Configuration questions are such questions where you need to outline how things will behave based on different cluster configurations. Let’s look at an example to demonstrate this.

Which of the following statements accurately describes Spark’s cluster execution mode?

1. Cluster mode runs executor processes on gateway nodes.
2. Cluster mode involves the driver being hosted on a gateway machine.
3. In cluster mode, the Spark driver and the cluster manager are not co-located.
4. The driver in cluster mode is located on a worker node.

Next, we’ll look at the code-based questions and their subcategories.

Code-based questions

The next category is code-based questions. A large number of Spark API-based questions lie in this category. Code-based questions are the questions where you will be given a code snippet, and you will be asked questions about it. Code-based questions can be subdivided further into different categories. Let’s look at some of these categories, along with example questions taken from previous exams that fall into these different subcategories.

Function identification questions

Function identification questions are such questions where you need to define which function does something. It is important to know the different functions that are available in Spark for data manipulation, along with their syntax. Let’s look at an example to demonstrate this.

Which of the following code blocks returns a copy of the df DataFrame, where the column salary has been renamed employeeSalary?

1. df.withColumn(["salary", "employeeSalary"])
2. df.withColumnRenamed("salary").alias("employeeSalary ")
3. df.withColumnRenamed("salary", " employeeSalary ")
4. df.withColumn("salary", " employeeSalary ")

Fill-in-the-blank questions

Fill-in-the-blank questions are such questions where you need to complete the code block by filling in the blanks. Let’s look at an example to demonstrate this.

The following code block should return a DataFrame with the employeeId, salary, bonus, and department columns from the transactionsDf DataFrame. Choose the answer that correctly fills the blanks to accomplish this.

df.__1__(__2__)

1. 1. drop
   2. "employeeId", "salary", "bonus", "department"
2. 1. filter
   2. "employeeId, salary, bonus, department"
3. 1. select
   2. ["employeeId", "salary", "bonus", "department"]
4. 1. select
   2. col(["employeeId", "salary", "bonus", "department"])

Order-lines-of-code questions

Order-lines-of-code questions are such questions where you need to place the lines of code in a certain order so that you can execute an operation correctly. Let’s look at an example to demonstrate this.

Which of the following code blocks creates a DataFrame that shows the mean of the salary column of the salaryDf DataFrame based on the department and state columns, where age is greater than 35?

1. salaryDf.filter(col("age") > 35)
2. .filter(col("employeeID")
3. .filter(col("employeeID").isNotNull())
4. .groupBy("department")
5. .groupBy("department", "state")
6. .agg(avg("salary").alias("mean_salary"))
7. .agg(average("salary").alias("mean_salary"))

1. i, ii, v, vi
2. i, iii, v, vi
3. i, iii, vi, vii
4. i, ii, iv, vi

This chapter provided an overview of the certification exam. At this point, you know what to expect in the exam and how to best prepare for it. To do so, we covered different types of questions that you will encounter.

Going forward, each chapter of this book will equip you with practical knowledge and hands-on examples so that you can harness the power of Apache Spark for various data processing and analytics tasks.