Using SparkSession and SQL

Spark exposes many SQL-like actions that can be taken upon a data frame. For example, we could load a data frame with product sales information in a CSV file:

from pyspark.sql import SparkSession spark = SparkSession(sc) df = spark.read.format("csv") \        .option("header", "true") \        .load("productsales.csv");df.show()

The example:

• Starts a SparkSession (needed for most data access)
• Uses the session to read a CSV formatted file, that contains a header record
• Displays initial rows

We have a few interesting columns in the sales data:

• Actual sales for the products by division
• Predicted sales for the products by division

If this were a bigger file, we could use SQL to determine the extent of the product list. Then the following is the Spark SQL to determine the product list:

df.groupBy("PRODUCT").count().show()

The data frame groupBy function works very similar to the SQL Group By clause. Group By collects the items in the dataset according to the values in the column...