Constructing a decision tree using Rpart

While OneR is very good at determining simple classification rules, it is not able to construct full decision trees. However, we can extract a sample from Spark and route it to any R decision tree algorithm, such as rpart.

First collect the sample

To illustrate this, let's first take a 50% sample of the stop and frisk dataframe. We also want to make sure that the amount of data we extract can be processed easily by base R, which has a memory limitation that is dependent upon the CPU size.

• The code below will first extract a 50% sample from Spark and store it in a local R dataframe named dflocal.
• Then it will run an str() command to verify the rowcount and the metadata:

dflocal...