Application monitoring

This section covers a different way of monitoring your Spark application. It is important to monitor your jobs to get a better understanding of your application's behavior. These observations can help you optimize your application code. There are different ways you can monitor your jobs.

Spark UI

Spark provides a built-in monitoring UI that provides useful information about your Spark applications. When you submit your job, Spark launches this UI at the driver host on the default port, 4040. If port 4040 is not available, then Spark tries to bind it on the next available port. You can also change this default setting by changing spark.ui.port property. Spark UI has multiple tabs:

• Jobs: Provides...