Two-level scheduling
Mesos has a two-level scheduling mechanism to allocate resources to and launch tasks on different frameworks. In the first level, the master process that manages slave processes running on each node in the Mesos cluster determines the free resources available on each node, groups them, and offers them to different frameworks based on organizational policies, such as priority or fair sharing. Organizations have the ability to define their own sharing policies via a custom allocation module as well.
In the second level, each framework's scheduler component that is registered as a client with the master accepts or rejects the resource offer made depending on the framework's requirements. If the offer is accepted, the framework's scheduler sends information regarding the tasks that need to be executed and the number of resources that each task requires to the Mesos master. The master transfers the tasks to the corresponding slaves, which assign the necessary resources to the framework's executor component, which manages the execution of all the required tasks in containers. When the tasks are completed, the containers are dismantled, and the resources are freed up for use by other tasks.
The following diagram and explanation from the Apache Mesos documentation (http://mesos.apache.org/documentation/latest/architecture/) explains this concept in more detail:
Let's have a look at the pointers mentioned in the preceding diagram:
- 1: Slave 1 reports to the master that it has four CPUs and 4 GB of memory free. The master then invokes the allocation module, which tells it that Framework 1 should be offered all the available resources.
- 2: The master sends a resource offer describing these resources to Framework 1.
- 3: The framework's scheduler replies to the master with information about two tasks to run on the slave using two CPUs and 1 GB RAM for the first task and one CPU and 2 GB RAM for the second task.
- 4: The master sends the tasks to the slave, which allocates appropriate resources to the framework's executor, which in turn launches the two tasks. As one CPU and 1 GB of RAM are still free, the allocation module may now offer them to Framework 2. In addition, this resource offers process repeats when tasks finish and new resources become free.
Mesos also provides frameworks with the ability to reject resource offers. A framework can reject the offers that do not meet its requirements. This allows frameworks to support a wide variety of complex resource constraints while keeping Mesos simple at the same time. A policy called delay scheduling, in which frameworks wait for a finite time to get access to the nodes storing their input data, gives a fair level of data locality albeit with a slight latency tradeoff.
If the framework constraints are complex, it is possible that a framework might need to wait before it receives a suitable resource offer that meets its requirements. To tackle this, Mesos allows frameworks to set filters specifying the criteria that they will use to always reject certain resources. A framework can set a filter stating that it can run only on nodes with at least 32 GB of RAM space free, for example. This allows it to bypass the rejection process, minimizes communication overheads, and thus reduces overall latency.
