spark-taskscheduler-schedulable.adoc

Schedulable

Schedulable is a contract of schedulable entities.

Note	Schedulable is a private[spark] Scala trait. You can find the sources in org.apache.spark.scheduler.Schedulable.

There are currently two types of Schedulable entities in Spark:

• Pool

• TaskSetManager

Schedulable Contract

Every Schedulable follows the following contract:

• It has a name.

  name: String

• It has a parent Pool (of other Schedulables).

  parent: Pool

  With the parent property you could build a tree of Schedulables

• It has a schedulingMode, weight, minShare, runningTasks, priority, stageId.

  schedulingMode: SchedulingMode
  weight: Int
  minShare: Int
  runningTasks: Int
  priority: Int
  stageId: Int

• It manages a collection of Schedulables and can add or remove one.

  schedulableQueue: ConcurrentLinkedQueue[Schedulable]
  addSchedulable(schedulable: Schedulable): Unit
  removeSchedulable(schedulable: Schedulable): Unit

  Note	schedulableQueue is java.util.concurrent.ConcurrentLinkedQueue.

• It can query for a Schedulable by name.

  getSchedulableByName(name: String): Schedulable

• It can return a sorted collection of TaskSetManagers.

• It can be informed about lost executors.

  executorLost(executorId: String, host: String, reason: ExecutorLossReason): Unit

  It is called by TaskSchedulerImpl to inform TaskSetManagers about executors being lost.

• It checks for speculatable tasks.

  checkSpeculatableTasks(): Boolean

  Caution

  FIXME What are speculatable tasks?

getSortedTaskSetQueue

getSortedTaskSetQueue: ArrayBuffer[TaskSetManager]

getSortedTaskSetQueue is used in TaskSchedulerImpl to handle resource offers (to let every TaskSetManager know about a new executor ready to execute tasks).

schedulableQueue

schedulableQueue: ConcurrentLinkedQueue[Schedulable]

schedulableQueue is used in SparkContext.getAllPools.