关于大数据:Spark主备切换机制原理

Master实际上能够配置两个，那么在spark原生的standalone上也是反对Master主备切换的，也就是说，当Active Master节点挂掉之后，咱们能够将Standby Master切换为Active Master

Spark Master的主备切换能够基于两种切换机制，一种是文件系统，一种是基于Zookeeper,基于文件系统的机制，是Active Master挂掉后，须要咱们手动去切换到Standby Master上，基于Zookeeper机制，呆以实现主动切换。

所以这里说的主备切换机制，其实指的是在Active Master挂掉之后，切换到Standby Master时，Master会做哪些操作

1.应用长久化引挚(FileSystemPersistence或者是ZookeeperPersisitence)去读取长久化的storedApps,storedDriver,storedWorker,

2.判断下面的三个长久化的storedApps,storedDriver,storedWorker,

有任何一个不为空，就将长久化有Application,Driver,Worker的信息从新注册，注册到Master外部的缓存构造中。

3.将Application和Worker的状态都批改为UNKNOWN,而后向Application对应的Driver，Worker发送Standby Master的地址.

4.Driver,Worker,实践上讲，如果他们目前是失常工作的话，那么在收到Master发送来的地址后，就会返回响应给新的Master。

5.此时，Master在陆续接管到Driver,Worker发送来的响应音讯之后，会应用completeRecovery()对没有收到发送响应音讯的Driver,Worker进行解决，过滤掉他们的信息。如下：

// Kill off any workers and apps that didn’t respond to us.
workers.filter(.state == WorkerState.UNKNOWN).foreach(removeWorker)
apps.filter(.state == ApplicationState.UNKNOWN).foreach(finishApplication)

// Reschedule drivers which were not claimed by any workers
drivers.filter(_.worker.isEmpty).foreach { d =>
logWarning(s"Driver ${d.id} was not found after master recovery")
if (d.desc.supervise) {

logWarning(s"Re-launching ${d.id}")relaunchDriver(d)

} else {

removeDriver(d.id, DriverState.ERROR, None)logWarning(s"Did not re-launch ${d.id} because it was not supervised")

}
6.调用Master的schedule()，对正在期待调度的Driver,Application进行调度，比方在某个Worker上启动Driver,或者为Application在Worker上启动Executor。

state = RecoveryState.ALIVE
schedule()
理解更多大数据面试问题欢送关注小编大数据培训专栏！