Kubernetes能够应用一个lock对象,进行多实例的选举:

  • 抢到lock对象更新权的实例即为leader;

lock对象能够是:

  • Lease
  • ConfigMap
  • Endpoints

controller-runtime罕用于operator的开发,其封装了client-go中leader选举的细节,leader选举的场景为:

  • 某一时刻只有一个实例能够reconcile,其它实例处于standby状态;
  • 一旦leader实例挂掉,standby实例能够顶上,继续执行reconcile;

一.operator

Operator开发中,应用controller-runtime实现leader选举非常简单,调用时传入election参数即可:

func main() {    flag.BoolVar(&enableLeaderElection, "enableLeaderElection", false, "default false, if enabled the cronHPA would be in primary and standby mode.")    flag.Parse()    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{        LeaderElection:     enableLeaderElection,       LeaderElectionID:   "kubernetes-cronhpa-controller",        MetricsBindAddress: metricsAddr,    })    ...}

多实例抢锁时,应用的资源对象为configmap:

# kubectl get cm -n kube-system kubernetes-cronhpa-controller -oyamlapiVersion: v1kind: ConfigMapmetadata:  annotations:    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"kubernetes-cronhpa-controller-6bcbdf9844-d2z4q_c36ac963-3e9c-4e32-ac14-ab48b6f2633b","leaseDurationSeconds":15,"acquireTime":"2023-04-08T16:14:30Z","renewTime":"2023-04-10T07:55:15Z","leaderTransitions":126}'  creationTimestamp: "2022-10-13T06:20:42Z"  managedFields:  - apiVersion: v1    fieldsType: FieldsV1    ...    manager: kubernetes-cronhpa-controller    operation: Update    time: "2022-10-13T06:20:42Z"  name: kubernetes-cronhpa-controller  namespace: kube-system  resourceVersion: "76431408"  uid: 5099216c-2fd6-452a-afeb-bd39047c2cbb

二.controller-runtime

若应用底层的client-go进行实例选举,通常有以下步骤:

  • 首先,创立resourcelock,指定锁的资源类型;
  • 而后,创立leaderElector对象,调用leaderElector.Run();

controller-runtime也是这么做的。

1. 创立resourcelock

创立ControllerManager的时候,创立resourcelock:

// controller-runtime/pkg/manager/manager.gofunc New(config *rest.Config, options Options) (Manager, error) {    ...    resourceLock, err := options.newResourceLock(leaderConfig, recorderProvider, leaderelection.Options{        LeaderElection:          options.LeaderElection,        LeaderElectionID:        options.LeaderElectionID,        LeaderElectionNamespace: options.LeaderElectionNamespace,    })    if err != nil {        return nil, err    }    return &controllerManager{        config:                  config,        ...        resourceLock:            resourceLock,    }}

创立resourcelock的细节:

  • 能够看出,这里创立的是ConfigMap类型的resourcelock;
// controller-runtime/pkg/leaderelection/leader_election.gofunc NewResourceLock(config *rest.Config, recorderProvider recorder.Provider, options Options) (resourcelock.Interface, error) {    ...    return resourcelock.New(resourcelock.ConfigMapsResourceLock,        options.LeaderElectionNamespace,        options.LeaderElectionID,        client.CoreV1(),        client.CoordinationV1(),        resourcelock.ResourceLockConfig{            Identity:      id,            EventRecorder: recorderProvider.GetEventRecorderFor(id),        })}

2. 创立LeaderElector对象,调用LeaderElector.Run():

首先,是ControllerManager.Start():

// controller-manager/pkg/manager/internal.gofunc (cm *controllerManager) Start(stop <-chan struct{}) (err error) {    ...    go func() {        if cm.resourceLock != nil {        //须要选举            err := cm.startLeaderElection()            if err != nil {                cm.errChan <- err            }        } else {                        //不须要选举            // Treat not having leader election enabled the same as being elected.            close(cm.elected)            go cm.startLeaderElectionRunnables()        }    }()    select {    case <-stop:        // We are done        return nil    case err := <-cm.errChan:        // Error starting or running a runnable        return err    }}

若须要选举:

  • 创立LeaderElector对象;
  • 调用LeaderElector.Run();
  • 选举胜利后,执行startLeaderElectionRunnables(),这里的runables即是Reconcile();
// controller-runtime/pkg/manager/internal.gofunc (cm *controllerManager) startLeaderElection() (err error) {    ...    l, err := leaderelection.NewLeaderElector(leaderelection.LeaderElectionConfig{        Lock:          cm.resourceLock,        LeaseDuration: cm.leaseDuration,        RenewDeadline: cm.renewDeadline,        RetryPeriod:   cm.retryPeriod,        Callbacks: leaderelection.LeaderCallbacks{            OnStartedLeading: func(_ context.Context) {                close(cm.elected)                cm.startLeaderElectionRunnables()    // 选举胜利后,执行Reconcile()            },            OnStoppedLeading: cm.onStoppedLeading,        },    })    ...    // Start the leader elector process    go l.Run(ctx)    return nil}

若不要选举,则间接执行startLeaderElectionRunnables(),即执行Reconcile()。

参考:

1.https://itnext.io/leader-election-in-kubernetes-using-client-...