Ha

Hadoop HA什么是 HAHA是High Available缩写，是双机集群系统简称，指高可用性集群，是保证业务连续性的有效解决方案，一般有两个或两个以上的节点，且分为活动节点及备用节点。通常把正在执行业务的称为活动节点，而作为活动节点的一个备份的则称为备用节点。当活动节点出现问题，导致正在运行的业务（任务）不能正常运行时，备用节点此时就会侦测到，并立即接续活动节点来执行业务。从而实现业务的不中断或短暂中断。 hadoop HA机制介绍hadoop2.0的HA 机制有两个namenode，一个是active namenode，状态是active；另外一个是standby namenode，状态是standby。两者的状态是可以切换的，但不能同时两个都是active状态，最多只有1个是active状态。只有active namenode提供对外的服务，standby namenode是不对外服务的。active namenode和standby namenode之间通过NFS或者JN（journalnode，QJM方式）来同步数据。active namenode会把最近的操作记录写到本地的一个edits文件中（edits file），并传输到NFS或者JN中。standby namenode定期的检查，从NFS或者JN把最近的edit文件读过来，然后把edits文件和fsimage文件合并成一个新的fsimage，合并完成之后会通知active namenode获取这个新fsimage。active namenode获得这个新的fsimage文件之后，替换原来旧的fsimage文件。这样，保持了active namenode和standby namenode的数据的实时同步，standby namenode可以随时切换成active namenode（譬如active namenode挂了）。而且还有一个原来hadoop1.0的secondarynamenode，checkpointnode，buckcupnode的功能：合并edits文件和fsimage文件，使fsimage文件一直保持更新。所以启动了hadoop2.0的HA机制之后，secondarynamenode，checkpointnode，buckcupnode这些都不需要了。 搭建 hadoop HA 集群环境linux: CentOS-7.5_x64hadoop: hadoop-3.2.0zookeeper: zookeeper-3.4.10 机器规划主机名IP安装软件运行进程node-1192.168.91.11hadoopNameNode,ResourceManager,DFSZKFailoverControllernode-2192.168.91.12hadoop,zookeeperNameNode,ResourceManager,QuorumPeerMain,DFSZKFailoverControllernode-3192.168.91.13hadoop,zookeeperQuorumPeerMain,DataNode,NodeManager,JournalNodenode-4192.168.91.14hadoop,zookeeperQuorumPeerMain,DataNode,NodeManager,JournalNode前置准备四台机器需要ssh免密登录，node-2,node-3,node-4需要安装zookeeper、java环境 集群搭建# 下载$ wget http://mirrors.shu.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.2.0.tar.gz# 解压$ tar -zxvf hadoop-3.2.0.tar.gz# 配置系统的环境变量$ vim /etc/profileexport JAVA_HOME=/usr/local/jdk1.8.0_191export PATH=$PATH:$JAVA_HOME/binexport HADOOP_HOME=/export/servers/hadoop-3.2.0export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin# 进入解压目录配置环境变量$ cd $HADOOP_HOME# 配置hadoop-env.sh 添加下面配置（不配置启动会报错）$ vim etc/hadoop/core-site.xmlexport JAVA_HOME=/usr/local/jdk1.8.0_191export HDFS_NAMENODE_USER=rootexport HDFS_DATANODE_USER=rootexport HDFS_JOURNALNODE_USER=rootexport HDFS_SECONDARYNAMENODE_USER=rootexport YARN_RESOURCEMANAGER_USER=rootexport YARN_NODEMANAGER_USER=root# 配置core-site.xml$ vim etc/hadoop/core-site.xml<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <!-- HA 配置指定 hdfs 的 nameService 为ns --> <property> <name>fs.defaultFS</name> <value>hdfs://ns</value> </property> <!-- HA 配置，指定zookeeper地址 --> <property> <name>ha.zookeeper.quorum</name> <value>node-2:2181,node-3:2181,node-4:2181</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/export/data/hadoop/temp</value> </property> <property> <name>hadoop.proxyuser.root.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.root.groups</name> <value>*</value> </property></configuration># 配置hdfs-site.xml$ vim etc/hadoop/hdfs-site.xml<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <!--指定hdfs的nameservice为ns，需要和core-site.xml中的保持一致 --> <property> <name>dfs.nameservices</name> <value>ns</value> </property> <!-- bi下面有两个NameNode，分别是nn1，nn2 --> <property> <name>dfs.ha.namenodes.ns</name> <value>nn1,nn2</value> </property> <!-- nn1的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns.nn1</name> <value>node-1:9000</value> </property> <!-- nn1的http通信地址 --> <property> <name>dfs.namenode.http-address.ns.nn1</name> <value>node-1:50070</value> </property> <!-- nn2的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns.nn2</name> <value>node-2:9000</value> </property> <!-- nn2的http通信地址 --> <property> <name>dfs.namenode.http-address.ns.nn2</name> <value>node-2:50070</value> </property> <!-- 指定NameNode的edits元数据在JournalNode上的存放位置 --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://node-3:8485;node-4:8485/ns</value> </property> <!-- 指定JournalNode在本地磁盘存放数据的位置 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/export/data/hadoop/journaldata</value> </property> <!-- 开启NameNode失败自动切换 --> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- 配置失败自动切换实现方式 --> <property> <name>dfs.client.failover.proxy.provider.bi</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行--> <property> <name>dfs.ha.fencing.methods</name> <value> sshfence shell(/bin/true) </value> </property> <!-- 使用sshfence隔离机制时需要ssh免登陆 --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_rsa</value> </property> <!-- 配置sshfence隔离机制超时时间 --> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> <property> <name>dfs.ha.namenodes.jn</name> <value>node-3,node-4</value> </property></configuration># 配置mapred-site.xml$ vim etc/hadoop/mapred-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> </property></configuration># 配置yarn-site.xml$ vim etc/hadoop/yarn-site.xml<?xml version="1.0"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><configuration><!-- Site specific YARN configuration properties --> <!-- 开启RM高可用 --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!-- 指定RM的cluster id --> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yarn-ha</value> </property> <!-- 指定RM的名字 --> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <!-- 分别指定RM的地址 --> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>node-1</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>node-2</value> </property> <!-- 指定zk集群地址 --> <property> <name>yarn.resourcemanager.zk-address</name> <value>node-2:2181,node-3:2181,node-4:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property></configuration># 配置workers节点$ vim $HADOOP_HOME/etc/hadoop/workersnode-3node-4# 拷贝hadoop到其他节点（node-2,node-3,node-4）$ scp -r hadoop-3.2.0 root@node-2:/xxx/xxx/hdfs HA 配置# 启动zookeeper集群$ $ZOOKEEPER_HOME/bin/zkServer.sh start# 查看zookeeper状态$ $ZOOKEEPER_HOME/bin/zkServer.sh status# 启动 JournalNode 集群 分别在 node-3、node-4 上执行以下命令$ hdfs --daemon start journalnode# 格式化 ZooKeeper 集群$ hdfs zkfc -formatZK# 格式化集群的 NameNode (在 node-1 上执行)$ hdfs namenode -format# 启动刚格式化的 NameNode (在 node-1 上执行)$ hdfs --daemon start namenode# 同步 NameNode1 元数据到 NameNode2 上 (在 node-2 上执行) $ hdfs namenode -bootstrapStandby# 启动 NameNode2 (在 node-2 上执行)$ hdfs --daemon start namenode# 启动集群中所有的DataNode (在 node-1 上执行) $ sbin/start-dfs.sh# 启动 ZKFC 进程 (在 node-1 和 node-2 的主机上分别执行如下命令)$ hdfs --daemon start zkfc# 验证ha(在node-1节点停掉namenode进程)$ hafs --daemon stop namenoderesourceManager HA 配置# 在 RM1 启动 YARN (在 node-1 上执行)$ yarn --daemon start resourcemanager# 在 RM2 启动 YARN (在 node-2 上执行)$ yarn --daemon start resourcemanager# 在任意节点执行获取resourceManager状态（active）$ yarn rmadmin -getServiceState rm1# 在任意节点执行获取resourceManager状态（standby）$ yarn rmadmin -getServiceState rm2# 验证 yarn 的 ha（在node-1节点执行）standby 的 resourcemanager 则会转换为 active$ yarn --daemon stop resourcemanager# 在任意节点执行获取resourceManager状态（active）$ yarn rmadmin -getServiceState rm2总结搭建hadoop HA 过程中遇到了很多各种各样的问题上述步骤都是经过验证的如在安装过程中遇到问题可以留言，谢谢！ ...

Rancher提供了两种安装方法，即单节点安装和高可用安装。单节点安装可以让用户快速部署适用于短期开发或PoC的Rancher 2.x，而高可用部署则明显更适合Rancher的长期部署。要点须知针对开源用户，对于从单个节点迁移到HA的工作，Rancher Labs不提供官方技术支持。以防在此过程中出现问题，您应该熟悉Rancher架构以及故障排除的方法。前期准备为了顺利将单个节点Rancher安装迁移到高可用性安装，您必须做如下准备：您需要运行Rancher的2.1.x版本以及RKE 的0.1.x版本server-url参数必须是可以被更改为指向HA Rancher安装的DNS名称。如果您的server-url是IP地址，则必须先将server-url更改为DNS名称并更新用户集群的节点/集群代理，以便在迁移后允许HA集群的nginx ingress控制器能够正确路由Rancher流量。否则，您将无法访问所有工作负载集群。您需要配置专门用于在HA中运行Rancher的新实例。因为执行就地迁移非常危险，并且没有回滚策略。我们强烈建议您参考官方建议的架构。(https://rancher.com/docs/ranc… ）您需要熟悉单节点安装和高可用性安装之间的体系结构差异。不要更改单节点安装的CA。换言之，如果您已经在使用自定义证书了，那么请继续使用它。迁移教程从Rancher的单个节点迁移到高可用性安装的过程可以总结为以下几个步骤：在Rancher单节点实例上1、 备份Rancher单节点容器2、 使用单节点容器中找到的证书组装pki.bundle.tar.gz3、 运行临时utility容器以执行容器内运行的嵌入式etcd的etcd快照4、 停止旧的Rancher单节点容器5、 将server-url的DNS记录更改为指向新的HA负载均衡器在您的工作站或者bastion host上1、 将生成的pki.bundle.tar.gz和single-node-etcd-snapshot从Rancher单节点实例传输到工作目录中的工作站上（工作目录原先最好是空的）2、 生成指向新HA节点的rke集群配置文件3、 rke etcd snapshot-restore从单节点容器中检索快照4、 rke up5、 根据文档在HA中安装Rancher开始之前在整个教程中，您将输入一系列命令，以将您环境中的数据替换为占位符。这些占位符用斜角括号和所有大写字母（<EXAMPLE>）表示。下表是找到本教程中使用的各种占位符。请您在开始之前记下此信息，这将有助于您后续的操作。在Rancher单节点实例上步骤1 备份Rancher单节点容器首先，您应该备份Rancher单节点容器，以确保在迁移过程中遇到问题时可以回滚到此前运行的Rancher单节点。有关这方面的更多信息，请参阅：https://rancher.com/docs/ranc…步骤2 使用单节点容器中找到的证书组装pki.bundle.tar.gz首先，键入docker exec，进入Rancher容器：[root@single-node-rancher ~]# docker exec -it <RANCHER_CONTAINER_NAME> /bin/bash进入容器后，将/ etc / kubernetes / ssl目录tar到pki.bundle.tar.gz：root@9f4b1729d8ca:/var/lib/rancher# tar -zcvf pki.bundle.tar.gz /etc/kubernetes/sslroot@9f4b1729d8ca:/var/lib/rancher# exit将刚刚生成的pki.bundle.tar.gz转移到单节点实例上的当前工作目录：[root@single-node-rancher ~]# docker cp <RANCHER_CONTAINER_NAME>:/var/lib/rancher/pki.bundle.tar.gz .步骤3 运行临时utility容器以执行容器内运行的嵌入式etcd的etcd快照docker运行一个名为etcd-utility的临时rke-tools容器，并附加了Rancher容器网络。此外，还挂载当前工作目录（将pki.bundle.tar.gz放入其中：docker run –net=container:<RANCHER_CONTAINER_NAME> -it -v $(pwd):/cwd –name etcd-utility rancher/rke-tools:v0.1.20在此容器中创建一个名为ssl的文件夹，然后将pki.bundle.tar.gz解压到：bash-4.4# mkdir ssl && cd sslbash-4.4# cp /cwd/pki.bundle.tar.gz .bash-4.4# tar -zxvf pki.bundle.tar.gz –strip-components 3将单节点etcd快照到一个名为single-node-etcd-snapshot的文件中bash-4.4# cd /bash-4.4# ETCDCTL_API=3 etcdctl snapshot save –cacert=/ssl/kube-ca.pem –cert=/ssl/kube-etcd-127-0-0-1.pem –key=/ssl/kube-etcd-127-0-0-1-key.pem single-node-etcd-snapshotbash-4.4# exit将etcd快照从etcd-utility容器复制到当前工作目录[root@single-node-rancher ~]# docker cp etcd-utility:/single-node-etcd-snapshot .单节点实例上的当前工作目录应包含两个文件：pki.bundle.tar.gz和single-node-etcd-snapshot。这是将Rancher从单节点迁移到HA所需的两个组件。步骤4 停止旧的Rancher单节点容器[root@single-node-rancher ~]# docker stop <RANCHER_CONTAINER_NAME>步骤5 将server-url的DNS记录更改为指向新的HA负载均衡器为了正确迁移Rancher，您应该更新DNS基础结构中的DNS记录，以将Rancher server-url指向新的HA负载均衡器。在您的工作站或者bastion host上将生成的pki.bundle.tar.gz和single-node-etcd-snapshot从Rancher单节点实例传输到工作目录中的工作站上（工作目录最好原先就是空的）Endeavor:single-node-to-ha-migration chriskim$ scp root@<RANCHER_SINGLE_NODE_HOST>:/root/pki.bundle.tar.gz .Endeavor:single-node-to-ha-migration chriskim$ scp root@<RANCHER_SINGLE_NODE_HOST>:/root/single-node-etcd-snapshot .生成指向您的新HA节点的rke集群配置文件举个例子，rancher-cluster.yml文件如下所示：nodes: - address: <RANCHER_HA_HOST_1> user: centos role: [controlplane,worker,etcd]services: etcd: snapshot: true creation: 6h retention: 24h在所需的HA节点上创建/ opt / rke / etcd-snapshots目录，并将single-node-etcd-snapshot和pki.bundle.tar.gz文件复制到该目录：Endeavor:single-node-to-ha-migration chriskim$ ssh root@<RANCHER_HA_HOST_1> “mkdir -p /opt/rke/etcd-snapshots"Endeavor:single-node-to-ha-migration chriskim$ scp pki.bundle.tar.gz root@<RANCHER_HA_HOST_1>:/opt/rke/etcd-snapshotsEndeavor:single-node-to-ha-migration chriskim$ scp single-node-etcd-snapshot root@<RANCHER_HA_HOST_1>:/opt/rke/etcd-snapshots使用RKE将 single-node- etcd-snapshot恢复到新的HA节点：rke etcd snapshot-restore –name single-node-etcd-snapshot –config rancher-cluster.yml完整命令行如下：INFO[0000] Starting restoring snapshot on etcd hostsINFO[0000] [dialer] Setup tunnel for host [ha-rancher01.fmt01.rancher.com]INFO[0000] [hosts] Cleaning up host [ha-rancher01.fmt01.rancher.com]INFO[0000] [hosts] Running cleaner container on host [ha-rancher01.fmt01.rancher.com]INFO[0000] [kube-cleaner] Pulling image [rancher/rke-tools:v0.1.15] on host [ha-rancher01.fmt01.rancher.com]INFO[0004] [kube-cleaner] Successfully pulled image [rancher/rke-tools:v0.1.15] on host [ha-rancher01.fmt01.rancher.com]INFO[0004] [kube-cleaner] Successfully started [kube-cleaner] container on host [ha-rancher01.fmt01.rancher.com]INFO[0004] [hosts] Removing cleaner container on host [ha-rancher01.fmt01.rancher.com]INFO[0004] [hosts] Removing dead container logs on host [ha-rancher01.fmt01.rancher.com]INFO[0005] [cleanup] Successfully started [rke-log-cleaner] container on host [ha-rancher01.fmt01.rancher.com]INFO[0005] [remove/rke-log-cleaner] Successfully removed container on host [ha-rancher01.fmt01.rancher.com]INFO[0005] [hosts] Successfully cleaned up host [ha-rancher01.fmt01.rancher.com]INFO[0005] [etcd] Restoring [single-node-etcd-snapshot] snapshot on etcd host [ha-rancher01.fmt01.rancher.com]INFO[0005] [etcd] Pulling image [rancher/coreos-etcd:v3.2.18] on host [ha-rancher01.fmt01.rancher.com]INFO[0007] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.18] on host [ha-rancher01.fmt01.rancher.com]INFO[0007] [etcd] Successfully started [etcd-restore] container on host [ha-rancher01.fmt01.rancher.com]INFO[0007] [etcd] Building up etcd plane..INFO[0007] [etcd] Successfully started [etcd] container on host [ha-rancher01.fmt01.rancher.com]INFO[0007] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [ha-rancher01.fmt01.rancher.com]INFO[0007] [etcd] Successfully started [etcd-rolling-snapshots] container on host [ha-rancher01.fmt01.rancher.com]INFO[0012] [certificates] Successfully started [rke-bundle-cert] container on host [ha-rancher01.fmt01.rancher.com]INFO[0013] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [ha-rancher01.fmt01.rancher.com]INFO[0013] [etcd] Successfully started [rke-log-linker] container on host [ha-rancher01.fmt01.rancher.com]INFO[0013] [remove/rke-log-linker] Successfully removed container on host [ha-rancher01.fmt01.rancher.com]INFO[0013] [etcd] Successfully started etcd plane..INFO[0013] [certificates] Successfully started [rke-bundle-cert] container on host [ha-rancher01.fmt01.rancher.com]INFO[0013] [certificates] successfully extracted certificate bundle on host [ha-rancher01.fmt01.rancher.com] to backup path [/etc/kubernetes/.tmp/]INFO[0013] Finished restoring snapshot [single-node-etcd-snapshot] on all etcd hosts此时，您可以运行rke up –config rancher-cluster.yml，并进行其他HA安装步骤。强烈建议您立即更改Rancher的DNS记录，以便在根据HA说明安装后，您的用户集群可以连接回Rancher。因此，您可能需要两次运行rke up –config rancher-cluster.yml以确保所有插件运行都成功。在成功安装Kubernetes并将Rancher单节点备份恢复到您的实例后，您可以将其他HA实例添加到rancher-cluster.yml并运行rke up –config rancher-cluster.yml以将这些节点添加到您的集群。此时，您可以使用此处的文档继续安装Rancher：https://rancher.com/docs/ranc…清 理迁移成功之后就可以开始进行清理工作了。您可以从单个节点实例中移除Rancher容器，也可以直接完全删除实例。[root@single-node-rancher ~]# docker rm <RANCHER_CONTAINER_NAME>回 滚如果迁移不成功，可以通过以下两个条件实现回滚：将server-url的DNS条目更改回单节点Rancher实例；在Rancher单节点实例上启动<RANCHER_CONTAINER_NAME>。[root@single-node-rancher ~]# docker start <RANCHER_CONTAINER_NAME> ...