OceanBase 环境根本都会先装置 OCP 来部署、监控、运维数据库集群。但如果有机器过保等问题，就须要有安稳的 OCP 节点的替换计划。
作者：张瑞远
上海某公司 DBA，已经从事银行、证券数仓设计、开发、优化类工作，现次要从事电信级 IT 零碎及数据库工作。有三年以上 OceanBase 工作教训。取得的专业技能与认证包含 OceanBase OBCP、Oracle OCP 11g、OracleOCM 11g 、MySQL OCP 5.7。
本文起源：原创投稿
- 爱可生开源社区出品，原创内容未经受权不得随便应用，转载请分割小编并注明起源。
前言
OceanBase 云平台（OceanBase Cloud Platform，OCP），是以 OceanBase 为外围的企业级数据库治理平台。
咱们生产环境根本都是须要先创立 OCP 平台，而后依赖 OCP 去创立及治理监控生产集群，所以装置 OCP 个别是零碎上线的第一步。之后可能随着机房布局等问题，就会有须要搬迁或者替换 OCP 的机器的需要。
别离介绍两种 OCP 节点的替换办法。一种是应用 OAT 平台（OceanBase Admin Toolkit，管理者工具）来替换；另一种就是应用 ANTMAN 工具替换。上次咱们介绍了第一种 OAT 的计划，本文介绍第二种。
PS：我的环境的 OCP 负载平衡应用的 F5，所以新的机器须要先配置 F5，其余负载平衡场景同理。
环境背景
大家如果有接触 OB 生产环境的教训的话，可能会理解，后期版本在装置 OCP 的时候，须要装置 OCP 软件/metadb/obproxy 三个 Docker 包，前期 OCP 版本将 DB+Proxy 集成在了一个 Docker 包里，OAT 的话只能纳管 DB + Proxy。集成的 metadb，离开的状况还须要应用 ANTMAN 工具来替换。
软件信息
| 软件 | 版本 |
|---|---|
| OCP | ocp-all-in-one:3.3.3-20220906114643 |
| metadb+proxy | OB2277_OBP320_x86_20220429 |
| Proxy | 4.1.1_20230519_x86 |
| antman | t-oceanbase-antman-1.4.3-20220807073355.alios7.x86_64 |
操作过程
3.1 环境查看/筹备
查看替换机器环境，包含分盘，创立 admin 用户，装置 Docker 软件等。装置好后查看下。
cd /root/t-oceanbase-antman/clonescripts/sh precheck.sh -m ocp登录 meta 库查看有没有 tenant 的主 ZONE 在要被替换的节点，提前切主。
MySQL [oceanbase]> select * from __all_Server;+----------------------------+----------------------------+--------------+----------+----+----------------+------------+-----------------+--------+-----------------------+--------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |+----------------------------+----------------------------+--------------+----------+----+----------------+------------+-----------------+--------+-----------------------+--------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+| 2022-03-17 22:59:19.979627 | 2023-03-20 10:27:01.147283 | 10.10.100.87 | 2882 | 6 | META_OB_ZONE_2 | 2881 | 0 | active | 0 | 2.2.76_20210406232249-a1e144bdc179fbf473cea37f199e8a76c736b8d4(Apr 6 2021 23:55:12) | 0 | 1679279220991796 | 0 | 1 | 1679278517144838 || 2022-03-17 23:54:49.277939 | 2023-03-20 09:29:22.079578 | 10.10.100.9 | 2882 | 7 | META_OB_ZONE_1 | 2881 | 1 | active | 0 | 2.2.76_20210406232249-a1e144bdc179fbf473cea37f199e8a76c736b8d4(Apr 6 2021 23:55:12) | 0 | 1679275725595691 | 0 | 1 | 0 || 2021-12-21 22:44:16.476503 | 2023-03-20 09:29:22.080425 | 122.44.11.2 | 2882 | 5 | META_OB_ZONE_3 | 2881 | 0 | active | 0 | 2.2.76_20210406232249-a1e144bdc179fbf473cea37f199e8a76c736b8d4(Apr 6 2021 23:55:12) | 0 | 1640097866698859 | 0 | 1 | 0 |+----------------------------+----------------------------+--------------+----------+----+----------------+------------+-----------------+--------+-----------------------+--------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+MySQL [oceanbase]> select tenant_name,primary_zone from __all_tenant;+----------------+----------------------------------------------+| tenant_name | primary_zone |+----------------+----------------------------------------------+| sys | META_OB_ZONE_1;META_OB_ZONE_3;META_OB_ZONE_2 || ocp_meta | META_OB_ZONE_1;META_OB_ZONE_3;META_OB_ZONE_2 || ocp_monitor | META_OB_ZONE_1;META_OB_ZONE_3;META_OB_ZONE_2 || oms_tt_tenant | META_OB_ZONE_1;META_OB_ZONE_3;META_OB_ZONE_2 || oms_cc7_tenant | META_OB_ZONE_3;META_OB_ZONE_2;META_OB_ZONE_1 || oms_ff9_tenant | META_OB_ZONE_1;META_OB_ZONE_2,META_OB_ZONE_3 || oms_cc9_tenant | META_OB_ZONE_3;META_OB_ZONE_1,META_OB_ZONE_2 || oms_dd_tenant | META_OB_ZONE_3;META_OB_ZONE_1,META_OB_ZONE_2 || obdw_meta | META_OB_ZONE_3;META_OB_ZONE_1,META_OB_ZONE_2 |+----------------+----------------------------------------------+MySQL [oceanbase]> alter tenant sys primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';Query OK, 0 rows affected (0.04 sec)MySQL [oceanbase]> alter tenant ocp_meta primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';Query OK, 0 rows affected (1.27 sec)MySQL [oceanbase]> alter tenant ocp_monitor primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';Query OK, 0 rows affected (0.02 sec)MySQL [oceanbase]> alter tenant oms_tt_tenant primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';Query OK, 0 rows affected (0.03 sec)MySQL [oceanbase]> alter tenant oms_ff9_tenant primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';Query OK, 0 rows affected (0.03 sec)因为应用 ANTMAN 工具迁徙，须要在执行机器上批改 obcluster.conf 文件，或者间接从原 OCP 上 copy 后查看下，镜像包也须要传到该机器 /root/t-oceanbase-antman 目录下。
55obffocp:~/t-oceanbase-antman # cat obcluster.confZONE1_RS_IP=10.10.100.9ZONE2_RS_IP=10.10.100.87ZONE3_RS_IP=122.44.11.2###### 主动配置，无需批改 / AUTO-CONFIGURATION ######OBSERVER01_HOSTNAME=OCP_META_SERVER_1OBSERVER02_HOSTNAME=OCP_META_SERVER_2OBSERVER03_HOSTNAME=OCP_META_SERVER_3ZONE1_NAME=META_OB_ZONE_1 --后续命令参数，次要和参数文件中对上ZONE2_NAME=META_OB_ZONE_2ZONE3_NAME=META_OB_ZONE_3##there must be more than half zone within same regionZONE1_REGION=OCP_META_REGIONZONE2_REGION=OCP_META_REGIONZONE3_REGION=OCP_META_REGIONMYSQL_PORT=2881RPC_PORT=2882OCP_VERSION=3.3.3查看执行 ANTMAN 脚本机器上默认集群明码是否正确。cd ~/t-oceanbase-antman/tools，执行 getpass.sh 的脚本，如果不对须要应用 setpass.sh 批改，因为后续 Proxy 的 Docker 迁徙后会有验证，OCP 的 Docker 迁徙前也会验证。
55obffocp:~/t-oceanbase-antman/tools # bash setpass.sh -s 0Aa255yK^F password file sys in /root/.key already exist!**********************Password of root@sys is CqVgg9}AutPassword of root@ocp_meta is r6kS^EINTUPassword of root@ocp_monitor is pkJv1a{7J7Password of root@odc is j{fjdd3X9fPassword of root@oms is {oOIsE9fdQ55obffocp:~ # mv .key .key_bak55obffocp:~ # cd /root/t-oceanbase-antman/tools/55obffocp:~/t-oceanbase-antman/tools # bash setpass.sh -s 0Aa255yK^F**********************Password of root@sys is 0Aa255yK^FPassword of root@ocp_meta is Password of root@ocp_monitor is Password of root@odc is Password of root@oms is 55obffocp:~/t-oceanbase-antman/tools # bash setpass.sh -c rSf@jO%6EO**********************Password of root@sys is 0Aa255yK^FPassword of root@ocp_meta is rSf@jO%6EOPassword of root@ocp_monitor is Password of root@odc is Password of root@oms is3.2 增加新机器
执行 ANTMAN 的 manage 脚本进行新机器的增加。
PS：这个版本 manage 会有报错，文末会有分享。
55obffocp:~/t-oceanbase-antman # ./manage.sh -i ob,ocp,obproxy -l 133.55.22.19 -z 1 -R Jnydzycscc@123 -A OceanBase#123[2023-06-16 16:31:45.375633] INFO [check conf file /root/t-oceanbase-antman/obcluster.conf format ...][2023-06-16 16:31:45.381844] INFO [conf file is upper case format.][2023-06-16 16:31:45.391446] INFO [SSH_AUTH=password SSH_USER=root SSH_PORT=22 SSH_PASSWORD= SSH_KEY_FILE=/root/.ssh/id_rsa]LB_MODE=f5INSTALL_COMPONENTS componets: ob obproxy ocpCLEAR_COMPONENTS: IP_LIST: 133.55.22.19ZONE_LIST: 1ROOT_PASSWORD_LIST: Jnydzycscc@123ADMIN_PASSWORD_LIST: OceanBase#123[2023-06-16 16:31:45.746503] INFO [INSTALL_COMPONENT: ob START ######################################][2023-06-16 16:31:45.751057] INFO [deploy_ob: check whether OBSERVER port 2881,2882 are in use or not on 133.55.22.19][2023-06-16 16:31:45.806500] INFO [deploy_ob: OBSERVER port 2881,2882 are idle on 133.55.22.19][2023-06-16 16:31:45.810773] INFO [deploy_ob: installing ob cluster, logfile: /root/t-oceanbase-antman/logs/deploy_ob.log]cp: '/root/t-oceanbase-antman/OB2276_x86_20210409.tar.gz' and '/root/t-oceanbase-antman/OB2276_x86_20210409.tar.gz' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/install_OB_docker.sh' and '/root/t-oceanbase-antman/install_OB_docker.sh' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/obcluster.conf' and '/root/t-oceanbase-antman/obcluster.conf' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/common/utils.sh' and '/root/t-oceanbase-antman/common/utils.sh' are the same fileskip copy same filecp: '/root/.key' and '/root/.key' are the same fileskip copy same filenohup: ignoring input[2023-06-16 16:31:45.841348] INFO [installing OB docker and starting OB server on 133.55.22.19, pid: 144513, log: /root/t-oceanbase-antman/logs/install_OB_docker.log and /home/admin/logs/ob-server/ inside docker][2023-06-16 16:31:45.925592] INFO [load docker image: docker load -i /root/t-oceanbase-antman/OB2276_x86_20210409.tar.gz][2023-06-16 16:31:45.930723] INFO [install_OB_docker.sh is still running on 133.55.22.19][2023-06-16 16:31:56.021465] INFO [install_OB_docker.sh is still running on 133.55.22.19]Loaded image: reg.docker.alibaba-inc.com/antman/ob-docker:OB2276_x86_20210409[2023-06-16 16:32:06.111458] INFO [install_OB_docker.sh is still running on 133.55.22.19][2023-06-16 16:32:06.359285] INFO [start container: docker run -d -it --cap-add SYS_RESOURCE --name META_OB_ZONE_1 --net=host -e OBCLUSTER_NAME=obcluster -e DEV_NAME=bond0 -e ROOTSERVICE_LIST="10.10.100.9:2882:2881;10.10.100.87:2882:2881;122.44.11.2:2882:2881" -e DATAFILE_DISK_PERCENTAGE=90 -e CLUSTER_ID=1632654636 -e ZONE_NAME=META_OB_ZONE_1 -e OBPROXY_PORT=2883 -e MYSQL_PORT=2881 -e RPC_PORT=2882 -e OCP_VIP=134.80.173.57 -e OCP_VPORT=80 -e app.password_root='Jnydzycscc@123' -e app.password_admin='OceanBase#123' -e OBPROXY_OPTSTR="" -e OPTSTR="cpu_count=64,system_memory=50G,memory_limit=254G,__min_full_resource_pool_memory=1073741824,_ob_enable_prepared_statement=false,memory_limit_percentage=90" --cpu-period 100000 --cpu-quota 6400000 --cpuset-cpus 0-63 --memory 256G -v /home/admin/oceanbase:/home/admin/oceanbase -v /data/log1:/data/log1 -v /data/1:/data/1 --restart on-failure:5 reg.docker.alibaba-inc.com/antman/ob-docker:OB2276_x86_20210409]WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.4f1c15e8194cc1ae2fffcc124ea9c982b3fda87ce1a6d0038db88435c737af89[2023-06-16 16:32:16.209761] INFO [install_OB_docker.sh finished and reg.docker.alibaba-inc.com/antman/ob-docker:OB2276_x86_20210409 started on 133.55.22.19][2023-06-16 16:32:16.214771] INFO [waiting on observer ready on 133.55.22.19][2023-06-16 16:35:16.244133] INFO [waiting on observer ready on 133.55.22.19 for 3 Minitues][2023-06-16 16:36:16.264776] INFO [waiting on observer ready on 133.55.22.19 for 4 Minitues][2023-06-16 16:37:16.285808] INFO [waiting on observer ready on 133.55.22.19 for 5 Minitues][2023-06-16 16:37:16.579057] INFO [observer on 133.55.22.19 is ready][2023-06-16 16:37:16.584583] INFO [deploy_ob: installation of ob cluster done][2023-06-16 16:37:16.588617] INFO [INSTALL_COMPONENT: ob DONE ######################################][2023-06-16 16:37:16.593604] INFO [INSTALL_COMPONENT: obproxy START ######################################]这里日志太多，就不都粘贴进去了，能够从下面看到 metadb 的 Docker 服务增加完后开始了 OBProxy 的 Docker 服务增加。
过程阐明
133.55.22.19是要去替换 OCP 的服务器的理论物理 IP。-z 1选项，指定的133.55.22.19会被增加到 OCP 环境中的第 1 个 ZONE，即和上文查到的10.10.100.9机器在同一个 ZONE 里。这里对于 ZONE 的定义，次要是针对 OCP 服务器上的
meta_ob docker而言，obproxy docker 和 ocp docker 并没有 ZONE 的概念。对于每台 OCP 服务器上的 meta_ob docker 属于哪一个 ZONE，请参考
obcluster.conf配置文件中的三个变量：ZONE1_RS_IPZONE2_RS_IPZONE3_RS_IP
-R和-A，前面须要别离填写成133.55.22.19服务器的root用户明码和admin用户明码。-i是装置，如果替换成-c就是革除。
这时候失常的话能够通过新增加节点的 IP:8080 前台登录 OCP，也能够通过这个机器的 2883 端口去连 meta 库了。
3.3 新增租户
登录 OCP 的 metadb 的 sys 租户新增 meta_ob Docker 的上线。
MySQL [oceanbase]> alter system add server '133.55.22.19:2882' zone 'META_OB_ZONE_1';Query OK, 0 rows affected (0.02 sec)MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;+--------------+----------------+-----------------+--------+--------------------+| svr_ip | zone | with_rootserver | status | start_service_time |+--------------+----------------+-----------------+--------+--------------------+| 10.10.100.87 | META_OB_ZONE_2 | 1 | active | 1679279220991796 || 10.10.100.9 | META_OB_ZONE_1 | 0 | active | 1679275725595691 || 122.44.11.2 | META_OB_ZONE_3 | 0 | active | 1640097866698859 || 133.55.22.19 | META_OB_ZONE_1 | 0 | active | 0 |+--------------+----------------+-----------------+--------+--------------------+4 rows in set (0.00 sec)MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;+--------------+----------------+-----------------+--------+--------------------+| svr_ip | zone | with_rootserver | status | start_service_time |+--------------+----------------+-----------------+--------+--------------------+| 10.10.100.87 | META_OB_ZONE_2 | 1 | active | 1679279220991796 || 10.10.100.9 | META_OB_ZONE_1 | 0 | active | 1679275725595691 || 122.44.11.2 | META_OB_ZONE_3 | 0 | active | 1640097866698859 || 133.55.22.19 | META_OB_ZONE_1 | 0 | active | 1686908200404755 |+--------------+----------------+-----------------+--------+--------------------+4 rows in set (0.01 sec)3.4 替换下线
登录 OCP 的 metadb 的 sys 租户将被替换 meta_ob Docker 的下线。
MySQL [oceanbase]> alter system delete server '10.10.100.9:2882' zone 'META_OB_ZONE_1';Query OK, 0 rows affected (0.19 sec)MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;+--------------+----------------+-----------------+----------+--------------------+| svr_ip | zone | with_rootserver | status | start_service_time |+--------------+----------------+-----------------+----------+--------------------+| 10.10.100.87 | META_OB_ZONE_2 | 1 | active | 1679279220991796 || 10.10.100.9 | META_OB_ZONE_1 | 0 | deleting | 1679275725595691 || 122.44.11.2 | META_OB_ZONE_3 | 0 | active | 1640097866698859 || 133.55.22.19 | META_OB_ZONE_1 | 0 | active | 1686908200404755 |+--------------+----------------+-----------------+----------+--------------------+4 rows in set (0.01 sec)MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;+--------------+----------------+-----------------+--------+--------------------+| svr_ip | zone | with_rootserver | status | start_service_time |+--------------+----------------+-----------------+--------+--------------------+| 10.10.100.87 | META_OB_ZONE_2 | 1 | active | 1679279220991796 || 122.44.11.2 | META_OB_ZONE_3 | 0 | active | 1640097866698859 || 133.55.22.19 | META_OB_ZONE_1 | 0 | active | 1686908200404755 |+--------------+----------------+-----------------+--------+--------------------+3.5 更新服务器信息
登录 ocp_meta 租户，手工更新 OCP 服务器信息。
后面步骤解决完，OCP 前台还能够看到残留的信息，须要替换下信息。
55obffocp:~/t-oceanbase-antman # mysql -h10.10.100.87 -P2883 -uroot@ocp_meta#obcluster -p'rSf@jO%6EO' -Docp -cMySQL [ocp]> select * from compute_host where inner_ip_address='10.10.100.9'\G*************************** 1. row *************************** id: 1 name: ocp1a description: NULLoperating_system: 4.12.14-120-default architecture: x86_64inner_ip_address: 10.10.100.9 ssh_port: 2022 kind: DEDICATED_PHYSICAL_MACHINE publish_ports: NULL status: ONLINE vpc_id: 1 idc_id: 1 host_type_id: 1 serial_number: NULL alias: NULL create_time: 2021-09-26 21:04:11 update_time: 2023-03-20 11:01:581 row in set (0.00 sec)MySQL [ocp]> update compute_host set inner_ip_address='133.55.22.19', name='55obffocp' where inner_ip_address='10.10.100.9';Query OK, 1 row affected (0.01 sec)Rows matched: 1 Changed: 1 Warnings: 0MySQL [ocp]> select * from compute_host where id =1;+----+-----------+-------------+---------------------+--------------+------------------+----------+----------------------------+---------------+--------+--------+--------+--------------+---------------+-------+---------------------+---------------------+| id | name | description | operating_system | architecture | inner_ip_address | ssh_port | kind | publish_ports | status | vpc_id | idc_id | host_type_id | serial_number | alias | create_time | update_time |+----+-----------+-------------+---------------------+--------------+------------------+----------+----------------------------+---------------+--------+--------+--------+--------------+---------------+-------+---------------------+---------------------+| 1 | 55obffocp | NULL | 4.12.14-120-default | x86_64 | 133.55.22.19 | 2022 | DEDICATED_PHYSICAL_MACHINE | NULL | ONLINE | 1 | 1 | 1 | NULL | NULL | 2021-09-26 21:04:11 | 2023-06-16 17:47:19 |+----+-----------+-------------+---------------------+--------------+------------------+----------+----------------------------+---------------+--------+--------+--------+--------------+---------------+-------+---------------------+---------------------+1 row in set (0.00 sec)3.6 清理被替换机器上残留的服务
ocp1a:~/t-oceanbase-antman # ./manage.sh -c ob,ocp,obproxy -l 10.10.100.9 -z 1 -R 'Dt!n(Rg4Av!t' -A OceanBase#123grep: /etc/system-release: No such file or directory[2023-06-16 22:45:44.101400] INFO [check conf file /root/t-oceanbase-antman/obcluster.conf format ...][2023-06-16 22:45:44.106779] INFO [conf file is upper case format.][2023-06-16 22:45:44.114437] INFO [SSH_AUTH=password SSH_USER=root SSH_PORT=22 SSH_PASSWORD= SSH_KEY_FILE=/root/.ssh/id_rsa]LB_MODE=f5INSTALL_COMPONENTS componets: CLEAR_COMPONENTS: ob obproxy ocpIP_LIST: 10.10.100.9ZONE_LIST: 1ROOT_PASSWORD_LIST: Dt!n(Rg4Av!tADMIN_PASSWORD_LIST: OceanBase#123[2023-06-16 22:45:44.474268] INFO [CLEAR_COMPONENT: ob START ######################################]cp: '/root/t-oceanbase-antman/uninstall.sh' and '/root/t-oceanbase-antman/uninstall.sh' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/obcluster.conf' and '/root/t-oceanbase-antman/obcluster.conf' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/common/utils.sh' and '/root/t-oceanbase-antman/common/utils.sh' are the same fileskip copy same filegrep: /etc/system-release: No such file or directory[2023-06-16 22:45:44.504069] INFO [remove OB server and docker on host: 10.10.100.9][2023-06-16 22:45:44.548697] INFO [docker rm -f 62ab623cb4ed]62ab623cb4ed[2023-06-16 22:46:01.260706] INFO [remove OB server and docker on host: 10.10.100.9 done!][2023-06-16 22:46:01.370808] INFO [uninstall.sh ob finished and reg.docker.alibaba-inc.com/antman/ob-docker:OB2276_x86_20210409 removed on 10.10.100.9][2023-06-16 22:46:01.375914] INFO [OB docker on 10.10.100.9 is removed][2023-06-16 22:46:01.380667] INFO [CLEAR_COMPONENT: ob DONE ######################################][2023-06-16 22:46:01.385398] INFO [CLEAR_COMPONENT: obproxy START ######################################]cp: '/root/t-oceanbase-antman/uninstall.sh' and '/root/t-oceanbase-antman/uninstall.sh' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/obcluster.conf' and '/root/t-oceanbase-antman/obcluster.conf' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/common/utils.sh' and '/root/t-oceanbase-antman/common/utils.sh' are the same fileskip copy same filegrep: /etc/system-release: No such file or directory[2023-06-16 22:46:01.416495] INFO [remove obproxy docker on host:10.10.100.9][2023-06-16 22:46:01.514215] INFO [docker rm -f 01bdcadf2e11]01bdcadf2e11[2023-06-16 22:46:01.765459] INFO [remove obproxy docker on host:10.10.100.9 done!][2023-06-16 22:46:01.858848] INFO [uninstall.sh obproxy finished and reg.docker.alibaba-inc.com/antman/obproxy:OBP186_20210315 removed on 10.10.100.9][2023-06-16 22:46:01.863806] INFO [obproxy docker on 10.10.100.9 is removed][2023-06-16 22:46:01.868778] INFO [CLEAR_COMPONENT: obproxy DONE ######################################][2023-06-16 22:46:01.873368] INFO [CLEAR_COMPONENT: ocp START ######################################]cp: '/root/t-oceanbase-antman/uninstall.sh' and '/root/t-oceanbase-antman/uninstall.sh' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/obcluster.conf' and '/root/t-oceanbase-antman/obcluster.conf' are the same fileskip copy same filecp: '/root/t-oceanbase-antman/common/utils.sh' and '/root/t-oceanbase-antman/common/utils.sh' are the same fileskip copy same filegrep: /etc/system-release: No such file or directory[2023-06-16 22:46:01.906934] INFO [remove ocp docker on host:10.10.100.9][2023-06-16 22:46:01.944811] INFO [docker rm -f 8b044744a92e]8b044744a92e[2023-06-16 22:46:26.162467] INFO [remove ocp docker on host:10.10.100.9 done][2023-06-16 22:46:26.253927] INFO [uninstall.sh ocp finished and reg.docker.alibaba-inc.com/oceanbase/ocp-all-in-one:3.3.3-20220906114643 removed on 10.10.100.9][2023-06-16 22:46:26.258281] INFO [ocp docker on 10.10.100.9 is removed][2023-06-16 22:46:26.263047] INFO [CLEAR_COMPONENT: ocp DONE ######################################]ocp1a:~/t-oceanbase-antman # docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES服务曾经被革除。
报错记录解决
manage 脚本执行报错。
55obffocp:~/t-oceanbase-antman # ./manage.sh -i ob,ocp,obproxy -l 133.55.22.19 -z 1 -R Jnydzycscc@123 -A OceanBase#123[2023-06-16 16:31:03.305062] INFO [check conf file /root/t-oceanbase-antman/obcluster.conf format ...][2023-06-16 16:31:03.309079] INFO [conf file is upper case format.][2023-06-16 16:31:03.315290] INFO [SSH_AUTH=password SSH_USER=root SSH_PORT=22 SSH_PASSWORD= SSH_KEY_FILE=/root/.ssh/id_rsa]LB_MODE=f5INSTALL_COMPONENTS componets: ob obproxy ocpCLEAR_COMPONENTS: IP_LIST: 133.55.22.19ZONE_LIST: 1ROOT_PASSWORD_LIST: Jnydzycscc@123ADMIN_PASSWORD_LIST: OceanBase#123/root/t-oceanbase-antman/common/utils.sh: line 484: -e: command not found[2023-06-16 16:31:03.636862] ERROR [: ssh authorization to 133.55.22.19 failed, Please check SSH affinity environment varialbes.]这个问题也须要批改脚本代码解决。
执行 alter system delete server 语句之后很久，被替换的 Server 没有 Delete 掉，始终是 deleting 状态，查看发现 OCP 的 meta 库内存参数调整过，新加的 Server 参数小，导致 UNIT 迁徙卡住。
MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;+--------------+----------------+-----------------+----------+--------------------+| svr_ip | zone | with_rootserver | status | start_service_time |+--------------+----------------+-----------------+----------+--------------------+| 10.10.100.87 | META_OB_ZONE_2 | 1 | active | 1679279220991796 || 10.10.100.9 | META_OB_ZONE_1 | 0 | deleting | 1679275725595691 || 122.44.11.2 | META_OB_ZONE_3 | 0 | active | 1640097866698859 || 133.55.22.19 | META_OB_ZONE_1 | 0 | active | 1686908200404755 |+--------------+----------------+-----------------+----------+--------------------+4 rows in set (0.01 sec)MySQL [oceanbase]> select count(*),svr_ip from gv$unit group by svr_ip;+----------+--------------+| count(*) | svr_ip |+----------+--------------+| 27 | 133.55.22.19 || 33 | 10.10.100.87 || 33 | 122.44.11.2 || 6 | 10.10.100.9 |+----------+--------------+4 rows in set (0.01 sec)MySQL [oceanbase]> select * from gv$unit where svr_ip='10.10.100.9';+---------+----------------+---------------------------------------------+------------------+----------------------------------------+----------------+-----------+-------------+-------------+----------+---------------------+-----------------------+---------+---------+-------------+-------------+----------+----------+---------------+-----------------+| unit_id | unit_config_id | unit_config_name | resource_pool_id | resource_pool_name | zone | tenant_id | tenant_name | svr_ip | svr_port | migrate_from_svr_ip | migrate_from_svr_port | max_cpu | min_cpu | max_memory | min_memory | max_iops | min_iops | max_disk_size | max_session_num |+---------+----------------+---------------------------------------------+------------------+----------------------------------------+----------------+-----------+-------------+-------------+----------+---------------------+-----------------------+---------+---------+-------------+-------------+----------+----------+---------------+-----------------+| 1106 | 1090 | config_oms_tt_tenant_META_OB_ZONE_1_S2_gpa | 1080 | pool_oms_tt_tenant_META_OB_ZONE_1_gpa | META_OB_ZONE_1 | NULL | NULL | 10.10.100.9 | 2882 | | 0 | 3 | 3 | 12884901888 | 12884901888 | 2500 | 2500 | 536870912000 | 750 || 1139 | 1094 | oms_unit | 1129 | oms_ff9_tenant_resource_pool | META_OB_ZONE_1 | NULL | NULL | 10.10.100.9 | 2882 | | 0 | 2 | 2 | 5368709120 | 4294967296 | 128 | 128 | 5368709120 | 10000 || 1122 | 1097 | config_oms_c55_tenant_META_OB_ZONE_1_S1_ifu | 1088 | pool_oms_c55_tenant_META_OB_ZONE_1_ifu | META_OB_ZONE_1 | NULL | NULL | 10.10.100.9 | 2882 | | 0 | 1.5 | 1.5 | 6442450944 | 6442450944 | 1250 | 1250 | 536870912000 | 375 || 1126 | 1100 | config_oms_ff6_tenant_META_OB_ZONE_1_S1_uzz | 1092 | pool_oms_ff6_tenant_META_OB_ZONE_1_uzz | META_OB_ZONE_1 | NULL | NULL | 10.10.100.9 | 2882 | | 0 | 1.5 | 1.5 | 6442450944 | 6442450944 | 1250 | 1250 | 536870912000 | 375 || 1127 | 1101 | config_oms_ff7_tenant_META_OB_ZONE_1_S1_gkj | 1093 | pool_oms_ff7_tenant_META_OB_ZONE_1_gkj | META_OB_ZONE_1 | NULL | NULL | 10.10.100.9 | 2882 | | 0 | 1.5 | 1.5 | 6442450944 | 6442450944 | 1250 | 1250 | 536870912000 | 375 || 1135 | 1108 | config_oms_cc8_tenant_META_OB_ZONE_1_S1_wwo | 1101 | pool_oms_cc8_tenant_META_OB_ZONE_1_wwo | META_OB_ZONE_1 | NULL | NULL | 10.10.100.9 | 2882 | | 0 | 1.5 | 1.5 | 6442450944 | 6442450944 | 1250 | 1250 | 536870912000 | 375 |+---------+----------------+---------------------------------------------+------------------+----------------------------------------+----------------+-----------+-------------+-------------+----------+---------------------+-----------------------+---------+---------+-------------+-------------+----------+----------+---------------+-----------------+6 rows in set (0.02 sec)MySQL [oceanbase]> alter system migrate unit=1106 destination='133.55.22.19:2882';ERROR 4624 (HY000): machine resource is not enough to hold a new unit ----------------手动去迁徙报资源有余MySQL [oceanbase]> select zone,svr_ip, cpu_total, cpu_assigned,cpu_assigned_percent cpu_ass_pct, round(mem_total/1024/1024/1024) mem_total_gb, -> round(mem_assigned/1024/1024/1024) mem_ass_gb, mem_assigned_percent mem_ass_pct, unit_num, migrating_unit_num, leader_count, round(`load`,2) `load` -> from __all_virtual_server_stat -> order by zone, svr_ip; ---------------------查看资源发现内存不足+----------------+--------------+-----------+--------------+-------------+--------------+------------+-------------+----------+--------------------+--------------+------+| zone | svr_ip | cpu_total | cpu_assigned | cpu_ass_pct | mem_total_gb | mem_ass_gb | mem_ass_pct | unit_num | migrating_unit_num | leader_count | load |+----------------+--------------+-----------+--------------+-------------+--------------+------------+-------------+----------+--------------------+--------------+------+| META_OB_ZONE_1 | 10.10.100.9 | 62 | 11 | 17 | 250 | 40 | 16 | 6 | 0 | 0 | 0.17 || META_OB_ZONE_1 | 133.55.22.19 | 62 | 48 | 77 | 204 | 196 | 96 | 27 | 0 | 0 | 0.87 || META_OB_ZONE_2 | 10.10.100.87 | 62 | 59 | 95 | 250 | 236 | 94 | 33 | 0 | 2935 | 0.95 || META_OB_ZONE_3 | 122.44.11.2 | 62 | 59 | 95 | 250 | 236 | 94 | 33 | 0 | 1051 | 0.95 |MySQL [oceanbase]> show parameters like '%memory_limit%' ;| META_OB_ZONE_3 | observer | 122.44.11.2 | 2882 | memory_limit | NULL | 300G | the size of the memory reserved for internal use(for testing purpose). Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE || META_OB_ZONE_1 | observer | 133.55.22.19 | 2882 | memory_limit | NULL | 254G | the size of the memory reserved for internal use(for testing purpose). Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE || META_OB_ZONE_2 | observer | 10.10.100.87 | 2882 | memory_limit | NULL | 300G | the size of the memory reserved for internal use(for testing purpose). Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE || META_OB_ZONE_1 | observer | 10.10.100.9 | 2882 | memory_limit | NULL | 300G | the size of the memory reserved for internal use(for testing purpose). Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |MySQL [oceanbase]> alter system set memory_limit ='300G' ;Query OK, 0 rows affected (0.05 sec)MySQL [oceanbase]> select count(*),svr_ip from gv$unit group by svr_ip;+----------+--------------+| count(*) | svr_ip |+----------+--------------+| 33 | 133.55.22.19 || 33 | 10.10.100.87 || 33 | 122.44.11.2 |+----------+--------------+3 rows in set (0.00 sec)MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;+--------------+----------------+-----------------+--------+--------------------+| svr_ip | zone | with_rootserver | status | start_service_time |+--------------+----------------+-----------------+--------+--------------------+| 10.10.100.87 | META_OB_ZONE_2 | 1 | active | 1679279220991796 || 122.44.11.2 | META_OB_ZONE_3 | 0 | active | 1640097866698859 || 133.55.22.19 | META_OB_ZONE_1 | 0 | active | 1686908200404755 |+--------------+----------------+-----------------+--------+--------------------+3 rows in set (0.00 sec)总结
到此，应用 ANTMAN 工具的形式去替换 OCP 机器的操作就完结了，包含后面一篇应用 OAT 替换 OCP 节点的文章可能看起来没什么难度，然而整个过程来回做了好几遍，充斥我的崎岖和泪水。为了他人当前少踩坑，所以写了这两篇文章分享。
如果看了上篇文章的话，应该晓得 OAT 替换 OCP 的时候，新加机器是在 metadb 中新创建了一个 ZONE，而后再把被替换机器下掉。其中还波及新建资源池批改 Locality，减少正本数等操作。其实应用 ANTMAN 工具的话这个步骤就不太一样，是将新机器退出到须要替换机器的同一个 ZONE 内，而后做同 ZONE 内 UNIT 的迁徙，而后把被替换的机器下线。
现阶段的话，相对来说应用 ANTMAN 工具替换之后对于 OCP 元数据的影响更小一些，然而 OAT 黑屏操作更少已些。对于 OBProxy 独自 Docker 的后期场景必须应用 ANTMAN，前期版本就看大家本人酌情抉择了。
行之所向，莫问远方。