乐趣区

关于oceanbase:技术分享-OceanBase-手滑误删了数据文件怎么办

jiezi

作者：张乾

外星人 2 号，现专任六位喵星人的资深铲屎官。

本文起源：原创投稿

* 爱可生开源社区出品，原创内容未经受权不得随便应用，转载请分割小编并注明起源。

手滑误删了数据文件，并且没有可替换的节点时，先别急着提桶跑路，能够思考利用参数 server_permanent_offline_time 来重建受影响的节点。

原理：

server_permanent_offline_time 是 OceanBase 数据库中用于管制节点永恒下线时长的参数。当集群中的某个节点宕机后，零碎会依据该参数的设置值来进行相应操作。

如果节点宕机工夫小于该参数设置的值，零碎会临时不做解决，以防止频繁的数据迁徙；如果宕机工夫超过该参数设置的值，该节点被标记为永恒下线，RootService 会将该 OBServer 上蕴含的数据正本从 Paxos 成员组中删除，并在同 zone 内其余可用 OBServer 上补充数据，以保证数据正本 Paxos 成员组残缺。该参数默认值是 3600 秒，个别设置较大，以防止不必要的正本复制。此外，当永恒下线的节点从新被拉起后，其上的全副数据都须要从其余正本从新拉取。

在本场景下，即是通过调低该参数，让故障节点疾速永恒下线再从新上线，达到数据重建的目标。

请留神，此过程会占用集群肯定的资源，可能会影响性能，因而倡议在业务低峰期进行。

官网倡议

对于 server_permanent_offline_time 的实用场景和倡议值，官网提供如下:

  1. OceanBase 数据库版本升级场景：倡议将该配置项的值设置为 72h。
  2. OBServer 硬件更换场景：倡议将该配置项的值设置为 4h。
  3. OBServer 清空上线场景：倡议将该配置项的值设置为 10m，使集群疾速上线。

筹备过程

准备一套环境

应用 OBD 工具疾速部署一套 3 节点 OB 以及一个 OBProxy，再创立好一个租户 sysbench_tenant，primary_zone 为 RANDOM。

注：本文基于 OB 3.1.2 版本，其余版本需注意另作验证。

筹备些数据

应用 sysbench 创立一个表 sbtest1 并插入 1W 数据。

sysbench ./oltp_insert.lua --mysql-host=10.186.60.3 --mysql-port=2883 --mysql-db=sysbenchdb --mysql-user="sysbench@sysbench_tenant"  --mysql-password=sysbench --tables=1 --table_size=10000 --threads=1 --time=600 --report-interval=10 --db-driver=mysql --db-ps-mode=disable --skip-trx=on --mysql-ignore-errors=6002,6004,4012,2013,4016,1062,5157,4038 prepare

这里改写了 sysbench 的建表语句，分了 3 个区，查问 sbtest1 表分区正本散布如下

MySQL [oceanbase]> select tenant.tenant_name, zone, svr_ip,svr_port, case when role=1 then 'leader' when role=2 then 'follower' else NULL end as role, count(1) as partition_cnt from __all_virtual_meta_table meta  inner join __all_tenant tenant  on meta.tenant_id=tenant.tenant_id inner join __all_virtual_table tab  on meta.tenant_id=tab.tenant_id and meta.table_id=tab.table_id where tenant.tenant_id=1001 and tab.table_name='sbtest1' group by  tenant.tenant_name,zone, svr_ip,svr_port, 5 order by  tenant.tenant_name, zone, svr_ip, role desc;
+-----------------+-------+--------------+----------+----------+---------------+
| tenant_name     | zone  | svr_ip       | svr_port | role     | partition_cnt |
+-----------------+-------+--------------+----------+----------+---------------+
| sysbench_tenant | zone1 | 10.186.64.74 |     2882 | leader   |             1 |
| sysbench_tenant | zone1 | 10.186.64.74 |     2882 | follower |             2 |
| sysbench_tenant | zone2 | 10.186.64.75 |     2882 | leader   |             1 |
| sysbench_tenant | zone2 | 10.186.64.75 |     2882 | follower |             2 |
| sysbench_tenant | zone3 | 10.186.64.79 |     2882 | leader   |             1 |
| sysbench_tenant | zone3 | 10.186.64.79 |     2882 | follower |             2 |
+-----------------+-------+--------------+----------+----------+---------------+

开始试验

应用 sysbench 继续写入数据，维持肯定的流量，便于在节点重建后比照各节点数据是否统一。

sysbench ./oltp_insert.lua --mysql-host=10.186.60.3 --mysql-port=2883 --mysql-db=sysbenchdb --mysql-user="sysbench@sysbench_tenant"  --mysql-password=sysbench --tables=1 --table_size=10000 --threads=1 --time=300 --report-interval=10 --db-driver=mysql --db-ps-mode=disable --skip-trx=on --mysql-ignore-errors=6002,6004,4012,2013,4016,1062,5157,4038 run

删除某节点的数据文件

抉择 zone3 下的 10.186.64.79 节点，将数据文件删除。

[root@localhost data]# rm -rf 1/sstable/block_file
[root@localhost data]# cd 1/sstable/
[root@localhost sstable]# ll
total 0

永恒下线故障节点

1. 调小参数 server_permanent_offline_time，缩短节点永恒下线工夫

server_permanent_offline_time 默认值为 3600s

MySQL [oceanbase]> alter system set server_permanent_offline_time='60s';
Query OK, 0 rows affected (0.030 sec)
 
MySQL [oceanbase]> SHOW PARAMETERS LIKE "%server_permanent_offline_time%";
+-------+----------+--------------+----------+-------------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------+--------------+---------+---------+-------------------+
| zone  | svr_type | svr_ip       | svr_port | name                          | data_type | value | info                                                                                                                              | section      | scope   | source  | edit_level        |
+-------+----------+--------------+----------+-------------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------+--------------+---------+---------+-------------------+
| zone3 | observer | 10.186.64.79 |     2882 | server_permanent_offline_time | NULL      | 60s   | the time interval between any two heartbeats beyond which a server is considered to be \'permanently\' offline. Range: [20s,+∞)   | ROOT_SERVICE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone1 | observer | 10.186.64.74 |     2882 | server_permanent_offline_time | NULL      | 60s   | the time interval between any two heartbeats beyond which a server is considered to be \'permanently\' offline. Range: [20s,+∞)   | ROOT_SERVICE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 10.186.64.75 |     2882 | server_permanent_offline_time | NULL      | 60s   | the time interval between any two heartbeats beyond which a server is considered to be \'permanently\' offline. Range: [20s,+∞)   | ROOT_SERVICE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+--------------+----------+-------------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------+--------------+---------+---------+-------------------+

2. 进行故障节点对外提供服务

在 kill ob 过程前，倡议应用隔离 (ISOLATE SERVER) 或者进行 (STOP SERVER) 节点的命令，停掉发往该节点的申请，转移正本 leader 角色。在节点重建复原后，再开启流量。

# 停掉 79 节点服务
MySQL [oceanbase]> ALTER SYSTEM STOP SERVER '10.186.64.79:2882' ZONE='zone3';
  
# 或者隔离
ALTER SYSTEM ISOLATE SERVER '10.186.64.79:2882' ZONE='zone3';

3.kill observer 过程

执行 kill -9 $observer_pid，期待 server_permanent_offline_time 的工夫，该 ob 进入 ” 永恒下线”状态。判断 ob 是否曾经永恒下线，能够查问表 __all_rootservice_event_history，存在名为 “permanent_offline “ 的 event 记录，确认工夫和 ip 都统一后，即可认为 ob 曾经永恒下线。

MySQL [oceanbase]> select * from __all_rootservice_event_history where event='permanent_offline' ; 

+----------------------------+--------+-------------------+--------+---------------------+-------+--------+-------+--------+-------+--------+-------+--------+-------+--------+------------+--------------+-------------+
| gmt_create                 | module | event             | name1  | value1              | name2 | value2 | name3 | value3 | name4 | value4 | name5 | value5 | name6 | value6 | extra_info | rs_svr_ip    | rs_svr_port |
+----------------------------+--------+-------------------+--------+---------------------+-------+--------+-------+--------+-------+--------+-------+--------+-------+--------+------------+--------------+-------------+
| 2023-03-29 17:34:09.596035 | server | permanent_offline | server | "10.186.64.79:2882" |       |        |       |        |       |        |       |        |       |        |            | 10.186.64.74 |        2882 |
+----------------------------+--------+-------------------+--------+---------------------+-------+--------+-------+--------+-------+--------+-------+--------+-------+--------+------------+--------------+-------------+

查问分区正本散布如下，已不存在 79 节点的分区正本信息，进一步确认了 79 节点已永恒下线。

zone2 下的 75 节点有一个从正本降级为 leader 角色，此时集群依然能够持续对外服务。

MySQL [oceanbase]> select tenant.tenant_name, zone, svr_ip,svr_port, case when role=1 then 'leader' when role=2 then 'follower' else NULL end as role, count(1) as partition_cnt from __all_virtual_meta_table meta  inner join __all_tenant tenant  on meta.tenant_id=tenant.tenant_id inner join __all_virtual_table tab  on meta.tenant_id=tab.tenant_id and meta.table_id=tab.table_id where tenant.tenant_id=1001 and tab.table_name='sbtest1' group by  tenant.tenant_name,zone, svr_ip,svr_port, 5 order by  tenant.tenant_name, zone, svr_ip, role desc;
+-----------------+-------+--------------+----------+----------+---------------+
| tenant_name     | zone  | svr_ip       | svr_port | role     | partition_cnt |
+-----------------+-------+--------------+----------+----------+---------------+
| sysbench_tenant | zone1 | 10.186.64.74 |     2882 | leader   |             1 |
| sysbench_tenant | zone1 | 10.186.64.74 |     2882 | follower |             2 |
| sysbench_tenant | zone2 | 10.186.64.75 |     2882 | leader   |             2 |
| sysbench_tenant | zone2 | 10.186.64.75 |     2882 | follower |             1 |
+-----------------+-------+--------------+----------+----------+---------------+
4 rows in set (0.005 sec)

拉起故障节点，触发数据主动重建

1. 启动 79 节点的 ob 过程，过程启动后会主动触发重建。

注：避免 ob 启动失败或存在其余问题，倡议启动前将数据文件和事务日志均清空。


[root@localhost data]# rm -rf log1/clog/*
[root@localhost data]# rm -rf log1/ilog/*
[root@localhost data]# rm -rf log1/slog/*
[root@localhost data]# rm -rf 1/sstable/block_file
[root@localhost data]# cd 1/sstable/
[root@localhost sstable]# ll
total 0
[root@localhost sstable]# su admin
bash-4.2$ cd /home/admin/ && ./bin/observer
./bin/observer

过程启动后，确认 ob 心跳复原状态为 active，而后查看分区正在一直补足中

MySQL [oceanbase]> select svr_ip,zone,with_rootserver,status,stop_time,start_service_time,build_version from __all_server;
+--------------+-------+-----------------+--------+-----------+--------------------+----------------------------------------------------------------------------------------+
| svr_ip       | zone  | with_rootserver | status | stop_time        | start_service_time | build_version                                                                          |
+--------------+-------+-----------------+--------+-----------+--------------------+----------------------------------------------------------------------------------------+
| 10.186.64.74 | zone1 |               1 | active |                0 |   1679984798650860 | 3.1.2_10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d(Dec 30 2021 02:47:29) |
| 10.186.64.75 | zone2 |               0 | active |                0 |   1679984801289281 | 3.1.2_10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d(Dec 30 2021 02:47:29) |
| 10.186.64.79 | zone3 |               0 | active | 1680082329964975 |   1680082511964975 | 3.1.2_10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d(Dec 30 2021 02:47:29) |
+--------------+-------+-----------------+--------+-----------+--------------------+----------------------------------------------------------------------------------------+
3 rows in set (0.002 sec)
 
MySQL [oceanbase]> select count(*),zone from gv$partition group by zone;
+----------+-------+
| count(*) | zone  |
+----------+-------+
|     1322 | zone1 |
|     1322 | zone2 |
|      152 | zone3 |
+----------+-------+
3 rows in set (0.228 sec)
  
MySQL [oceanbase]> select count(*),zone from gv$partition group by zone;
+----------+-------+
| count(*) | zone  |
+----------+-------+
|     1322 | zone1 |
|     1322 | zone2 |
|      664 | zone3 |
+----------+-------+
3 rows in set (0.113 sec)
MySQL [oceanbase]> select count(*),zone from gv$partition group by zone;                                                
+----------+-------+
| count(*) | zone  |
+----------+-------+
|     1322 | zone1 |
|     1322 | zone2 |
|     1179 | zone3 |
+----------+-------+
3 rows in set (0.112 sec)
  
MySQL [oceanbase]> select count(*),zone from gv$partition group by zone;
+----------+-------+
| count(*) | zone  |
+----------+-------+
|     1322 | zone1 |
|     1322 | zone2 |
|     1322 | zone3 |
+----------+-------+
3 rows in set (0.116 sec)

当 3 个 zone 内的分区个数统一后，同时查看 zone3 已存在正本信息，认为重建结束。

因为 79 节点处于隔离状态，所以还没有 leader 正本。

MySQL [oceanbase]> select tenant.tenant_name, zone, svr_ip,svr_port, case when role=1 then 'leader' when role=2 then 'follower' else NULL end as role, count(1) as partition_cnt from __all_virtual_meta_table meta  inner join __all_tenant tenant  on meta.tenant_id=tenant.tenant_id inner join __all_virtual_table tab  on meta.tenant_id=tab.tenant_id and meta.table_id=tab.table_id where tenant.tenant_id=1001 and tab.table_name='sbtest1' group by  tenant.tenant_name,zone, svr_ip,svr_port, 5 order by  tenant.tenant_name, zone, svr_ip, role desc;
+-----------------+-------+--------------+----------+----------+---------------+
| tenant_name     | zone  | svr_ip       | svr_port | role     | partition_cnt |
+-----------------+-------+--------------+----------+----------+---------------+
| sysbench_tenant | zone1 | 10.186.64.74 |     2882 | leader   |             1 |
| sysbench_tenant | zone1 | 10.186.64.74 |     2882 | follower |             2 |
| sysbench_tenant | zone2 | 10.186.64.75 |     2882 | leader   |             2 |
| sysbench_tenant | zone2 | 10.186.64.75 |     2882 | follower |             1 |
| sysbench_tenant | zone3 | 10.186.64.79 |     2882 | follower |             3 |
+-----------------+-------+--------------+----------+----------+---------------+
6 rows in set (0.005 sec)

2. 开启故障节点服务

执行命令解除 79 节点的隔离状态。

ALTER SYSTEM START SERVER '10.186.64.79:2882' ZONE='zone3';

查问分区正本散布如下，leader 角色已迁回 79 节点。

MySQL [oceanbase]> select tenant.tenant_name, zone, svr_ip,svr_port, case when role=1 then 'leader' when role=2 then 'follower' else NULL end as role, count(1) as partition_cnt from __all_virtual_meta_table meta  inner join __all_tenant tenant  on meta.tenant_id=tenant.tenant_id inner join __all_virtual_table tab  on meta.tenant_id=tab.tenant_id and meta.table_id=tab.table_id where tenant.tenant_id=1001 and tab.table_name='sbtest1' group by  tenant.tenant_name,zone, svr_ip,svr_port, 5 order by  tenant.tenant_name, zone, svr_ip, role desc;
+-----------------+-------+--------------+----------+----------+---------------+
| tenant_name     | zone  | svr_ip       | svr_port | role     | partition_cnt |
+-----------------+-------+--------------+----------+----------+---------------+
| sysbench_tenant | zone1 | 10.186.64.74 |     2882 | leader   |             1 |
| sysbench_tenant | zone1 | 10.186.64.74 |     2882 | follower |             2 |
| sysbench_tenant | zone2 | 10.186.64.75 |     2882 | leader   |             1 |
| sysbench_tenant | zone2 | 10.186.64.75 |     2882 | follower |             2 |
| sysbench_tenant | zone3 | 10.186.64.79 |     2882 | leader   |             1 |
| sysbench_tenant | zone3 | 10.186.64.79 |     2882 | follower |             2 |
+-----------------+-------+--------------+----------+----------+---------------+

3. 把 server_permanent_offline_time 参数的预知从新设置为默认的 3600s

MySQL [oceanbase]> alter system set server_permanent_offline_time='3600s';
Query OK, 0 rows affected (0.028 sec)
 
MySQL [oceanbase]> SHOW PARAMETERS LIKE "%server_permanent_offline_time%";
+-------+----------+--------------+----------+-------------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------+--------------+---------+---------+-------------------+
| zone  | svr_type | svr_ip       | svr_port | name                          | data_type | value | info                                                                                                                              | section      | scope   | source  | edit_level        |
+-------+----------+--------------+----------+-------------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------+--------------+---------+---------+-------------------+
| zone2 | observer | 10.186.64.75 |     2882 | server_permanent_offline_time | NULL      | 3600s | the time interval between any two heartbeats beyond which a server is considered to be \'permanently\' offline. Range: [20s,+∞)   | ROOT_SERVICE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone1 | observer | 10.186.64.74 |     2882 | server_permanent_offline_time | NULL      | 3600s | the time interval between any two heartbeats beyond which a server is considered to be \'permanently\' offline. Range: [20s,+∞)   | ROOT_SERVICE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 10.186.64.79 |     2882 | server_permanent_offline_time | NULL      | 3600s | the time interval between any two heartbeats beyond which a server is considered to be \'permanently\' offline. Range: [20s,+∞)   | ROOT_SERVICE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+--------------+----------+-------------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------+--------------+---------+---------+-------------------+
3 rows in set (0.007 sec)

校验各 ob 节点数据量

sysbench 已运行完结，直连各 observer，校验数据量是统一的。

[root@localhost ~]#  obclient -h10.186.64.74 -P2881 -usysbench@sysbench_tenant -Dsysbenchdb -A -psysbench
Welcome to the OceanBase.  Commands end with ; or \g.
Your MySQL connection id is 3221545401
Server version: 5.7.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MySQL [sysbenchdb]> select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
|    53195 |
+----------+
1 row in set (0.036 sec)
 
MySQL [sysbenchdb]> exit
Bye
[root@localhost ~]#  obclient -h10.186.64.75 -P2881 -usysbench@sysbench_tenant -Dsysbenchdb -A -psysbench
Welcome to the OceanBase.  Commands end with ; or \g.
Your MySQL connection id is 3221823448
Server version: 5.7.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MySQL [sysbenchdb]> select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
|    53195 |
+----------+
1 row in set (0.040 sec)
 
MySQL [sysbenchdb]> exit
Bye
[root@localhost ~]#  obclient -h10.186.64.79 -P2881 -usysbench@sysbench_tenant -Dsysbenchdb -A -psysbench
Welcome to the OceanBase.  Commands end with ; or \g.
Your MySQL connection id is 3222011907
Server version: 5.7.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MySQL [sysbenchdb]> select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
|    53195 |
+----------+
1 row in set (0.037 sec)
 
MySQL [sysbenchdb]>

总结

数据文件损坏或者失落时，可通过调整参数 server_permanent_offline_time 来重建受影响的节点。

1. 设小 server_permanent_offline_time 阈值

2. 进行故障节点对外服务

3. 终止该节点过程。

4. 超过阈值后，节点将被标记为永恒下线，零碎会主动清空正本以及向同 zone 内其余节点迁徙数据。

5. 启动 OB 过程，主动触发重建节点数据。

6. 开启故障节点服务。

7. 把 server_permanent_offline_time 参数改回原来的值

退出移动版