像 Rabbit MQ 这种消息中间件,一旦在我的项目中用到了,就阐明我的项目的体量曾经到了肯定的级别,单机节点就显得顾此失彼了。所以,个别都会搭建一个集群。上面就试着搭建三台机器的集群。
节点信息
节点一:192.168.0.116 centos1
节点二:192.168.0.117 centos2
节点三:192.168.0.118 centos3
配置 hosts 文件
配置各个节点的 hosts 文件,让各个节点都能相互辨认对方。
# /etc/hosts
192.168.0.116 centos1
192.168.0.117 centos2
192.168.0.118 centos3
复制 cookie 文件
编辑 Rabbit MQ 的 cookie 文件,以确保各个节点的 cookie 文件是同一个。这里应用 centos1 节点的 cookie 文件,将 centos1 的 cookie 文件复制到 centos2 和 centos3 的 /var/lib/rabbit/.erlang.cookie
或者 $HOME/.erlang.cookie
下。
cookie 文件地位
cookie 文件默认门路为 /var/lib/rabbit/.erlang.cookie
或者 $HOME/.erlang.cookie
。cookie 相当于密钥令牌,集群中的节点须要通过替换密钥令牌取得互相认证。
- 如果应用解压缩装置形式 (二进制装置或者编译装置),那么该文件存在地位为 $HOME 目录下。即
$HOME/.erlang.cookie
。如果咱们应用 root 装置,则地位为:/root/.erlang.cookie
,其余用户为/home/ 用户名 /.erlang.cookie
。 - 如果应用 rpm 包形式装置,那么这个文件会存在于 /var/lib/rabbitmq 目录下。
查找 .erlang.cookie 文件
能够通过查看 Rabbit MQ log 日志的信息,如上面所示:所以我的 .erlang.cookie 文件在 /root/.erlang.cookie
[info] <0.270.0>
node : rabbit@centos1
home dir : /root(我是 root 用户启动的)config file(s) : (none)
cookie hash : tCXB8mlCcGEGGV1cYRkQCg==
log(s) : /usr/local/rabbitmq_server/var/log/rabbitmq/rabbit@centos1.log
: /usr/local/rabbitmq_server/var/log/rabbitmq/rabbit@centos1_upgrade.log
database dir : /usr/local/rabbitmq_server/var/lib/rabbitmq/mnesia/rabbit@centos1
配置集群
配置集群有三种形式,这里应用 rabbitmqctl 工具配置:
- 通过 rabbitmqctl 工具配置;
- 通过 rabbitmq.config 配置文件配置;
- 通过 rabbitmq-autocluster 插件配置;
启动三个节点的 Rabbit MQ 服务:
[root@centos1 ~]# rabbitmq-server -detached
[root@centos2 ~]# rabbitmq-server -detached
[root@centos3 ~]# rabbitmq-server -detached
启动之后,目前三个节点还都是独立的,能够通过 rabbitmqctl cluster_status
命令查看状态。
退出集群
以 centos1 节点为基准,将 centos2 和 centos3 节点退出 centos1,以 centos2 为例步骤如下:
[root@centos2 ~]# rabbitmqctl stop_app
Stopping rabbit application on node rabbit@centos2 ...
[root@centos2 ~]# rabbitmqctl reset
Resetting node rabbit@centos2 ...
[root@centos2 ~]# rabbitmqctl join_cluster rabbit@centos1
Clustering node rabbit@centos2 with rabbit@centos1
[root@centos2 ~]# rabbitmqctl start_app
Starting node rabbit@centos2 ...
在任意节点上应用命令 rabbitmqctl cluster_status
[{nodes,[{disc,[rabbit@centos1,rabbit@centos2,rabbit@centos3]}]},
{running_nodes,[rabbit@centos1,rabbit@centos2,rabbit@centos3]},
{cluster_name,<<"rabbit@centos1">>},
{partitions,[]},
{alarms,[{rabbit@centos1,[]},{rabbit@centos2,[]},{rabbit@centos3,[]}]}
]
Web 端展现
停掉某个节点
以上就是三个节点的集群的搭建,如果说停掉启动一个节点,会是什么后果?停掉 centos2 试试看。
[root@centos2 ~]# rabbitmqctl stop_app
rabbitmqctl cluster_statua 查看
[{nodes,[{disc,[rabbit@centos1,rabbit@centos2,rabbit@centos3]}]},
{running_nodes,[rabbit@centos1,rabbit@centos3]},
{cluster_name,<<"rabbit@centos1">>},
{partitions,[]},
{alarms,[{rabbit@centos1,[]},{rabbit@centos3,[]}]}
]
Web 端展现
集群节点敞开和启动
如果敞开了集群中的所有节点,则须要确保在启动的时候最初敞开的那个节点是第一个启动。如果第一个启动的不是最初敞开的节点,那么这个节点会期待最初敞开的节点的启动。这个等待时间是 30 秒,如果没有等到,那么这个先启动的节点也会失败。
会有重试机制,默认重试 20 次,每次 30 秒以期待最初敞开节点的启动。以后版本:Rabbit MQ:RabbitMQ 3.8.9 on Erlang 23.0
在重试失败后,以后节点也会因为失败而敞开本身的利用。
重试日志:
2021-11-27 17:12:21.783 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-11-27 17:12:22.206 [debug] <0.2664.0> Lager installed handler lager_backend_throttle into lager_event
2021-11-27 17:12:51.784 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:12:51.784 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 8 retries left
2021-11-27 17:13:21.785 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:13:21.785 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 7 retries left
2021-11-27 17:13:51.786 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:13:51.786 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 6 retries left
2021-11-27 17:14:21.787 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:14:21.787 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 5 retries left
2021-11-27 17:14:51.788 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:14:51.788 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 4 retries left
2021-11-27 17:15:21.789 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:15:21.789 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 3 retries left
2021-11-27 17:15:51.790 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:15:51.790 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 2 retries left
2021-11-27 17:16:21.791 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:16:21.791 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 1 retries left
2021-11-27 17:16:51.792 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}
2021-11-27 17:16:51.792 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2021-11-27 17:17:21.793 [error] <0.2656.0> Feature flag `quorum_queue`: migration function crashed: {error,{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_durable_queue]}}
[{rabbit_table,wait,3,[{file,"src/rabbit_table.erl"},{line,120}]},{rabbit_core_ff,quorum_queue_migration,3,[{file,"src/rabbit_core_ff.erl"},{line,60}]},{rabbit_feature_flags,run_migration_fun,3,[{file,"src/rabbit_feature_flags.erl"},{line,1602}]},{rabbit_feature_flags,'-verify_which_feature_flags_are_actually_enabled/0-fun-2-',3,[{file,"src/rabbit_feature_flags.erl"},{line,2269}]},{maps,fold_1,3,[{file,"maps.erl"},{line,233}]},{rabbit_feature_flags,verify_which_feature_flags_are_actually_enabled,0,[{file,"src/rabbit_feature_flags.erl"},{line,2267}]},{rabbit_feature_flags,sync_feature_flags_with_cluster,3,[{file,"src/rabbit_feature_flags.erl"},{line,2082}]},{rabbit_mnesia,ensure_feature_flags_are_in_sync,2,[{file,"src/rabbit_mnesia.erl"},{line,647}]}]
2021-11-27 17:17:21.793 [warning] <0.2656.0> Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/usr/local/rabbitmq_server/var/lib/rabbitmq/mnesia/rabbit@centos2-feature_flags`:
2021-11-27 17:17:21.793 [warning] <0.2656.0> Feature flags: - list of previously disabled feature flags now marked as such: [empty_basic_get_metric]
2021-11-27 17:17:21.800 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-11-27 17:17:51.801 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:17:51.801 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 8 retries left
2021-11-27 17:18:21.802 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:18:21.802 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 7 retries left
2021-11-27 17:18:51.803 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:18:51.803 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 6 retries left
2021-11-27 17:19:21.804 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:19:21.804 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 5 retries left
2021-11-27 17:19:51.805 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:19:51.805 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 4 retries left
2021-11-27 17:20:21.806 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:20:21.806 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 3 retries left
2021-11-27 17:20:51.807 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:20:51.807 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 2 retries left
2021-11-27 17:21:21.808 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:21:21.808 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 1 retries left
2021-11-27 17:21:51.809 [warning] <0.2656.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]}
2021-11-27 17:21:51.809 [info] <0.2656.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2021-11-27 17:22:21.813 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-11-27 17:22:21.813 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-11-27 17:22:21.817 [error] <0.2656.0>
2021-11-27 17:22:21.818 [error] <0.2656.0> BOOT FAILED
2021-11-27 17:22:21.818 [error] <0.2656.0> ===========
2021-11-27 17:22:21.818 [error] <0.2656.0> Timeout contacting cluster nodes: [rabbit@centos3,rabbit@centos1].
2021-11-27 17:22:21.818 [error] <0.2656.0>
2021-11-27 17:22:21.818 [error] <0.2656.0> BACKGROUND
2021-11-27 17:22:21.818 [error] <0.2656.0> ==========
2021-11-27 17:22:21.818 [error] <0.2656.0>
2021-11-27 17:22:21.818 [error] <0.2656.0> This cluster node was shut down while other nodes were still running.
2021-11-27 17:22:21.818 [error] <0.2656.0> To avoid losing data, you should start the other nodes first, then
2021-11-27 17:22:21.818 [error] <0.2656.0> start this one. To force this node to start, first invoke
2021-11-27 17:22:21.818 [error] <0.2656.0> "rabbitmqctl force_boot". If you do so, any changes made on other
2021-11-27 17:22:21.818 [error] <0.2656.0> cluster nodes after this one was shut down may be lost.
2021-11-27 17:22:21.818 [error] <0.2656.0>
2021-11-27 17:22:21.818 [error] <0.2656.0> DIAGNOSTICS
2021-11-27 17:22:21.818 [error] <0.2656.0> ===========
2021-11-27 17:22:21.819 [error] <0.2656.0>
2021-11-27 17:22:21.819 [error] <0.2656.0> attempted to contact: [rabbit@centos3,rabbit@centos1]
2021-11-27 17:22:21.819 [error] <0.2656.0>
2021-11-27 17:22:21.819 [error] <0.2656.0> rabbit@centos3:
2021-11-27 17:22:21.819 [error] <0.2656.0> * connected to epmd (port 4369) on centos3
2021-11-27 17:22:21.819 [error] <0.2656.0> * node rabbit@centos3 up, 'rabbit' application not running
2021-11-27 17:22:21.819 [error] <0.2656.0> * running applications on rabbit@centos3: [lager,observer_cli,
2021-11-27 17:22:21.819 [error] <0.2656.0> stdout_formatter,
2021-11-27 17:22:21.819 [error] <0.2656.0> gen_batch_server,aten,cuttlefish,
2021-11-27 17:22:21.819 [error] <0.2656.0> inets,credentials_obfuscation,
2021-11-27 17:22:21.820 [error] <0.2656.0> recon,ranch,jsx,goldrush,xmerl,
2021-11-27 17:22:21.820 [error] <0.2656.0> tools,syntax_tools,ssl,
2021-11-27 17:22:21.820 [error] <0.2656.0> public_key,asn1,crypto,compiler,
2021-11-27 17:22:21.820 [error] <0.2656.0> sasl,stdlib,kernel]
2021-11-27 17:22:21.820 [error] <0.2656.0> * suggestion: use rabbitmqctl start_app on rabbit@centos3
2021-11-27 17:22:21.820 [error] <0.2656.0> rabbit@centos1:
2021-11-27 17:22:21.820 [error] <0.2656.0> * connected to epmd (port 4369) on centos1
2021-11-27 17:22:21.923 [error] <0.2656.0> * node rabbit@centos1 up, 'rabbit' application not running
2021-11-27 17:22:21.924 [error] <0.2656.0> * running applications on rabbit@centos1: [lager,observer_cli,
2021-11-27 17:22:21.924 [error] <0.2656.0> stdout_formatter,
2021-11-27 17:22:21.924 [error] <0.2656.0> gen_batch_server,aten,cuttlefish,
2021-11-27 17:22:21.924 [error] <0.2656.0> inets,credentials_obfuscation,
2021-11-27 17:22:21.924 [error] <0.2656.0> recon,ranch,jsx,goldrush,xmerl,
2021-11-27 17:22:21.924 [error] <0.2656.0> tools,syntax_tools,ssl,
2021-11-27 17:22:21.924 [error] <0.2656.0> public_key,asn1,crypto,compiler,
2021-11-27 17:22:21.924 [error] <0.2656.0> sasl,stdlib,kernel]
2021-11-27 17:22:21.924 [error] <0.2656.0> * suggestion: use rabbitmqctl start_app on rabbit@centos1
2021-11-27 17:22:21.924 [error] <0.2656.0>
2021-11-27 17:22:21.924 [error] <0.2656.0> Current node details:
2021-11-27 17:22:21.924 [error] <0.2656.0> * node name: rabbit@centos2
2021-11-27 17:22:21.924 [error] <0.2656.0> * effective user's home directory: /root
2021-11-27 17:22:21.924 [error] <0.2656.0> * Erlang cookie hash: tCXB8mlCcGEGGV1cYRkQCg==
2021-11-27 17:22:21.924 [error] <0.2656.0>
2021-11-27 17:22:21.924 [error] <0.2656.0>
2021-11-27 17:22:22.926 [info] <0.2655.0> [{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.2655.0>},{registered_name,[]},{error_info,{exit,{{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[<0.2654.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.2656.0>,normal}]},{links,[<0.2654.0>,<0.44.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,1598},{stack_size,28},{reductions,368}], []
2021-11-27 17:22:22.926 [error] <0.2655.0> CRASH REPORT Process <0.2655.0> with 0 neighbours exited with reason: {{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}} in application_master:init/4 line 138
2021-11-27 17:22:22.927 [info] <0.44.0> Application rabbit exited with reason: {{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}}
2021-11-27 17:22:22.927 [info] <0.44.0> Application rabbit exited with reason: {{timeout_waiting_for_tables,[rabbit@centos3,rabbit@centos2,rabbit@centos1],[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]},{rabbit,start,[normal,[]]}}
2021-11-27 17:22:22.928 [info] <0.44.0> Application sysmon_handler exited with reason: stopped
2021-11-27 17:22:22.928 [info] <0.44.0> Application sysmon_handler exited with reason: stopped
2021-11-27 17:22:22.932 [info] <0.44.0> Application ra exited with reason: stopped
2021-11-27 17:22:22.932 [info] <0.44.0> Application ra exited with reason: stopped
2021-11-27 17:22:22.933 [info] <0.44.0> Application os_mon exited with reason: stopped
2021-11-27 17:22:22.933 [info] <0.44.0> Application os_mon exited with reason: stopped
失败信息返回
[root@centos2 ~]# rabbitmqctl start_app
Starting node rabbit@centos2 ...
Error:
{:rabbit, {{:timeout_waiting_for_tables, [:rabbit@centos3, :rabbit@centos2, :rabbit@centos1], [:rabbit_user, :rabbit_user_permission, :rabbit_topic_permission, :rabbit_vhost, :rabbit_durable_route, :rabbit_durable_exchange, :rabbit_runtime_parameters, :rabbit_durable_queue]}, {:rabbit, :start, [:normal, []]}}}
剔除节点
如果最初一个敞开的节点最终因为某些异样而无奈启动,则能够通过 rabbitmqctl forget_cluster_node
命令将节点剔除集群。
如果集群中所有节点因为非正常因素敞开,那么集群中的节点都会认为本人不是最初敞开的,此时须要调用 rabbitmqctl force_boot
命令来启动一个节点,之后集群能力失常启动。
[root@centos2 ~]# rabbitmqctl force boot
Forcing boot for Mnesia dir /usr/local/rabbitmq_server/var/lib/rabbitmq/mnesia/rabbit@centos2
[root@centos2 ~]# rabbitmq server detached
总结
以上就是多机多节点配置的流程。特地须要留神的是,牢记集群中节点的启动程序,肯定要先启动最初敞开的节点,防止启动失败。