关于zookeeper:zk启动正常但zkServersh-status报错Error-contacting-service

参考文章

1. zkServer.sh start启动失常 zkServer.sh status启动异样 解决办法

   • 呃…2181端口被占用还能启动胜利?

2. 【已解决】zookeeper显示Error contacting service. It is probably not running等问题

   • 扯, server.X中的X能够是从0开始的(尽管如同网上所有文章都是从1开始, 连官网也是).
3. 人工智能 装置zookeeper时候，能够查看过程启动，然而状态显示报错 Error contacting service. It is probably not running

4. CentOS下ZooKeeper的装置教程（单机模式）

   • 同参考文章2
5. bin/zkServer.sh status fails when jmx_prometheus_javaagent added as agent #392

zk: 3.4.9 (3节点)

kubernetes: 1.17.2

容器镜像应用 CentOS7 + zk安装包自行封装.

问题形容

zk的3个节点全副启动, 集群也建起来了, 失常对外提供服务. 然而执行zkServer.sh status查问节点状态异样, 如下

$ ./bin/zkServer.sh status 
ZooKeeper JMX enabled by default 
Using config: /usr/zookeeper-3.4.9/bin/../conf/zoo.cfg 
Error contacting service. It is probably not running

应用jps能够看到zk过程

$ jps
26 QuorumPeerMain
120300 Jps

排查过程

依照参考文章1中所说, 批改bin/zkServer.sh, 查看status子命令的具体过程.

STAT=`"$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \ -cp "$CLASSPATH" $JVMFLAGS org.apache.zookeeper.client.FourLetterWordMain \ $clientPortAddress $clientPort srvr 2> /dev/null \ | $GREP Mode`

将上述语句的2> /dev/null以及前面的管道过滤语句移除, 查看srvr指令的输入, 失去如下

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:100)
    at sun.net.httpserver.HttpServerImpl.<init>(HttpServerImpl.java:50)
    at sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35)
    at com.sun.net.httpserver.HttpServer.create(HttpServer.java:130)
    at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:179)
    at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:31) ... 6 more 
FATAL ERROR in native method: processing of -javaagent failed

和参考文章1中说的一样, 都是Address already in use, 然而ta没说是哪个端口啊…🤔

而且zk都失常运行了, 2181/3888这种被占用不是很失常的事件嘛…

上面就是碰运气的过程了, 依照参考文章2提到的dataLogDir目录没有胜利创立, 或是myid门路不正确的问题…这些配置不正确能启动胜利?

参考文章3提到了防火墙, 嗯…容器里基本没有防火墙. 另外/etc/hosts也没必要, server.X中的地址各节点是能够互相通信的(当然我也试了下, 不出所料🤔).

解决办法

最终的解决办法是参考文章5, 为了实现在容器环境下对zk容器的监控, 咱们应用了jmx_exporter, 于是批改了zkServer.sh中的JVMFLAGS变量.

JVMFLAGS="$JVMFLAGS -javaagent:$JMX_DIR/jmx_prometheus_javaagent-0.15.0.jar=19105:$JMX_DIR/zookeeper.yaml"

将这一行正文掉, 再执行zkServer.sh status, 就能够了.

另外, 除了这个起因, 还有一个起因是, 咱们的zoo.cfg配置不规范, 如下

clientPort:2181
server.0:zk-ha-test-busi-kafka-0.zk-ha-test-busi-kafka-svc.zjjpt-zk.svc.cs-hua.hpc:2888:3888
server.1:zk-ha-test-busi-kafka-1.zk-ha-test-busi-kafka-svc.zjjpt-zk.svc.cs-hua.hpc:2888:3888
server.2:zk-ha-test-busi-kafka-2.zk-ha-test-busi-kafka-svc.zjjpt-zk.svc.cs-hua.hpc:2888:3888
dataLogDir:/data/zk-ha-test-busi-kafka-0/log
dataDir:/data/zk-ha-test-busi-kafka-0
## ...省略

用冒号:代替了等号=, zk居然能够失常运行😱.

这样导致在zkServer.sh中在执行status子命令时, 从zoo.cfg中解析clientPort变量会有问题.

clientPort=`$GREP "^[[:space:]]*clientPort[^[:alpha:]]" "$ZOOCFG" | sed -e 's/.*=//'`

前面的sed指令是按=进行切分的, 这样失去的后果为clientPort:2181, 是个非法数值.

批改zoo.cfg的格局后就能够了.