关于php:Skywalking-Php注册不上问题排查

4次阅读

共计 7934 个字符，预计需要花费 20 分钟才能阅读完成。

Skywalking 是一款分布式追踪利用，具体介绍能够参考 skywalking。

最近公司的一个 Php 利用在 Skywalking 后盾查不到数据了：

登录到某台服务器上发现注册不上，启动时就报错了：

先来整顿下 Skywalking php 的整个流程，php 扩大在系统启动时注册利用和实例，而后在每次申请拦挡相干调用，将相干调用状况保留下来；注册相干代码在 skywalking.c 的 module_init 中：

static void module_init() {

    application_instance = -100000;
    application_id = -100000;

    int i = 0;

    do {application_id = serviceRegister(SKYWALKING_G(grpc), SKYWALKING_G(app_code));

        if(application_id == -100000) {sleep(1);
        }

        i++;
    } while (application_id == -100000 && i <= 1);

    if (application_id == -100000) {
        sky_close = 1;
        return;
    }

    char *ipv4s = _get_current_machine_ip();

    char hostname[100] = {0};
    if (gethostname(hostname, sizeof(hostname)) < 0) {strcpy(hostname, "");
    }

    char *l_millisecond = get_millisecond();
    long millisecond = zend_atol(l_millisecond, strlen(l_millisecond));
    efree(l_millisecond);

    i = 0;
    do {application_instance = serviceInstanceRegister(SKYWALKING_G(grpc), application_id, millisecond, SKY_OS_NAME,
                                                       hostname, getpid(),
                                                       ipv4s);
        if(application_instance == -100000) {sleep(2);
        }
        i++;
    } while (application_instance == -100000 && i <= 5);


    if (application_instance == -100000) {
        sky_close = 1;
        php_error(E_WARNING, "skywalking: register service error");
        return;
    }
    
    php_error(E_WARNING, "skywalking: register service success");
}

能够看到，注册利用是调用 serviceRegister 函数注册，而后调用 serviceInstanceRegister 来注册实例的，后者会调用 GreeterClient::serviceInstanceRegister 以下函数实现注册：

 int serviceInstanceRegister(int applicationid, long registertime, char *osname, char *hostname, int processno,
                                char *ipv4s) {
        ServiceInstances request;
        ServiceInstance *s = request.add_instances();

        if (uuid == NULL) {std::string uuid_str = boost::uuids::to_string(boost_uuid);
            uuid = (char *) malloc(uuid_str.size() + 1);
            bzero(uuid, uuid_str.size() + 1);
            strncpy(uuid, uuid_str.c_str(), uuid_str.size() + 1);
        }

        s->set_serviceid(applicationid);
        s->set_instanceuuid(std::string(uuid));
        s->set_time(registertime);

        KeyStringValuePair *os = s->add_properties();
        KeyStringValuePair *host = s->add_properties();
        KeyStringValuePair *process = s->add_properties();
        KeyStringValuePair *ipv4 = s->add_properties();
        KeyStringValuePair *language = s->add_properties();

        os->set_key("os_name");
        os->set_value(osname);
        host->set_key("host_name");
        host->set_value(hostname);
        process->set_key("process_no");
        process->set_value(std::to_string(processno));
        ipv4->set_key("ipv4");
        ipv4->set_value(ipv4s);
        language->set_key("language");
        language->set_value("php");

        ServiceInstanceRegisterMapping reply;

        ClientContext context;

        Status status = stub_->doServiceInstanceRegister(&context, request, &reply);

        if (status.ok()) {for (int i = 0; i < reply.serviceinstances_size(); i++) {const KeyIntValuePair &kv = reply.serviceinstances(i);
//                std::cout << "Register Instance:"<< std::endl;
//                std::cout << kv.key() << ":" << kv.value() << std::endl;

                if (kv.key() == uuid) {return kv.value();
                }
            }
        }

        return -100000;

    }

通过 gdb 的断点，发现注册利用是胜利的，注册实例失败了，而后在 GreeterClient::serviceInstanceRegister 加上相应的日志：

Status status = stub_->doServiceInstanceRegister(&context, request, &reply);
        if (status.ok()) {std::cout << "size:" << reply.serviceinstances_size() << std::endl;
            for (int i = 0; i < reply.serviceinstances_size(); i++) {const KeyIntValuePair &kv = reply.serviceinstances(i);
                std::cout << "Register Instance:"<< std::endl;
                std::cout << kv.key() << ":" << kv.value() << std::endl;

                if (kv.key() == uuid) {return kv.value();
                }
            }
        }else{printf("instance register error");
        }

这里也把 UUID 打印进去了，发现响应里每次都是空的，所以导致失败。

客户端曾经没有线索了，只好从服务端动手，因为服务端是 Java 实现的，不大不便调试，因而在本地搭了个环境想调试下，哪知服务端跑起来了，Php 客户端死活编译不上，因为 Skywalking 依赖 protobuf、grpc 等组件，这些组件之间有版本依赖关系的，官网文档也没有阐明，一时陷入困境。

因之前服务端保护的同学走了，只好本人硬着头皮看代码，发现注册入口代码在 RegisterServiceHandler::doServiceInstanceRegister 中：

@Override
    public void doServiceInstanceRegister(ServiceInstances request,
        StreamObserver<ServiceInstanceRegisterMapping> responseObserver) {ServiceInstanceRegisterMapping.Builder builder = ServiceInstanceRegisterMapping.newBuilder();

        request.getInstancesList().forEach(instance -> {ServiceInventory serviceInventory = serviceInventoryCache.get(instance.getServiceId());

            JsonObject instanceProperties = new JsonObject();
            List<String> ipv4s = new ArrayList<>();

            for (KeyStringValuePair property : instance.getPropertiesList()) {String key = property.getKey();
                switch (key) {
                    case HOST_NAME:
                        instanceProperties.addProperty(HOST_NAME, property.getValue());
                        break;
                    case OS_NAME:
                        instanceProperties.addProperty(OS_NAME, property.getValue());
                        break;
                    case LANGUAGE:
                        instanceProperties.addProperty(LANGUAGE, property.getValue());
                        break;
                    case "ipv4":
                        ipv4s.add(property.getValue());
                        break;
                    case PROCESS_NO:
                        instanceProperties.addProperty(PROCESS_NO, property.getValue());
                        break;
                }
            }
            instanceProperties.addProperty(IPV4S, ServiceInstanceInventory.PropertyUtil.ipv4sSerialize(ipv4s));

            String instanceName = serviceInventory.getName();
            if (instanceProperties.has(PROCESS_NO)) {instanceName += "-pid:" + instanceProperties.get(PROCESS_NO).getAsString();}
            if (instanceProperties.has(HOST_NAME)) {instanceName += "@" + instanceProperties.get(HOST_NAME).getAsString();}

            int serviceInstanceId = serviceInstanceInventoryRegister.getOrCreate(instance.getServiceId(), instanceName, instance.getInstanceUUID(), instance.getTime(), instanceProperties);

            if (serviceInstanceId != Const.NONE) {logger.info("register service instance id={} [UUID:{}]", serviceInstanceId, instance.getInstanceUUID());
                builder.addServiceInstances(KeyIntValuePair.newBuilder().setKey(instance.getInstanceUUID()).setValue(serviceInstanceId));
            }
        });

        responseObserver.onNext(builder.build());
        responseObserver.onCompleted();}

要害是这行代码来生成实例 id 的：

int serviceInstanceId = serviceInstanceInventoryRegister.getOrCreate(instance.getServiceId(), instanceName, instance.getInstanceUUID(), instance.getTime(), instanceProperties);

再跟进去：

@Override public int getOrCreate(int serviceId, String serviceInstanceName, String uuid, long registerTime,
        JsonObject properties) {if (logger.isDebugEnabled()) {logger.debug("Get or create service instance by service instance name, service id: {}, service instance name: {},uuid: {}, registerTime: {}", serviceId, serviceInstanceName, uuid, registerTime);
        }

        int serviceInstanceId = getServiceInstanceInventoryCache().getServiceInstanceId(serviceId, uuid);

        if (serviceInstanceId == Const.NONE) {ServiceInstanceInventory serviceInstanceInventory = new ServiceInstanceInventory();
            serviceInstanceInventory.setServiceId(serviceId);
            serviceInstanceInventory.setName(serviceInstanceName);
            serviceInstanceInventory.setInstanceUUID(uuid);
            serviceInstanceInventory.setIsAddress(BooleanUtils.FALSE);
            serviceInstanceInventory.setAddressId(Const.NONE);

            serviceInstanceInventory.setRegisterTime(registerTime);
            serviceInstanceInventory.setHeartbeatTime(registerTime);

            serviceInstanceInventory.setProperties(properties);

            InventoryStreamProcessor.getInstance().in(serviceInstanceInventory);
        }
        return serviceInstanceId;
    }

这里的逻辑就比拟清晰了，先从缓存中拿实例 ID：

getServiceInstanceInventoryCache().getServiceInstanceId(serviceId, uuid);

拿不到则退出后台任务解决生成 ID。

再跟进 getServiceInstanceId 办法，

 Integer serviceInstanceId = serviceInstanceNameCache.getIfPresent(ServiceInstanceInventory.buildId(serviceId, uuid));

        if (Objects.isNull(serviceInstanceId) || serviceInstanceId == Const.NONE) {serviceInstanceId = getCacheDAO().getServiceInstanceId(serviceId, uuid);
            if (serviceId != Const.NONE) {serviceInstanceNameCache.put(ServiceInstanceInventory.buildId(serviceId, uuid), serviceInstanceId);
            }
        }
        return serviceInstanceId;

从缓存中拿不到则从 DAO 中拿，

 GetResponse response = getClient().get(ServiceInstanceInventory.INDEX_NAME, id);
            if (response.isExists()) {return (int)response.getSource().getOrDefault(RegisterSource.SEQUENCE, 0);
            } else {return Const.NONE;}

后者从 ES 索引 service_instance_inventory 去拿。

为了证实上述逻辑无误，从 ES 中读取数据试下，果然实例 ID 都注册在 ES 外面：

再从客户端证实下，既然实例 ID 是写入 ES 的，那么用以前的 ID 必定是能注册胜利的，因而批改客户端代码，将 UUID 写死注册试下：

 int serviceInstanceRegister(int applicationid, long registertime, char *osname, char *hostname, int processno,
 char *ipv4s) {
    ServiceInstances request;
    ServiceInstance *s = request.add_instances();
    uuid= "7e22c317-e2e2-4f81-a53d-fe011013e0a3";
    // 省略无关代码

马上注册胜利了：

7e22c317-e2e2-4f81-a53d-fe011013e0a3
size:1
Register Instance:
7e22c317-e2e2-4f81-a53d-fe011013e0a3: 3386041
PHP Warning:  skywalking: register service success in Unknown on line 0
PHP Warning:  skywalking: hook redis handler success in Unknown on line 0
PHP Warning:  skywalking: hook session handler success in Unknown on line 0

再回到这个问题，起因曾经晓得了，如何解决呢，有两个方法：

1、加大注册时等待时间，如期待到 100 秒；

2、记录最近一次注册胜利的 UUID 并且长久化，下次启动时间接用上次的；

因为 2 波及到改代码，因而先用计划 1 解决问题。

Skywalking Php 二：代码剖析

故障演练利器之 ChaosBlade 介绍

寰球智能 DNS 解析实际

一次线上 Mysql 死锁剖析

正文完
php
发表至： php
2020-09-19
 0
关于php:yiilogtarget-监控系统异常且多渠道发送异常信息通知
关于php:PHP中的static
关于php:error14090086SSL-routinesssl3getservercertificate
关于php:PHP中的一些杂项函数学习

站内搜索

-「」