关于java:分布式时序计算框架vortex-metrics使用介绍

46次阅读

共计 9805 个字符,预计需要花费 25 分钟才能阅读完成。

vortex metrics是一款用 Java 写的,轻量级的分布式时序计算框架。
vortex metrics 依赖分布式流式计算框架 vortex, 它是 vortex 对于时序计算场景的一个具体利用,vortex 更像是一套 API, 使用这套 API 能够演绎和解决各种业务场景,而 vortex metrics 则是依据这套 API 提出了对时序计算业务场景一种解决方案。

介绍之前,先比拟另外一个类似的时序计算工具 OpenTSDB, OpenTSDB 的存储依赖 HBase, 它更像一个时序数据库。OpenTSDB 通过 TCollector 和内置的 collectors 接收数据,并提供了界面供用户查问。
相比之下,vortex metrics 更偏差实时计算,将提早尽可能地放弃在秒级,但因为是内存计算,计算结果是存储在内存中的,无限的存储空间让它并不适用于存储大量指标的业务场景。但同时,对于一个指标的窗口数据须要保留的时间跨度较长时,vortex metrics 提供了定制接口,能够保留到内部存储,比方数据库或缓存,但这须要做一些额定的开发工作。此外,vortex metrics 也提供了 Web 查问界面,并且在继续改良中。

实际上,vortex metrics 的性能是很繁多的,因为它只做一件事,那就是依据客户端源源不断的上报数据实时地计算并输入单位工夫窗口内的统计数据,记住这点很重要。

环境配置

Maven:

<dependency>
    <groupId>com.github.paganini2008.atlantis</groupId>
    <artifactId>vortex-spring-boot-starter</artifactId>
    <version>1.0-RC1</version>
</dependency>

vortex metrics 它首先是一个基于 SpringBoot 的应用程序,后面说过,它依赖 vortex 框架, 而 vortex 是通过微服务分布式协调框架 tridenter 实现集群模式的,并反对动静程度扩大,默认端口为 6150

参考配置

# 集群配置
spring.application.name=vortex-metrics
spring.application.cluster.name=vortex-metrics-cluster

# vortex 配置
atlantis.framework.vortex.bufferzone.collectionName=metric  # 缓冲区汇合名称
atlantis.framework.vortex.bufferzone.pullSize=1000  # 缓冲区每次拉取的音讯数量
atlantis.framework.vortex.processor.threads=200  # 数据消费者线程数

输出地址:http://localhost:6150/metric/ 能够看到这个界面

vortex metrics 是将 server 数据接收端和监控界面做在同一个利用中的,高并发场景下,可能会呈现数据计算实现并已同步,但界面会滞后更新的状况。

HTTP API 阐明

vortex metrics 目前只提供了 2 个接口:

  1. 上报数据接口:
    POST  http://localhost:6150/metrics/sequence/{dataType}
    申请体示例:

    {
    "name": "car",
    "metric": "speed",
    "value": 120,
    "timestamp": 1613200609281
    }

    参数阐明:

    参数名 类型 形容 举例
    dateType 字符串 数据类型,取值范畴:bigint,numeric,bool bigint
    name 字符串 利用名称 car
    metric 字符串 上报指标名称 speed
    value 数值 上报数值 120
    timestamp 长整型 上报工夫戳 1613200609281
  2. 查看数据接口:
    GET  http://localhost:6150/metrics/sequence/{dataType}/{name}/{metric}
    响应体示例:

    {
     "dataType": "bigint",
     "name": "car",
     "metric": "speed",
     "data": {
         "00:03:00": {
             "speed": {
                 "middleValue": 5052,
                 "count": 155796,
                 "highestValue": 9999,
                 "lowestValue": 100,
                 "timestamp": 1623513794876
             }
         },
         "00:04:00": {
             "speed": {
                 "middleValue": 5051,
                 "count": 79754,
                 "highestValue": 9999,
                 "lowestValue": 100,
                 "timestamp": 1623513840003
             }
         },
         "00:05:00": {
             "speed": {
                 "middleValue": 5039,
                 "count": 61014,
                 "highestValue": 9999,
                 "lowestValue": 100,
                 "timestamp": 1623513933165
             }
         },
         "00:06:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623513960876
             }
         },
         "00:07:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514020876
             }
         },
         "00:08:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514080876
             }
         },
         "00:09:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514140876
             }
         },
         "00:10:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514200876
             }
         },
         "00:11:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514260876
             }
         },
         "00:12:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514320876
             }
         },
         "00:13:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514380876
             }
         },
         "00:14:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514440876
             }
         },
         "00:15:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514500876
             }
         },
         "00:16:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514560876
             }
         },
         "00:17:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514620876
             }
         },
         "00:18:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514680876
             }
         },
         "00:19:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514740876
             }
         },
         "00:20:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514800876
             }
         },
         "00:21:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514860876
             }
         },
         "00:22:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514920876
             }
         },
         "00:23:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514980876
             }
         },
         "00:24:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515040876
             }
         },
         "00:25:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515100876
             }
         },
         "00:26:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515160876
             }
         },
         "00:27:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515220876
             }
         },
         "00:28:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515280876
             }
         },
         "00:29:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515340876
             }
         },
         "00:30:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515400876
             }
         },
         "00:31:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515460876
             }
         },
         "00:32:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515520876
             }
         },
         "00:33:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515580876
             }
         },
         "00:34:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515640876
             }
         },
         "00:35:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515700876
             }
         },
         "00:36:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515760876
             }
         },
         "00:37:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515820876
             }
         },
         "00:38:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515880876
             }
         },
         "00:39:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515940876
             }
         },
         "00:40:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516000876
             }
         },
         "00:41:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516060876
             }
         },
         "00:42:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516120876
             }
         },
         "00:43:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516180876
             }
         },
         "00:44:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516240876
             }
         },
         "00:45:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516300876
             }
         },
         "00:46:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516360876
             }
         },
         "00:47:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516420876
             }
         },
         "00:48:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516480876
             }
         },
         "00:49:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516540876
             }
         },
         "00:50:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516600876
             }
         },
         "00:51:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516660876
             }
         },
         "00:52:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516720876
             }
         },
         "00:53:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516780876
             }
         },
         "00:54:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516840876
             }
         },
         "00:55:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516900876
             }
         },
         "00:56:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516960876
             }
         },
         "00:57:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517020876
             }
         },
         "00:58:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517080876
             }
         },
         "00:59:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517140876
             }
         },
         "01:00:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517200876
             }
         },
         "01:01:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517260876
             }
         },
         "01:02:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517320876
             }
         }
     }
    }

    响应体 json 中的 data 为时序数据,vortex metrics 默认的统计窗口为 1 分钟,滚动保留前 60 条记录,也就是用户能够通过界面看到前 60 分钟的统计数据。其中,highestValue,lowestValue,middleValue 别离为最大值,最小值,平均值,count 是 这 1 分钟内的数据条数,timestamp 是工夫戳,精确点讲,是这 1 分钟内最初一条数据的工夫戳。
    vortex metrics 默认的统计窗口,保留前多少条记录,能够通过配置文件设置的,但零碎须要重启能力失效。

如何监控指标

上图中的 location, 输出查问数据接口地址,比方举个下面的例子,统计小客车的平均速度,接口应该是,http://localhost:6150/metrics/sequence/bigint/car/speed
将这个 ur 复制到 location 输入框,点击 search, 能够看到:

此时是没数据的
vortex metrics 内置了 3 个测试接口, 都是 GET 形式

  1. http://localhost:6150/metrics/test/bigint/{name}/{metric}
  2. http://localhost:6150/metrics/test/numeric/{name}/{metric}
  3. http://localhost:6150/metrics/test/bool/{name}/{metric}
    这里为了测试 vortex metrics 集群性能,我起了两台服务,端口别离为 6150 和 6151

而后,关上你的压测工具,这里以 JMeter 为例
设置线程组

设置 HTTP 申请

察看压测后果:

能够看到,实时统计 1 分钟内的最大值,最小值,平均值和个数,提早在秒级
后面说过,vortex metrics 是内存计算,计算结果驻留在内存中的,如果存在大量指标数据,可能会呈现延时,所以它不适用于存储大量指标的运算场景(不过你能够部署多个集群来解决这个问题),目前只测算过指标在 1000 个以内的场景,高并发下延时放弃在 1~3 分钟内。

如何保证数据一致性

vortex metrics 集群的节点通过外部网络通讯实现互相复制,保证数据最终一致性,即每个节点的数据最终都是一样的,全量的。
也就是说:
http://localhost:6150/metrics/sequence/bigint/car/speed
http://localhost:6151/metrics/sequence/bigint/car/speed
两个接口返回的数据是最终统一的

如何保留历史数据

后面说到,vortex metrics 默认的统计窗口为 1 分钟,滚动保留前 60 条记录,那如果你想保留之前的历史记录,须要另外做开发。
首先要实现接口:

public interface MetricEvictionHandler<I, T extends Metric<T>> {void onEldestMetricRemoval(I identifier, String metric, T metricUnit);

}

具体可参考:LoggingMetricEvictionHandler 类(默认实现)
你还要实现接口:

public interface MetricSequencerFactory {GenericUserMetricSequencer<String, BigInt> getBigIntMetricSequencer();

    GenericUserMetricSequencer<String, Numeric> getNumericMetricSequencer();

    GenericUserMetricSequencer<String, Bool> getBoolMetricSequencer();}

具体可参考:DefaultMetricSequencerFactory 类(默认实现)

正文完
 0