关于java:分布式时序计算框架vortex-metrics使用介绍

vortex metrics是一款用Java写的,轻量级的分布式时序计算框架。
vortex metrics依赖分布式流式计算框架vortex, 它是vortex对于时序计算场景的一个具体利用,vortex更像是一套API, 使用这套API能够演绎和解决各种业务场景,而vortex metrics则是依据这套API提出了对时序计算业务场景一种解决方案。

介绍之前,先比拟另外一个类似的时序计算工具OpenTSDB, OpenTSDB的存储依赖HBase, 它更像一个时序数据库。OpenTSDB通过TCollector和内置的collectors接收数据,并提供了界面供用户查问。
相比之下,vortex metrics更偏差实时计算,将提早尽可能地放弃在秒级,但因为是内存计算,计算结果是存储在内存中的,无限的存储空间让它并不适用于存储大量指标的业务场景。但同时,对于一个指标的窗口数据须要保留的时间跨度较长时,vortex metrics提供了定制接口,能够保留到内部存储,比方数据库或缓存,但这须要做一些额定的开发工作。此外,vortex metrics也提供了Web查问界面,并且在继续改良中。

实际上,vortex metrics的性能是很繁多的,因为它只做一件事,那就是依据客户端源源不断的上报数据实时地计算并输入单位工夫窗口内的统计数据,记住这点很重要。

环境配置

Maven:

<dependency>
    <groupId>com.github.paganini2008.atlantis</groupId>
    <artifactId>vortex-spring-boot-starter</artifactId>
    <version>1.0-RC1</version>
</dependency>

vortex metrics它首先是一个基于SpringBoot的应用程序,后面说过,它依赖vortex框架, 而vortex是通过微服务分布式协调框架tridenter实现集群模式的,并反对动静程度扩大,默认端口为6150

参考配置

# 集群配置
spring.application.name=vortex-metrics
spring.application.cluster.name=vortex-metrics-cluster

# vortex配置
atlantis.framework.vortex.bufferzone.collectionName=metric  # 缓冲区汇合名称
atlantis.framework.vortex.bufferzone.pullSize=1000  # 缓冲区每次拉取的音讯数量
atlantis.framework.vortex.processor.threads=200  # 数据消费者线程数

输出地址:http://localhost:6150/metric/ 能够看到这个界面

vortex metrics是将server数据接收端和监控界面做在同一个利用中的,高并发场景下,可能会呈现数据计算实现并已同步,但界面会滞后更新的状况。

HTTP API阐明

vortex metrics目前只提供了2个接口:

  1. 上报数据接口:
    POST  http://localhost:6150/metrics/sequence/{dataType}
    申请体示例:

    {
    "name": "car",
    "metric": "speed",
    "value": 120,
    "timestamp": 1613200609281
    }

    参数阐明:

    参数名 类型 形容 举例
    dateType 字符串 数据类型,取值范畴:bigint,numeric,bool bigint
    name 字符串 利用名称 car
    metric 字符串 上报指标名称 speed
    value 数值 上报数值 120
    timestamp 长整型 上报工夫戳 1613200609281
  2. 查看数据接口:
    GET  http://localhost:6150/metrics/sequence/{dataType}/{name}/{metric}
    响应体示例:

    {
     "dataType": "bigint",
     "name": "car",
     "metric": "speed",
     "data": {
         "00:03:00": {
             "speed": {
                 "middleValue": 5052,
                 "count": 155796,
                 "highestValue": 9999,
                 "lowestValue": 100,
                 "timestamp": 1623513794876
             }
         },
         "00:04:00": {
             "speed": {
                 "middleValue": 5051,
                 "count": 79754,
                 "highestValue": 9999,
                 "lowestValue": 100,
                 "timestamp": 1623513840003
             }
         },
         "00:05:00": {
             "speed": {
                 "middleValue": 5039,
                 "count": 61014,
                 "highestValue": 9999,
                 "lowestValue": 100,
                 "timestamp": 1623513933165
             }
         },
         "00:06:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623513960876
             }
         },
         "00:07:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514020876
             }
         },
         "00:08:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514080876
             }
         },
         "00:09:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514140876
             }
         },
         "00:10:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514200876
             }
         },
         "00:11:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514260876
             }
         },
         "00:12:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514320876
             }
         },
         "00:13:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514380876
             }
         },
         "00:14:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514440876
             }
         },
         "00:15:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514500876
             }
         },
         "00:16:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514560876
             }
         },
         "00:17:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514620876
             }
         },
         "00:18:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514680876
             }
         },
         "00:19:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514740876
             }
         },
         "00:20:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514800876
             }
         },
         "00:21:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514860876
             }
         },
         "00:22:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514920876
             }
         },
         "00:23:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623514980876
             }
         },
         "00:24:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515040876
             }
         },
         "00:25:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515100876
             }
         },
         "00:26:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515160876
             }
         },
         "00:27:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515220876
             }
         },
         "00:28:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515280876
             }
         },
         "00:29:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515340876
             }
         },
         "00:30:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515400876
             }
         },
         "00:31:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515460876
             }
         },
         "00:32:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515520876
             }
         },
         "00:33:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515580876
             }
         },
         "00:34:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515640876
             }
         },
         "00:35:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515700876
             }
         },
         "00:36:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515760876
             }
         },
         "00:37:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515820876
             }
         },
         "00:38:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515880876
             }
         },
         "00:39:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623515940876
             }
         },
         "00:40:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516000876
             }
         },
         "00:41:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516060876
             }
         },
         "00:42:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516120876
             }
         },
         "00:43:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516180876
             }
         },
         "00:44:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516240876
             }
         },
         "00:45:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516300876
             }
         },
         "00:46:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516360876
             }
         },
         "00:47:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516420876
             }
         },
         "00:48:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516480876
             }
         },
         "00:49:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516540876
             }
         },
         "00:50:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516600876
             }
         },
         "00:51:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516660876
             }
         },
         "00:52:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516720876
             }
         },
         "00:53:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516780876
             }
         },
         "00:54:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516840876
             }
         },
         "00:55:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516900876
             }
         },
         "00:56:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623516960876
             }
         },
         "00:57:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517020876
             }
         },
         "00:58:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517080876
             }
         },
         "00:59:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517140876
             }
         },
         "01:00:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517200876
             }
         },
         "01:01:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517260876
             }
         },
         "01:02:00": {
             "speed": {
                 "middleValue": 0,
                 "count": 0,
                 "highestValue": 0,
                 "lowestValue": 0,
                 "timestamp": 1623517320876
             }
         }
     }
    }

    响应体json中的data为时序数据,vortex metrics默认的统计窗口为1分钟,滚动保留前60条记录,也就是用户能够通过界面看到前60分钟的统计数据。其中,highestValue,lowestValue,middleValue别离为最大值,最小值,平均值,count是 这1分钟内的数据条数,timestamp是工夫戳,精确点讲,是这1分钟内最初一条数据的工夫戳。
    vortex metrics默认的统计窗口,保留前多少条记录,能够通过配置文件设置的,但零碎须要重启能力失效。

如何监控指标

上图中的location, 输出查问数据接口地址,比方举个下面的例子,统计小客车的平均速度,接口应该是,http://localhost:6150/metrics/sequence/bigint/car/speed
将这个ur复制到location输入框,点击search, 能够看到:

此时是没数据的
vortex metrics内置了3个测试接口, 都是GET形式

  1. http://localhost:6150/metrics/test/bigint/{name}/{metric}
  2. http://localhost:6150/metrics/test/numeric/{name}/{metric}
  3. http://localhost:6150/metrics/test/bool/{name}/{metric}
    这里为了测试vortex metrics集群性能,我起了两台服务,端口别离为6150和6151

而后,关上你的压测工具,这里以JMeter为例
设置线程组

设置HTTP申请

察看压测后果:

能够看到,实时统计1分钟内的最大值,最小值,平均值和个数,提早在秒级
后面说过,vortex metrics是内存计算,计算结果驻留在内存中的,如果存在大量指标数据,可能会呈现延时,所以它不适用于存储大量指标的运算场景(不过你能够部署多个集群来解决这个问题),目前只测算过指标在1000个以内的场景,高并发下延时放弃在1~3分钟内。

如何保证数据一致性

vortex metrics集群的节点通过外部网络通讯实现互相复制,保证数据最终一致性,即每个节点的数据最终都是一样的,全量的。
也就是说:
http://localhost:6150/metrics/sequence/bigint/car/speed
http://localhost:6151/metrics/sequence/bigint/car/speed
两个接口返回的数据是最终统一的

如何保留历史数据

后面说到,vortex metrics默认的统计窗口为1分钟,滚动保留前60条记录,那如果你想保留之前的历史记录,须要另外做开发。
首先要实现接口:

public interface MetricEvictionHandler<I, T extends Metric<T>> {

    void onEldestMetricRemoval(I identifier, String metric, T metricUnit);

}

具体可参考:LoggingMetricEvictionHandler类(默认实现)
你还要实现接口:

public interface MetricSequencerFactory {

    GenericUserMetricSequencer<String, BigInt> getBigIntMetricSequencer();

    GenericUserMetricSequencer<String, Numeric> getNumericMetricSequencer();

    GenericUserMetricSequencer<String, Bool> getBoolMetricSequencer();

}

具体可参考:DefaultMetricSequencerFactory类(默认实现)

【腾讯云】轻量 2核2G4M,首年65元

阿里云限时活动-云数据库 RDS MySQL  1核2G配置 1.88/月 速抢

本文由乐趣区整理发布,转载请注明出处,谢谢。

您可能还喜欢...

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据