乐趣区

关于后端:后端接口压测总结

原因

个别公司会有性能测试团队,我的项目上线前须要对页面和接口进行压测,大多数是对接口进行压测,对接口的并发度和耗时进行测评。公司临时没有性能测试的岗位人员,需开发人员对我的项目接口进行自行测试,在理解一番接口压测后,开始了压测工作,压测技能对开发人员还是比拟重要的,一方面能够验证功能设计是否达标,另一方面也是提前检测出性能的性能问题防止线上扩容后,程序性能仍然不能晋升的难堪,还能反向带动开发人员的技术素养。

简述

压测与开发相比,存在很多不同,开发人员上手必定有一些不习惯,明天总结这篇文章,心愿能够帮忙开发人员顺利开展压测工作。

注释

压测工作有专门的文档介绍,能够自行百度,这里会从实战的角度进行讲述。

筹备工作

  1. 首先须要对压测的我的项目业务流程非常相熟,哪些业务场景是高频的,须要重点关注。
  2. 接着就要筹备压测环境,搭建 Mysql \ Redis \ Nacos \ Seata \ Sentinel \ Promethues \ Grafana … 部署我的项目
  3. 最初一步就是依据业务须要编写 Jmeter 脚本。

抉择的业务场景没有相对的规范,能够外部探讨哪些接口须要进行压测,也能够对本人开发的接口进行压测。

开始筹备压测环境,倡议抉择云服务器,抢占式实例真的很便宜,一个小时 2 毛钱,买个 10 台服务器,测试 5 个小时的老本和一份快餐费用差不多。

部署的中间件最好和生产环境保持一致,千万不要应用和生产环境相差太多的版本。

# 部署 Docker
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum list docker-ce --showduplicates | sort -r
yum install -y docker-ce-19.03.13-3.el8 docker-ce-cli-19.03.13-3.el8 containerd.io

cat >> /etc/docker/daemon.json  << EOF
{"registry-mirrors": ["https://i1ntfhil.mirror.aliyuncs.com"],
 "exec-opts": ["native.cgroupdriver=systemd"],
 "log-driver": "json-file",
 "log-opts": {"max-size": "100m"},
 "storage-driver": "overlay2"
}
EOF

systemctl enable docker && systemctl daemon-reload && systemctl restart docker
# 部署 Mysql 5.7
yum install -y mysql-server
systemctl enable mysqld && systemctl start mysqld

# yum 装置的 MySQL 一开始是没有 root 明码的 能够进入 MySQL 后再设置
# 输出 mysql 间接登录即可

# 创立 export mysql 用户
create user 'mysql_monitor'@'localhost' identified by 'xxxx';
GRANT REPLICATION CLIENT, PROCESS ON *.* TO 'mysql_monitor'@'localhost';
GRANT SELECT ON performance_schema.* TO 'mysql_monitor'@'localhost';

# 批改明码
ALTER USER mysql_monitor@localhost IDENTIFIED WITH mysql_native_password BY 'xxxxx';

# 创立用户并受权
create user 'test'@'%' identified by 'xxxxxxxxxxxxxxx';
CREATE DATABASE `my_db`;
grant all privileges on my_db.* to 'test'@'%' with grant option; 

# 设置 root 明码
alter user 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'xxxxxxxxxxxxxxx';

# 刷新权限
flush privileges; 
# 部署 redis
wget https://download.redis.io/releases/redis-5.0.14.tar.gz
tar -zxvf redis-5.0.14.tar.gz
cd redis-5.0.14/
vim ./redis.conf
make install
./src/redis-server redis.conf

promethues 和 grafana 的部署大家能够自行补充,版本抉择比拟随便,与我的项目关联不大,上面是一个 grafana 模板,用了良久挺不错的,有须要的能够拿去尝试。


{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "limit": 100,
        "name": "Annotations & Alerts",
        "showIn": 0,
        "type": "dashboard"
      },
      {
        "datasource": "Prometheus",
        "enable": true,
        "expr": "resets(process_uptime_seconds{application=\"$application\", instance=\"$instance\"}[1m]) > 0",
        "iconColor": "rgba(255, 96, 96, 1)",
        "name": "Restart Detection",
        "showIn": 0,
        "step": "1m",
        "tagKeys": "restart-tag",
        "textFormat": "uptime reset",
        "titleFormat": "Restart"
      }
    ]
  },
  "description": "Dashboard for Micrometer instrumented applications (Java, Spring Boot, Micronaut)",
  "editable": true,
  "gnetId": 4701,
  "graphTooltip": 1,
  "id": 4,
  "iteration": 1680770549619,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 125,
      "panels": [],
      "repeat": null,
      "title": "Quick Facts",
      "type": "row"
    },
    {
      "cacheTimeout": null,
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
          "decimals": 1,
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {"text": "N/A"}
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []},
      "gridPos": {
        "h": 3,
        "w": 6,
        "x": 0,
        "y": 1
      },
      "id": 63,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": ["lastNotNull"],
          "fields": "","values": false
        },
        "text": {},
        "textMode": "auto"
      },
      "pluginVersion": "8.0.3",
      "targets": [
        {"expr": "process_uptime_seconds{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "","metric":"",
          "refId": "A",
          "step": 14400
        }
      ],
      "title": "Uptime",
      "type": "stat"
    },
    {
      "cacheTimeout": null,
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {"text": "N/A"}
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "dateTimeAsIso"
        },
        "overrides": []},
      "gridPos": {
        "h": 3,
        "w": 6,
        "x": 6,
        "y": 1
      },
      "id": 92,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": ["lastNotNull"],
          "fields": "","values": false
        },
        "text": {"valueSize": 33},
        "textMode": "auto"
      },
      "pluginVersion": "8.0.3",
      "targets": [
        {"expr": "process_start_time_seconds{application=\"$application\", instance=\"$instance\"}*1000",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "","metric":"",
          "refId": "A",
          "step": 14400
        }
      ],
      "title": "Start time",
      "type": "stat"
    },
    {
      "cacheTimeout": null,
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
          "decimals": 2,
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {"text": "N/A"}
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {"color": "rgba(50, 172, 45, 0.97)",
                "value": null
              },
              {"color": "rgba(237, 129, 40, 0.89)",
                "value": 70
              },
              {"color": "rgba(245, 54, 54, 0.9)",
                "value": 90
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []},
      "gridPos": {
        "h": 3,
        "w": 6,
        "x": 12,
        "y": 1
      },
      "id": 65,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": ["lastNotNull"],
          "fields": "","values": false
        },
        "text": {},
        "textMode": "auto"
      },
      "pluginVersion": "8.0.3",
      "targets": [
        {"expr": "sum(jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\", area=\"heap\"})*100/sum(jvm_memory_max_bytes{application=\"$application\",instance=\"$instance\", area=\"heap\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "","refId":"A","step": 14400
        }
      ],
      "title": "Heap used",
      "type": "stat"
    },
    {
      "cacheTimeout": null,
      "datasource": "Prometheus",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
          "decimals": 2,
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {"text": "N/A"}
              },
              "type": "special"
            },
            {
              "options": {
                "from": -1e+32,
                "result": {"text": "N/A"},
                "to": 0
              },
              "type": "range"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {"color": "rgba(50, 172, 45, 0.97)",
                "value": null
              },
              {"color": "rgba(237, 129, 40, 0.89)",
                "value": 70
              },
              {"color": "rgba(245, 54, 54, 0.9)",
                "value": 90
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []},
      "gridPos": {
        "h": 3,
        "w": 6,
        "x": 18,
        "y": 1
      },
      "id": 75,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": ["lastNotNull"],
          "fields": "","values": false
        },
        "text": {},
        "textMode": "auto"
      },
      "pluginVersion": "8.0.3",
      "targets": [
        {"expr": "sum(jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\", area=\"nonheap\"})*100/sum(jvm_memory_max_bytes{application=\"$application\",instance=\"$instance\", area=\"nonheap\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "","refId":"A","step": 14400
        }
      ],
      "title": "Non-Heap used",
      "type": "stat"
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 4
      },
      "id": 126,
      "panels": [],
      "repeat": null,
      "title": "I/O Overview",
      "type": "row"
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 0,
        "y": 5
      },
      "hiddenSeries": false,
      "id": 111,
      "legend": {
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "sum(rate(http_server_requests_seconds_count{application=\"$application\", instance=\"$instance\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "HTTP",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Rate",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "yaxes": [
        {
          "decimals": null,
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "HTTP": "#890f02",
        "HTTP - 5xx": "#bf1b00"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 6,
        "y": 5
      },
      "hiddenSeries": false,
      "id": 112,
      "legend": {
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "sum(rate(http_server_requests_seconds_count{application=\"$application\", instance=\"$instance\", status=~\"5..\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "HTTP - 5xx",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Errors",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "yaxes": [
        {
          "decimals": null,
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 12,
        "y": 5
      },
      "hiddenSeries": false,
      "id": 113,
      "legend": {
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "sum(rate(http_server_requests_seconds_sum{application=\"$application\", instance=\"$instance\", status!~\"5..\"}[1m]))/sum(rate(http_server_requests_seconds_count{application=\"$application\", instance=\"$instance\", status!~\"5..\"}[1m]))",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "HTTP - AVG",
          "refId": "A"
        },
        {"expr": "max(http_server_requests_seconds_max{application=\"$application\", instance=\"$instance\", status!~\"5..\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "HTTP - MAX",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Duration",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "yaxes": [
        {
          "format": "s",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "description": "","fill": 1,"fillGradient": 0,"gridPos": {"h": 7,"w": 6,"x": 18,"y": 5},
      "hiddenSeries": false,
      "id": 119,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "tomcat_threads_busy_threads{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
          "legendFormat": "TOMCAT - BSY",
          "refId": "A"
        },
        {"expr": "tomcat_threads_current_threads{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
          "legendFormat": "TOMCAT - CUR",
          "refId": "B"
        },
        {"expr": "tomcat_threads_config_max_threads{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
          "legendFormat": "TOMCAT - MAX",
          "refId": "C"
        },
        {"expr": "jetty_threads_busy{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
          "legendFormat": "JETTY - BSY",
          "refId": "D"
        },
        {"expr": "jetty_threads_current{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
          "legendFormat": "JETTY - CUR",
          "refId": "E"
        },
        {"expr": "jetty_threads_config_max{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
          "legendFormat": "JETTY - MAX",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Utilisation",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 12
      },
      "id": 127,
      "panels": [],
      "repeat": null,
      "title": "JVM Memory",
      "type": "row"
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 0,
        "y": 13
      },
      "hiddenSeries": false,
      "id": 24,
      "legend": {
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "sum(jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\", area=\"heap\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "used",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "sum(jvm_memory_committed_bytes{application=\"$application\", instance=\"$instance\", area=\"heap\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "committed",
          "refId": "B",
          "step": 2400
        },
        {"expr": "sum(jvm_memory_max_bytes{application=\"$application\", instance=\"$instance\", area=\"heap\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "max",
          "refId": "C",
          "step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "JVM Heap",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "mbytes",
        "short"
      ],
      "yaxes": [
        {
          "format": "bytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 6,
        "y": 13
      },
      "hiddenSeries": false,
      "id": 25,
      "legend": {
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "sum(jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\", area=\"nonheap\"})",
          "format": "time_series",
          "interval": "","intervalFactor": 2,"legendFormat":"used","metric":"",
          "refId": "A",
          "step": 2400
        },
        {"expr": "sum(jvm_memory_committed_bytes{application=\"$application\", instance=\"$instance\", area=\"nonheap\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "committed",
          "refId": "B",
          "step": 2400
        },
        {"expr": "sum(jvm_memory_max_bytes{application=\"$application\", instance=\"$instance\", area=\"nonheap\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "max",
          "refId": "C",
          "step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "JVM Non-Heap",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "mbytes",
        "short"
      ],
      "yaxes": [
        {
          "format": "bytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 12,
        "y": 13
      },
      "hiddenSeries": false,
      "id": 26,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "sum(jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "used",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "sum(jvm_memory_committed_bytes{application=\"$application\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "committed",
          "refId": "B",
          "step": 2400
        },
        {"expr": "sum(jvm_memory_max_bytes{application=\"$application\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "max",
          "refId": "C",
          "step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "JVM Total",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "mbytes",
        "short"
      ],
      "yaxes": [
        {
          "format": "bytes",
          "label": "","logBase": 1,"max": null,"min": 0,"show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 18,
        "y": 13
      },
      "hiddenSeries": false,
      "id": 86,
      "legend": {
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "process_memory_vss_bytes{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 2,
          "legendFormat": "vss",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "process_memory_rss_bytes{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "rss",
          "refId": "B"
        },
        {"expr": "process_memory_swap_bytes{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "swap",
          "refId": "C"
        },
        {"expr": "process_memory_rss_bytes{application=\"$application\", instance=\"$instance\"} + process_memory_swap_bytes{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "total",
          "refId": "D"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "JVM Process Memory",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "mbytes",
        "short"
      ],
      "yaxes": [
        {
          "format": "bytes",
          "label": "","logBase": 1,"max": null,"min":"0","show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 20
      },
      "id": 128,
      "panels": [],
      "repeat": null,
      "title": "JVM Misc",
      "type": "row"
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 0,
        "y": 21
      },
      "hiddenSeries": false,
      "id": 106,
      "legend": {
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "system_cpu_usage{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "system",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "process_cpu_usage{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "process",
          "refId": "B"
        },
        {"expr": "avg_over_time(process_cpu_usage{application=\"$application\", instance=\"$instance\"}[1h])",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "process-1h",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "CPU Usage",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "decimals": 1,
          "format": "percentunit",
          "label": "","logBase": 1,"max":"1","min": 0,"show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 6,
        "y": 21
      },
      "hiddenSeries": false,
      "id": 93,
      "legend": {
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "system_load_average_1m{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "system-1m",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "system_cpu_count{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "cpus",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Load",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "decimals": 1,
          "format": "short",
          "label": "","logBase": 1,"max": null,"min": 0,"show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 12,
        "y": 21
      },
      "hiddenSeries": false,
      "id": 32,
      "legend": {
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_threads_live_threads{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "live",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "jvm_threads_daemon_threads{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "daemon",
          "metric": "","refId":"B","step": 2400
        },
        {"expr": "jvm_threads_peak_threads{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "peak",
          "refId": "C",
          "step": 2400
        },
        {"expr": "process_threads{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "interval": "","intervalFactor": 2,"legendFormat":"process","refId":"D","step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Threads",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "decimals": 0,
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "blocked": "#bf1b00",
        "new": "#fce2de",
        "runnable": "#7eb26d",
        "terminated": "#511749",
        "timed-waiting": "#c15c17",
        "waiting": "#eab839"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 18,
        "y": 21
      },
      "hiddenSeries": false,
      "id": 124,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "rightSide": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_threads_states_threads{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{state}}",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Thread States",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "debug": "#1F78C1",
        "error": "#BF1B00",
        "info": "#508642",
        "trace": "#6ED0E0",
        "warn": "#EAB839"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 18,
        "x": 0,
        "y": 28
      },
      "height": "","hiddenSeries": false,"id": 91,"legend": {"alignAsTable": false,"avg": false,"current": true,"hideEmpty": false,"hideZero": false,"max": true,"min": false,"rightSide": false,"show": true,"total": false,"values": true},
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": true,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "error",
          "yaxis": 1
        },
        {
          "alias": "warn",
          "yaxis": 1
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "increase(logback_events_total{application=\"$application\", instance=\"$instance\"}[1m])",
          "format": "time_series",
          "interval": "","intervalFactor": 2,"legendFormat":"{{level}}","metric":"",
          "refId": "A",
          "step": 1200
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Log Events",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "decimals": 0,
          "format": "opm",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 18,
        "y": 28
      },
      "hiddenSeries": false,
      "id": 61,
      "legend": {
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "process_files_open_files{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
          "legendFormat": "open",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "process_files_max_files{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
          "legendFormat": "max",
          "metric": "","refId":"B","step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "File Descriptors",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "decimals": 0,
          "format": "short",
          "label": null,
          "logBase": 10,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 35
      },
      "id": 129,
      "panels": [],
      "repeat": "persistence_counts",
      "title": "JVM Memory Pools (Heap)",
      "type": "row"
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 8,
        "x": 0,
        "y": 36
      },
      "hiddenSeries": false,
      "id": 3,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "rightSide": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "maxPerRow": 3,
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "repeat": "jvm_memory_pool_heap",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\", id=~\"$jvm_memory_pool_heap\"}",
          "format": "time_series",
          "hide": false,
          "interval": "","intervalFactor": 2,"legendFormat":"used","metric":"",
          "refId": "A",
          "step": 1800
        },
        {"expr": "jvm_memory_committed_bytes{application=\"$application\", instance=\"$instance\", id=~\"$jvm_memory_pool_heap\"}",
          "format": "time_series",
          "hide": false,
          "interval": "","intervalFactor": 2,"legendFormat":"commited","metric":"",
          "refId": "B",
          "step": 1800
        },
        {"expr": "jvm_memory_max_bytes{application=\"$application\", instance=\"$instance\", id=~\"$jvm_memory_pool_heap\"}",
          "format": "time_series",
          "hide": false,
          "interval": "","intervalFactor": 2,"legendFormat":"max","metric":"",
          "refId": "C",
          "step": 1800
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "$jvm_memory_pool_heap",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "mbytes",
        "short"
      ],
      "yaxes": [
        {
          "format": "bytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 43
      },
      "id": 130,
      "panels": [],
      "repeat": null,
      "title": "JVM Memory Pools (Non-Heap)",
      "type": "row"
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 8,
        "x": 0,
        "y": 44
      },
      "hiddenSeries": false,
      "id": 78,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": true,
        "min": false,
        "rightSide": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "maxPerRow": 3,
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "repeat": "jvm_memory_pool_nonheap",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\", id=~\"$jvm_memory_pool_nonheap\"}",
          "format": "time_series",
          "hide": false,
          "interval": "","intervalFactor": 2,"legendFormat":"used","metric":"",
          "refId": "A",
          "step": 1800
        },
        {"expr": "jvm_memory_committed_bytes{application=\"$application\", instance=\"$instance\", id=~\"$jvm_memory_pool_nonheap\"}",
          "format": "time_series",
          "hide": false,
          "interval": "","intervalFactor": 2,"legendFormat":"commited","metric":"",
          "refId": "B",
          "step": 1800
        },
        {"expr": "jvm_memory_max_bytes{application=\"$application\", instance=\"$instance\", id=~\"$jvm_memory_pool_nonheap\"}",
          "format": "time_series",
          "hide": false,
          "interval": "","intervalFactor": 2,"legendFormat":"max","metric":"",
          "refId": "C",
          "step": 1800
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "$jvm_memory_pool_nonheap",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "mbytes",
        "short"
      ],
      "yaxes": [
        {
          "format": "bytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 51
      },
      "id": 131,
      "panels": [],
      "repeat": null,
      "title": "Garbage Collection",
      "type": "row"
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 8,
        "x": 0,
        "y": 52
      },
      "hiddenSeries": false,
      "id": 98,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "rate(jvm_gc_pause_seconds_count{application=\"$application\", instance=\"$instance\"}[1m])",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{action}} ({{cause}})",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Collections",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "yaxes": [
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": "","logBase": 1,"max": null,"min": null,"show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 8,
        "x": 8,
        "y": 52
      },
      "hiddenSeries": false,
      "id": 101,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "rate(jvm_gc_pause_seconds_sum{application=\"$application\", instance=\"$instance\"}[1m])/rate(jvm_gc_pause_seconds_count{application=\"$application\", instance=\"$instance\"}[1m])",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "intervalFactor": 1,
          "legendFormat": "avg {{action}} ({{cause}})",
          "refId": "A"
        },
        {"expr": "jvm_gc_pause_seconds_max{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "intervalFactor": 1,
          "legendFormat": "max {{action}} ({{cause}})",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Pause Durations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "yaxes": [
        {
          "format": "s",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": "","logBase": 1,"max": null,"min": null,"show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 8,
        "x": 16,
        "y": 52
      },
      "hiddenSeries": false,
      "id": 99,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "rate(jvm_gc_memory_allocated_bytes_total{application=\"$application\", instance=\"$instance\"}[1m])",
          "format": "time_series",
          "interval": "","intervalFactor": 1,"legendFormat":"allocated","refId":"A"
        },
        {"expr": "rate(jvm_gc_memory_promoted_bytes_total{application=\"$application\", instance=\"$instance\"}[1m])",
          "format": "time_series",
          "interval": "","intervalFactor": 1,"legendFormat":"promoted","refId":"B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Allocated/Promoted",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "yaxes": [
        {
          "format": "Bps",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 59
      },
      "id": 132,
      "panels": [],
      "repeat": null,
      "title": "Classloading",
      "type": "row"
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 0,
        "y": 60
      },
      "hiddenSeries": false,
      "id": 37,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_classes_loaded_classes{application=\"$application\", instance=\"$instance\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "loaded",
          "metric": "","refId":"A","step": 1200
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Classes loaded",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 12,
        "x": 12,
        "y": 60
      },
      "hiddenSeries": false,
      "id": 38,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "delta(jvm_classes_loaded_classes{application=\"$application\",instance=\"$instance\"}[1m])",
          "format": "time_series",
          "hide": false,
          "interval": "","intervalFactor": 1,"legendFormat":"delta-1m","metric":"",
          "refId": "A",
          "step": 1200
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Class delta",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "ops",
        "short"
      ],
      "yaxes": [
        {
          "decimals": null,
          "format": "short",
          "label": "","logBase": 1,"max": null,"min": null,"show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 67
      },
      "id": 133,
      "panels": [],
      "repeat": null,
      "title": "Buffer Pools",
      "type": "row"
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 0,
        "y": 68
      },
      "hiddenSeries": false,
      "id": 33,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_buffer_memory_used_bytes{application=\"$application\", instance=\"$instance\", id=\"direct\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "used",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "jvm_buffer_total_capacity_bytes{application=\"$application\", instance=\"$instance\", id=\"direct\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "capacity",
          "metric": "","refId":"B","step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Direct Buffers",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "format": "bytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 6,
        "y": 68
      },
      "hiddenSeries": false,
      "id": 83,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_buffer_count_buffers{application=\"$application\", instance=\"$instance\", id=\"direct\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "count",
          "metric": "","refId":"A","step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Direct Buffers",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "decimals": 0,
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 12,
        "y": 68
      },
      "hiddenSeries": false,
      "id": 85,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_buffer_memory_used_bytes{application=\"$application\", instance=\"$instance\", id=\"mapped\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "used",
          "metric": "","refId":"A","step": 2400
        },
        {"expr": "jvm_buffer_total_capacity_bytes{application=\"$application\", instance=\"$instance\", id=\"mapped\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "capacity",
          "metric": "","refId":"B","step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Mapped Buffers",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "format": "bytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {"aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Prometheus",
      "editable": true,
      "error": false,
      "fill": 1,
      "fillGradient": 0,
      "grid": {
        "leftLogBase": 1,
        "leftMax": null,
        "leftMin": null,
        "rightLogBase": 1,
        "rightMax": null,
        "rightMin": null
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 18,
        "y": 68
      },
      "hiddenSeries": false,
      "id": 84,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {"alertThreshold": true},
      "percentage": false,
      "pluginVersion": "8.0.3",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {"expr": "jvm_buffer_count_buffers{application=\"$application\", instance=\"$instance\", id=\"mapped\"}",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "count",
          "metric": "","refId":"A","step": 2400
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Mapped Buffers",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "x-axis": true,
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []},
      "y-axis": true,
      "y_formats": [
        "short",
        "short"
      ],
      "yaxes": [
        {
          "decimals": 0,
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": "5s",
  "schemaVersion": 30,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {
          "selected": false,
          "text": "服务",
          "value": "服务"
        },
        "datasource": "Prometheus",
        "definition": "label_values(job)",
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "job",
        "options": [],
        "query": {"query": "label_values(job)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "","skipUrlSync": false,"sort": 0,"type":"query"
      },
      {
        "allValue": null,
        "current": {
          "selected": false,
          "text": "xld-business-user",
          "value": "xld-business-user"
        },
        "datasource": "Prometheus",
        "definition": "label_values(application)",
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Application",
        "multi": false,
        "name": "application",
        "options": [],
        "query": {"query": "label_values(application)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 2,
        "regex": "","skipUrlSync": false,"sort": 0,"tagValuesQuery":"",
        "tagsQuery": "","type":"query","useTags": false
      },
      {
        "allFormat": "glob",
        "allValue": null,
        "current": {
          "selected": false,
          "text": "172.29.109.194:8081",
          "value": "172.29.109.194:8081"
        },
        "datasource": "Prometheus",
        "definition": "label_values(jvm_memory_used_bytes{application=\"$application\",job='$job'}, instance)",
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Instance",
        "multi": false,
        "multiFormat": "glob",
        "name": "instance",
        "options": [],
        "query": {"query": "label_values(jvm_memory_used_bytes{application=\"$application\",job='$job'}, instance)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 2,
        "regex": "","skipUrlSync": false,"sort": 0,"tagValuesQuery":"",
        "tagsQuery": "","type":"query","useTags": false
      },
      {
        "allFormat": "glob",
        "allValue": null,
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": "Prometheus",
        "definition": "","description": null,"error": null,"hide": 0,"includeAll": true,"label":"JVM Memory Pools Heap","multi": false,"multiFormat":"glob","name":"jvm_memory_pool_heap","options": [],"query": {"query":"label_values(jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\", area=\"heap\"},id)","refId":"Prometheus-jvm_memory_pool_heap-Variable-Query"},"refresh": 1,"regex":"",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "","tagsQuery":"",
        "type": "query",
        "useTags": false
      },
      {
        "allFormat": "glob",
        "allValue": null,
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": "Prometheus",
        "definition": "","description": null,"error": null,"hide": 0,"includeAll": true,"label":"JVM Memory Pools Non-Heap","multi": false,"multiFormat":"glob","name":"jvm_memory_pool_nonheap","options": [],"query": {"query":"label_values(jvm_memory_used_bytes{application=\"$application\", instance=\"$instance\", area=\"nonheap\"},id)","refId":"Prometheus-jvm_memory_pool_nonheap-Variable-Query"},"refresh": 1,"regex":"",
        "skipUrlSync": false,
        "sort": 2,
        "tagValuesQuery": "","tagsQuery":"",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {
    "now": true,
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "browser",
  "title": "服务监控",
  "uid": "_Yowz9B4z",
  "version": 3
}

Nacos Sentinel Seata 能够应用我的项目原有的基础设施。

开始压测

  1. 针对次要接口的申请放入 Jmeter 脚本中
  2. 零碎大多数接口都是基于登录用户,用户 Token 问题能够在后盾生成所有用户的 Token,放入一个文件中,在 Jmeter 中读取文件,设置一个循环器,循环取出文件中的 Token,在申请中设置申请头,寄存 Token,这样能够遍历用户申请接口。
  3. 倡议应用非界面的模式运行 Jmeter
  4. 如需更大的并发量,能够应用分布式形式部署 Jmeter
  5. 个别会先依照单并发的模式压测系统,再逐步调整并发度对接口测试。

报告剖析

  1. 聚合报告中的最大响应工夫和均匀响应工夫须要重点关注,个别接口有并发问题,随着并发度的逐步回升,最大响应工夫就会高的离谱,千万不要感到诧异,呈现并发问题后,申请响应能够有限增长。
  2. 除了聚合报告还要关注服务器的负载和网络带宽,另外留神中间件和应用服务器的负载是否平衡,比方应用服务器 cpu 负载 100%,Mysql 服务器的 CPU 负载基本没有稳定,那么就能够证实利用性能逻辑呈现问题,或者就是应用了缓存。
  3. 对于接口响应超过 1 秒的申请就属于并发问题了,要不就是利用呈现不明报错,这时候就能够断定性能有性能问题。
  4. 性能良好的接口,随着并发量的回升不 CPU 负载和其余中间件的负载会比拟均衡,比方应用服务器 CPU 100%,中间件的负载也会很高,这就是属于压力传递的状况,能够认为是失常景象。

总结

压测倡议:

  1. 不要适度优化和过早优化,业务性能的实现才是工作的重点。
  2. 甄别接口的申请频繁度,进行优化时,能够将接口应用的频繁水平作为是否优化的必要因素,避免出现没啥申请的接口也进行了优化,优化毕竟也须要老本,另外也会减少接口的复杂度,索引优化也不是做的越多越好。
  3. 优化前后必须有性能测试的数据撑持,避免出现优化不起作用的问题,所有改变必须通过测试能力上线,像优化这类改变,必须通过性能测试环节能力上线。

压测工作其实也没那么难以了解,做过几遍后感觉压测是真是一件有意思的事件,压测后果的剖析能够发现一些编码是发现不到的问题,每次压测都会有不一样的体验,喜爱压测工作的小伙伴能够留言议论。

退出移动版