关于springboot:SpringBoot应用ELK日志收集

3次阅读

共计 11236 个字符，预计需要花费 29 分钟才能阅读完成。

服务初始化相干的配置
- Docker 装置
  - 装置 yum-utils（装置 docker-ce 所需依赖）
  - 为 yum 源增加 docker 仓库地位
  - 装置 docker
  - 启动 docker
- Elasticsearch 装置
  - 应用如下命令启动 Elasticsearch 服务
  - 启动的时候会报错
  - 装置中文分词器 IKAnalyzer，并重新启动
  - 开启防火墙
  - 拜访会返回版本信息：http://39.103.203.41:9200/
- Logstash 装置
  - 下载 Logstash7.6.2 的 docker 镜像
  - 如下配置文件 logstash.conf
  - 创立 /mydata/logstash 目录，并将 Logstash 的配置文件 logstash.conf 拷贝到该目录
  - 应用如下命令启动 Logstash 服务
  - 在 logstash 中装置 json_lines 插件
- Kibana 装置
  - 下载 Kibana7.6.2 的 docker 镜像
  - 应用如下命令启动 Kibana 服务
  - 拜访地址进行测试 http://39.103.203.41:5601/
  - SpringBoot 利用集成 Logstash
  - 增加配置文件 logback-spring.xml 让 logback 的日志输入到 logstash
  - 运行 SpringBoot 利用
- 在 kibana 中查看日志信息
  - 创立 index pattern
  - 调用该接口并查看日志

(不同的操作系统能够参考 https://yeasy.gitbook.io/dock…)

 yum install -y yum-utils device-mapper-persistent-data lvm2

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

yum.repos.d

# 仓库源寄存的地位
cd /etc/yum.repos.d 

vim docker-ce.repo

⚠️留神 docker-ce 与 docker 两个仓库版本有差别（自己爬坑比拟多 docker 不反对文件的挂载比拟老的版本）

 yum install docker-ce

# 如果报错以下信息
errors during downloading metadata for repository 'base':
  - Curl error (28): Timeout was reached for http://mirrors.aliyuncs.com/centos/3/os/x86_64/repodata/repomd.xml [Connection timed out after 30001 milliseconds]
  - Status code: 404 for http://mirrors.cloud.aliyuncs.com/centos/3/os/x86_64/repodata/repomd.xml (IP: 100.100.2.148)
  - Status code: 404 for http://mirrors.aliyun.com/centos/3/os/x86_64/repodata/repomd.xml (IP: 39.96.118.193)
Error: Failed to download metadata for repo 'base': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

# 将 /etc/yum.repos.d/docker-ce.repo 的地址进行批改
# 将数据源替换成版本 7 不然下载的时候会报错 链接地址不存在.
baseurl=https://download.docker.com/linux/centos/7/debug-$basearch/stable

 systemctl start docker

docker run -p 9200:9200 -p 9300:9300 --name elasticsearch \
-e "discovery.type=single-node" \
-e "cluster.name=elasticsearch" \
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \
-d elasticsearch:7.6.2

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2021-06-03T11:24:02,468Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "4f5aaf7716c0", "message": "uncaught exception in thread [main]", 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.2.jar:7.6.2]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:615) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:274) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:211) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more"] }
uncaught exception in thread [main]

启动时会发现 /usr/share/elasticsearch/data 目录没有拜访权限，只须要批改 /mydata/elasticsearch/data 目录的权限，再重新启动即可

chmod 777 /mydata/elasticsearch/data/


# 进入 docker 容器中
docker exec -it elasticsearch /bin/bash

#此命令须要在容器中运行
elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip
-> Installing https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip
-> Downloading https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip
[=================================================] 100%?? 
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed analysis-ik

# 退出去后重启失效
docker restart elasticsearch

firewall-cmd --zone=public --add-port=9200/tcp --permanent
firewall-cmd --reload

{
  "name" : "4f5aaf7716c0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "PBB0ZytDStO-9SJjAarNyw",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

docker pull logstash:7.6.2

input {
  tcp {

    mode => "server"
    host => "0.0.0.0"
    port => 4560
    codec => json_lines
    type => "debug"

  }
  tcp {

    mode => "server"
    host => "0.0.0.0"
    port => 4561
    codec => json_lines
    type => "error"

  }
  tcp {

    mode => "server"
    host => "0.0.0.0"
    port => 4562
    codec => json_lines
    type => "business"

  }
  tcp {

    mode => "server"
    host => "0.0.0.0"
    port => 4563
    codec => json_lines
    type => "record"

  }
}
filter{if [type] == "record" {

    mutate {
      remove_field => "port"
      remove_field => "host"
      remove_field => "@version"
    }
    json {
      source => "message"
      remove_field => ["message"]
    }

  }
}
output {
  elasticsearch {
    hosts => "39.103.203.41:9200"
    index => "mall-%{type}-%{+YYYY.MM.dd}"

  }
}

mkdir /mydata/logstash

docker run --name logstash -p 4560:4560 -p 4561:4561 -p 4562:4562 -p 4563:4563 \
--link elasticsearch:es \
-v /mydata/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
-d logstash:7.6.2

# 进入 logstash 容器
docker exec -it logstash /bin/bash
# 进入 bin 目录
cd /bin/
# 装置插件
logstash-plugin install logstash-codec-json_lines

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Validating logstash-codec-json_lines
Installing logstash-codec-json_lines
Installation successful
# 退出容器
exit
# 重启 logstash 服务
docker restart logstash

7.6.2: Pulling from library/kibana
ab5ef0e58194: Already exists 
c64d415fc4c4: Pull complete 
40a228497f87: Pull complete 
047cebeb3d2b: Pull complete 
a1e90407e522: Pull complete 
b665bda75e65: Pull complete 
12bc27d9cfdc: Pull complete 
2611a8427d9d: Pull complete 
12efd486dee3: Pull complete 
d2dfc5062b56: Pull complete 
Digest: sha256:097e2b7f33f353a8fc19bbf2a6558431c63637113fdc625e6d34fc46f96c0130
Status: Downloaded newer image for kibana:7.6.2
docker.io/library/kibana:7.6.2

docker run --name kibana -p 5601:5601 \
--link elasticsearch:es \
-e "elasticsearch.hosts=http://es:9200" \
-d kibana:7.6.2

<!-- 集成 logstash-->
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>5.3</version>
</dependency>

留神 appender 节点下的 destination 须要改成你本人的 logstash 服务地址，例如我本人的是：39.103.203.41:4560

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
    <!-- 利用名称 -->
    <property name="APP_NAME" value="mall-admin"/>
    <!-- 日志文件保留门路 -->
    <property name="LOG_FILE_PATH" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/logs}"/>
    <contextName>${APP_NAME}</contextName>

    <!-- 每天记录日志到文件 appender-->
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOG_FILE_PATH}/${APP_NAME}-%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>${FILE_LOG_PATTERN}</pattern>
        </encoder>
    </appender>

    <!-- 输入到 logstash 的 appender-->
    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <!-- 能够拜访的 logstash 日志收集端口 -->
        <destination>39.103.203.41:4560</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
        <appender-ref ref="LOGSTASH"/>
    </root>
</configuration>

package com.macro.mall.tiny;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class MallApplication {public static void main(String[] args) {SpringApplication.run(MallApplication.class, args);
 }

}

正文完

springboot

发表至： springboot

2021-06-07

0

关于springboot:SpringBoot基础之Spring-Data-Jpa

关于springboot:SpringBoot如何优雅地进行响应数据封装异常处理

关于springboot:springboot-RestTemplate不能注入的原因

关于springboot:08springboot工程下lombok的应用

关于kubernetes:Kubelet运行在容器下使用LocalPV磁盘地址不存在问题

关于springboot:SpringBoot应用ELK日志收集

服务初始化相干的配置

Docker 装置

装置 yum-utils（装置 docker-ce 所需依赖）

为 yum 源增加 docker 仓库地位

装置 docker

启动 docker

Elasticsearch 装置

应用如下命令启动 Elasticsearch 服务

启动的时候会报错

装置中文分词器 IKAnalyzer，并重新启动

开启防火墙

拜访会返回版本信息：http://39.103.203.41:9200/

Logstash 装置

下载 Logstash7.6.2 的 docker 镜像

如下配置文件 logstash.conf

创立 /mydata/logstash 目录，并将 Logstash 的配置文件 logstash.conf 拷贝到该目录

应用如下命令启动 Logstash 服务

在 logstash 中装置 json_lines 插件

Kibana 装置

下载 Kibana7.6.2 的 docker 镜像

应用如下命令启动 Kibana 服务

拜访地址进行测试 http://39.103.203.41:5601/

SpringBoot 利用集成 Logstash

增加配置文件 logback-spring.xml 让 logback 的日志输入到 logstash

运行 SpringBoot 利用

在 kibana 中查看日志信息

创立 index pattern

调用该接口并查看日志

站内搜索