乐趣区

关于springboot:SpringBoot应用ELK日志收集

作者：

jiezi

在

服务初始化相干的配置

  • 服务初始化相干的配置

    • Docker装置

      • 装置yum-utils（装置docker-ce所需依赖）
      • 为yum源增加docker仓库地位
      • 装置docker
      • 启动docker

    • Elasticsearch装置

      • 应用如下命令启动Elasticsearch服务
      • 启动的时候会报错
      • 装置中文分词器IKAnalyzer，并重新启动
      • 开启防火墙
      • 拜访会返回版本信息：http://39.103.203.41:9200/

    • Logstash装置

      • 下载Logstash7.6.2的docker镜像
      • 如下配置文件logstash.conf
      • 创立/mydata/logstash目录，并将Logstash的配置文件logstash.conf拷贝到该目录
      • 应用如下命令启动Logstash服务
      • 在logstash中装置json_lines插件

    • Kibana装置

      • 下载Kibana7.6.2的docker镜像
      • 应用如下命令启动Kibana服务
      • 拜访地址进行测试 http://39.103.203.41:5601/
      • SpringBoot利用集成Logstash
      • 增加配置文件logback-spring.xml让logback的日志输入到logstash
      • 运行SpringBoot利用

    • 在kibana中查看日志信息

      • 创立index pattern
      • 调用该接口并查看日志

Docker装置

(不同的操作系统能够参考 https://yeasy.gitbook.io/dock…)

装置yum-utils（装置docker-ce所需依赖）

 yum install -y yum-utils device-mapper-persistent-data lvm2

为yum源增加docker仓库地位

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

yum.repos.d

# 仓库源寄存的地位
cd /etc/yum.repos.d 

vim docker-ce.repo

装置docker

⚠️留神 docker-ce与docker两个仓库版本有差别（自己爬坑比拟多 docker不反对文件的挂载比拟老的版本）

 yum install docker-ce

# 如果报错以下信息
errors during downloading metadata for repository 'base':
  - Curl error (28): Timeout was reached for http://mirrors.aliyuncs.com/centos/3/os/x86_64/repodata/repomd.xml [Connection timed out after 30001 milliseconds]
  - Status code: 404 for http://mirrors.cloud.aliyuncs.com/centos/3/os/x86_64/repodata/repomd.xml (IP: 100.100.2.148)
  - Status code: 404 for http://mirrors.aliyun.com/centos/3/os/x86_64/repodata/repomd.xml (IP: 39.96.118.193)
Error: Failed to download metadata for repo 'base': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

# 将/etc/yum.repos.d/docker-ce.repo的地址进行批改
# 将数据源替换成版本7 不然下载的时候会报错 链接地址不存在.
baseurl=https://download.docker.com/linux/centos/7/debug-$basearch/stable

启动docker

 systemctl start docker

Elasticsearch装置

应用如下命令启动Elasticsearch服务

docker run -p 9200:9200 -p 9300:9300 --name elasticsearch \
-e "discovery.type=single-node" \
-e "cluster.name=elasticsearch" \
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \
-d elasticsearch:7.6.2

启动的时候会报错

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2021-06-03T11:24:02,468Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "4f5aaf7716c0", "message": "uncaught exception in thread [main]", 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.2.jar:7.6.2]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:615) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:274) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:211) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more"] }
uncaught exception in thread [main]

启动时会发现/usr/share/elasticsearch/data目录没有拜访权限，只须要批改/mydata/elasticsearch/data目录的权限，再重新启动即可

chmod 777 /mydata/elasticsearch/data/

装置中文分词器IKAnalyzer，并重新启动


# 进入docker容器中
docker exec -it elasticsearch /bin/bash

#此命令须要在容器中运行
elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip
-> Installing https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip
-> Downloading https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip
[=================================================] 100%?? 
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed analysis-ik

# 退出去后重启失效
docker restart elasticsearch

开启防火墙

firewall-cmd --zone=public --add-port=9200/tcp --permanent
firewall-cmd --reload

拜访会返回版本信息：http://39.103.203.41:9200/

{
  "name" : "4f5aaf7716c0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "PBB0ZytDStO-9SJjAarNyw",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Logstash装置

下载Logstash7.6.2的docker镜像

docker pull logstash:7.6.2

如下配置文件logstash.conf

input {
  tcp {

    mode => "server"
    host => "0.0.0.0"
    port => 4560
    codec => json_lines
    type => "debug"

  }
  tcp {

    mode => "server"
    host => "0.0.0.0"
    port => 4561
    codec => json_lines
    type => "error"

  }
  tcp {

    mode => "server"
    host => "0.0.0.0"
    port => 4562
    codec => json_lines
    type => "business"

  }
  tcp {

    mode => "server"
    host => "0.0.0.0"
    port => 4563
    codec => json_lines
    type => "record"

  }
}
filter{
  if [type] == "record" {

    mutate {
      remove_field => "port"
      remove_field => "host"
      remove_field => "@version"
    }
    json {
      source => "message"
      remove_field => ["message"]
    }

  }
}
output {
  elasticsearch {
    hosts => "39.103.203.41:9200"
    index => "mall-%{type}-%{+YYYY.MM.dd}"

  }
}

创立/mydata/logstash目录，并将Logstash的配置文件logstash.conf拷贝到该目录

mkdir /mydata/logstash

应用如下命令启动Logstash服务

docker run --name logstash -p 4560:4560 -p 4561:4561 -p 4562:4562 -p 4563:4563 \
--link elasticsearch:es \
-v /mydata/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
-d logstash:7.6.2

在logstash中装置json_lines插件

# 进入logstash容器
docker exec -it logstash /bin/bash
# 进入bin目录
cd /bin/
# 装置插件
logstash-plugin install logstash-codec-json_lines

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Validating logstash-codec-json_lines
Installing logstash-codec-json_lines
Installation successful
# 退出容器
exit
# 重启logstash服务
docker restart logstash

Kibana装置

下载Kibana7.6.2的docker镜像

7.6.2: Pulling from library/kibana
ab5ef0e58194: Already exists 
c64d415fc4c4: Pull complete 
40a228497f87: Pull complete 
047cebeb3d2b: Pull complete 
a1e90407e522: Pull complete 
b665bda75e65: Pull complete 
12bc27d9cfdc: Pull complete 
2611a8427d9d: Pull complete 
12efd486dee3: Pull complete 
d2dfc5062b56: Pull complete 
Digest: sha256:097e2b7f33f353a8fc19bbf2a6558431c63637113fdc625e6d34fc46f96c0130
Status: Downloaded newer image for kibana:7.6.2
docker.io/library/kibana:7.6.2

应用如下命令启动Kibana服务

docker run --name kibana -p 5601:5601 \
--link elasticsearch:es \
-e "elasticsearch.hosts=http://es:9200" \
-d kibana:7.6.2

拜访地址进行测试 http://39.103.203.41:5601/

SpringBoot利用集成Logstash

<!--集成logstash-->
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>5.3</version>
</dependency>

增加配置文件logback-spring.xml让logback的日志输入到logstash

留神appender节点下的destination须要改成你本人的logstash服务地址，例如我本人的是：39.103.203.41:4560

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
    <!--利用名称-->
    <property name="APP_NAME" value="mall-admin"/>
    <!--日志文件保留门路-->
    <property name="LOG_FILE_PATH" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/logs}"/>
    <contextName>${APP_NAME}</contextName>

    <!--每天记录日志到文件appender-->
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOG_FILE_PATH}/${APP_NAME}-%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>${FILE_LOG_PATTERN}</pattern>
        </encoder>
    </appender>

    <!--输入到logstash的appender-->
    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <!--能够拜访的logstash日志收集端口-->
        <destination>39.103.203.41:4560</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
        <appender-ref ref="LOGSTASH"/>
    </root>
</configuration>

运行SpringBoot利用

package com.macro.mall.tiny;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class MallApplication {

 public static void main(String[] args) {
  SpringApplication.run(MallApplication.class, args);
 }

}

在kibana中查看日志信息

创立index pattern




调用该接口并查看日志


评论

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注

这个站点使用 Akismet 来减少垃圾评论。了解你的评论数据如何被处理。

更多文章