前言
Elasticsearch 是一个开源的分布式 RESTful 搜索和分析引擎。它可以在近实时条件下,存储,查询和分析海量的数据。它还支持将快照备份至 HDFS/S3 上面,而阿里云 OSS 兼容 S3 的 API,本文将介绍如何使用 ES 的 Repository-S3 插件将快照备份至 OSS。
部署与配置
首先,我们需要安装 repository-s3,可以参考官方文档:
https://www.elastic.co/guide/en/elasticsearch/plugins/7.2/repository-s3.html
启动 ES,我们可以从 log 中看到,ES 已经 load 了这个 plugin:
[2019-07-15T14:12:09,225][INFO][o.e.p.PluginsService] [master] loaded module [aggs-matrix-stats]
[2019-07-15T14:12:09,225][INFO][o.e.p.PluginsService] [master] loaded module [analysis-common]
[2019-07-15T14:12:09,225][INFO][o.e.p.PluginsService] [master] loaded module [ingest-common]
[2019-07-15T14:12:09,226][INFO][o.e.p.PluginsService] [master] loaded module [ingest-geoip]
[2019-07-15T14:12:09,226][INFO][o.e.p.PluginsService] [master] loaded module [ingest-user-agent]
[2019-07-15T14:12:09,226][INFO][o.e.p.PluginsService] [master] loaded module [lang-expression]
[2019-07-15T14:12:09,226][INFO][o.e.p.PluginsService] [master] loaded module [lang-mustache]
[2019-07-15T14:12:09,227][INFO][o.e.p.PluginsService] [master] loaded module [lang-painless]
[2019-07-15T14:12:09,227][INFO][o.e.p.PluginsService] [master] loaded module [mapper-extras]
[2019-07-15T14:12:09,227][INFO][o.e.p.PluginsService] [master] loaded module [parent-join]
[2019-07-15T14:12:09,227][INFO][o.e.p.PluginsService] [master] loaded module [percolator]
[2019-07-15T14:12:09,227][INFO][o.e.p.PluginsService] [master] loaded module [rank-eval]
[2019-07-15T14:12:09,228][INFO][o.e.p.PluginsService] [master] loaded module [reindex]
[2019-07-15T14:12:09,228][INFO][o.e.p.PluginsService] [master] loaded module [repository-url]
[2019-07-15T14:12:09,228][INFO][o.e.p.PluginsService] [master] loaded module [transport-netty4]
[2019-07-15T14:12:09,228][INFO][o.e.p.PluginsService] [master] loaded plugin [repository-s3]
[2019-07-15T14:12:12,375][INFO][o.e.d.DiscoveryModule] [master] using discovery type [zen] and seed hosts providers [settings]
[2019-07-15T14:12:12,801][INFO][o.e.n.Node] [master] initialized
[2019-07-15T14:12:12,802][INFO][o.e.n.Node] [master] starting ...
然后,我们需要将 OSS 使用的 Access Key 和 Secret Key 配置到 ES 去,分别执行下面的命令:
bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key
运行
首先,我们创建一个备份:
[root@master ~]# curl -XPUT 'http://localhost:9200/_snapshot/test' -H 'Content-Type: application/json' -d '{"type":"s3","settings": {"bucket":"hadoop-oss-test","endpoint":"oss-cn-zhangjiakou-internal.aliyuncs.com"} }'
{"acknowledged":true}
NOTE:
上面的命令默认使用 https 协议来传输数据,如果想使用 http 协议,需要将 "protocol": "http", "disable_chunked_encoding": true
加到 settings
里面(这个特性将会在新版本发布后可用)。
可以使用下面的命令来确实创建是否成功:
[root@master ~]# curl -XGET localhost:9200/_snapshot/test?pretty
{
"test" : {
"type" : "s3",
"settings" : {
"bucket" : "hadoop-oss-test",
"endpoint" : "oss-cn-zhangjiakou-internal.aliyuncs.com"
}
}
}
我们可以写入一些测试数据到 ES,然后看下目前集群的索引信息:
[root@master ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open sales 89ouBy6RQsuT34QRbn_jeQ 10 0 271786 0 15mb 15mb
green open customer fQCMEvXsQOu0UgMm1SAJlA 5 0 10000 0 717kb 717kb
假设我们只备份 sales 索引:
[root@master ~]# curl -XPUT 'http://localhost:9200/_snapshot/test/sales' -H 'Content-Type: application/json' -d '{"indices":"sales"}'
{"accepted":true}
然后我们可以从 OSS 控制台看到备份的结果:
现在我们再往 sales 索引里面写一些数据:
[root@master ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open sales 89ouBy6RQsuT34QRbn_jeQ 10 0 281502 0 15.6mb 15.6mb
green open customer fQCMEvXsQOu0UgMm1SAJlA 5 0 10000 0 717kb 717kb
我们利用刚才备份到 OSS 的快照来恢复 sales 索引,分别执行下面的命令:
[root@master ~]# curl -XPOST localhost:9200/sales/_close
{"acknowledged":true,"shards_acknowledged":true,"indices":{"sales":{"closed":true}}}
[root@master ~]# curl -XPOST 'http://localhost:9200/_snapshot/test/sales/_restore?pretty'
{"accepted" : true}
[root@master ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open sales 89ouBy6RQsuT34QRbn_jeQ 10 0 271786 0 15mb 15mb
green open customer fQCMEvXsQOu0UgMm1SAJlA 5 0 10000 0 717kb 717kb
我们可以看到,sales 索引跟之前的一致。
参考资料
https://www.elastic.co/guide/en/elasticsearch/plugins/7.2/repository-s3.html
https://www.elastic.co/cn/products/elasticsearch
本文作者:冷月_wjh
阅读原文
本文为云栖社区原创内容,未经允许不得转载。