1 引言
es，不必多说，基本上软件标配，即便不用其全文检索功能，作为ELK的组合之一也是常见，见此文：elk日志管理。
最新的发行版是7.5.1，依赖jre 11+，本文简介安装过程，系统是ubuntu 18.04
运行端口9200/9300，常用命令：
cd /usr/local/elasticsearch-7.5.1
su elk
bin/elasticsearch -d
2 下载两个安装包
-
es包elasticsearch-7.5.1：下载地址
-
jdk包jdk-11.0.5，windows版下载地址 ，linux版下载地址
3 安装jdk
解压即可，设置环境变量：vim /etc/profile，新增配置如下：
export JAVA_HOME=/usr/local/jdk-11.0.5
export PATH=$JAVA_HOME/bin:$PATH
使配置生效：source /etc/profile
4 安装es
解压即可，尝试运行：bin/elasticsearch，报错如下：
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-12-19T14:15:37,086][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [iZ8vbj9qj1ctpijae0bxunZ] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
-
es启动脚本里带了一些jdk不再支持的启动参数
-
不能用root用户去启动es
4.1 配置es用户
ubuntu下新增用户：
groupadd elk
useradd elk -g elk -p elk # 在组elk下新增用户elk，其密码为elk
chown -R elk.elk elasticsearch-7.5.1 # 更换所有者，这样才有启动权限
4.2 启动
启动之前，看是否有需要进行一些配置vim elasticsearch-7.5.1/config/elasticsearch.yml，比如绑定的IP，是否有跨域访问的需求等，注意以下配置：
network.host: 172.26.71.241 # 绑定的IP，默认是127.0.0.1
# 跨域访问，如es head等
http.cors.enabled: true
http.cors.allow-origin: "*"
更多配置也可以修改：
# Path to directory where to store the data (separate multiple locations by comma):
path.data: /usr/local/elasticsearch-7.5.1/data
# Path to log files:
path.logs: /usr/local/elasticsearch-7.5.1/logs
cd elasticsearch-7.5.1/
su elk # 切换用户
bin/elasticsearch # 启动es，加参数-d表示守护运行
4.3 一些操作
查看索引：curl 'yidu.seaeverit.com:9200/_cat/indices?v'，返回：
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana 6GJjyPeZQ5Cko4G1YuTEqg 1 1 5 0 21.5kb 21.5kb
yellow open nightlionsecurity.com N40sp-0RR9-HAaL0j8LjCA 1 1 0 0 283b 283b
yellow open toc_logstash hDxdQ9v3SWGU5DrWrp5dEw 1 1 89889 0 34.5mb 34.5mb
4.4 一些问题
- 启动时，报以下错误：
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决办法：在*/etc/sysctl.conf*文件最后添加一行：vm.max_map_count=262144，重启服务器（不重启可以试试：sysctl -p）
，查看命令：sysctl -a|grep vm.max_map_count，更多相关问题可以看这里
- 上一个解决后，第二个问题报：
[1]: the default discovery settings are unsuitable for production use; at least one of
[discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
打开一个，修改配置：discovery.seed_hosts: ["172.26.71.241"]
- 启动成功了，但是一直报：
master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster,
and [cluster.initial_master_nodes] is empty on this node: have discovered [{iZ8vbj9qj1ctpijae0bxunZ}{tn8yAs33Qa6ZRf1zPp5ing}{dVK5gLrGTjSSc_SYL6o8CA}
{172.26.71.241}{172.26.71.241:9300}{dilm}{ml.machine_memory=32655335424, xpack.installed=true, ml.max_open_jobs=20}];
discovery will continue using [] from hosts providers and [{iZ8vbj9qj1ctpijae0bxunZ}{tn8yAs33Qa6ZRf1zPp5ing}{dVK5gLrGTjSSc_SYL6o8CA}
{172.26.71.241}{172.26.71.241:9300}{dilm}{ml.machine_memory=32655335424, xpack.installed=true, ml.max_open_jobs=20}]
from last-known cluster state; node term 0, last-accepted version 0 in term 0
打开配置：cluster.initial_master_nodes: ["172.26.71.241"]
4.5 基本信息
es的基本信息查看，curl http://172.26.71.241:9200，如下：
{
"name" : "iZ8vbj9qj1ctpijae0bxunZ",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "CFqItLYzTL2PNQnu-ll9rQ",
"version" : {
"number" : "7.5.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
"build_date" : "2019-12-16T22:57:37.835892Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
坑：
- es的node满了？碰到的问题，跟这个相似，以下方法验证无效：
（1）退出只读模式
curl -X PUT 'xx.xxxx.com:9200/xxxx/_settings' -H 'Content-Type: application/json'
-d '{"index.blocks.read_only_allow_delete": null}'
返回{"acknowledged":true}
（2）重配：
curl -X PUT 'xxx.xxxxx.com:9200/_cluster/settings' -H 'Content-Type: application/json' -d
'{"transient":{"cluster.routing.allocation.disk.watermark.low":"100gb","cluster.routing.allocation.disk.watermark.high":"50gb","cluster.routing.allocation.disk.watermark.flood_stage":"10gb","cluster.info.update.interval":"1m"}}'
返回
{
"acknowledged": true,
"persistent": {},
"transient": {
"cluster": {
"routing": {
"allocation": {
"disk": {
"watermark": {
"low": "100gb",
"flood_stage": "10gb",
"high": "50gb"
}
}
}
},
"info": {
"update": {
"interval": "1m"
}
}
}
}
}
4.6 head插件安装
项目开源地址在这里。克隆代码：
git clone git://github.com/mobz/elasticsearch-head.git
环境nodjs v8.10.0，npm的版本是3.5.2，进入目录，运行npm install，还提示安装最新的nodejs和npm，出错：
Failed at the phantomjs-prebuilt@2.1.16 install script
安装老版本，npm install phantomjs-prebuilt@2.1.14 --ignore-scripts，过程比较漫长：
npm WARN deprecated phantomjs-prebuilt@2.1.14: this package is now deprecated
npm WARN deprecated hawk@3.1.3: This module moved to @hapi/hawk.
Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
再npm install，OK
后台运行：nohup npm start &
4.7 kibana安装
kibana来查看日志，核心传输是logstash，另外有介绍，见此文：es_logstash；核心存储使用的是es。 kibana的中文手删链接在这里
下载页在这里https://www.elastic.co/cn/downloads/kibana)，下载的最新版：7.5.1
解压，修改基本配置：vim config/kibana.yml，比如：
server.host: "172.26.71.241"
server.basePath: "/kibana"
elasticsearch.username: "kibana"
elasticsearch.password: "pass"
后台运行：nohup bin/kibana --allow-root --no-sandbox & # allow-root参数是允许root运行，默认不允许的
用elk运行，报错：
The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports.
4.8 es数据如何迁移
下面介绍的是通过logstash迁移数据，参考自这里 ，这个还是阿里云的英文版的技术介绍文档。
比如，新建下列logstash脚本：
input {
ElasticSearch {
hosts => ["********your host**********"]
user => "*******"
password => "*********"
index => "logstash-2017.11.07"
size => 1000
scroll => "1m"
}
}
# a note in this section indicates that filter can be selected
filter {
}
output {
ElasticSearch {
hosts => ["***********your host**************"]
user => "********"
password => "**********"
index => "logstash-2017.11.07"
}
}
我相应修改的版本是：
input {
elasticsearch {
hosts => "127.0.0.1:9200"
index => "toc_logstash"
size => 1000
scroll => "1m"
}
}
# a note in this section indicates that filter can be selected
filter {
}
output {
elasticsearch {
hosts => "47.xx.255.224:9200"
index => "toc_logstash"
}
}
然后的源机器上，运行命令：bin/logstash -f config/migration.conf，依次出现了以下几个错误：
- 存在运行中的实例
Sending Logstash logs to /usr/local/logstash-6.4.2/logs which is now configured via log4j2.properties
[2019-12-27T21:54:06,340][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml'
file because modules or command line options are specified
[2019-12-27T21:54:06,379][FATAL][logstash.runner ] Logstash could not be started
because there is already another instance using the configured data directory.
If you wish to run multiple instances, you must change the "path.data" setting.
[2019-12-27T21:54:06,390][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException:
Logstash stopped processing because of an error: (SystemExit) exit
解决办法：bin/logstash -f config/migration.conf --path.data ./logs/
上述命令，主窗口关闭程序就会结束，后台运行的版本是：
nohup bin/logstash -f config/migration.conf --path.data ./logs/ > sync.out 2>&1 &，日志输出到sync.out文件， 关于这个命令的nohup的几个参数的解释，参考这里
- 配置出错
[2019-12-27T21:56:06,229][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, } at line 3, column 23 (byte 51) after input {\n
ElasticSearch {\n hosts => 127.0", :backtrace=>["/usr/local/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:41:in
`compile_imperative'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'",
"/usr/local/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'",
"org/jruby/RubyArray.java:2486:in `map'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'",
"org/logstash/execution/AbstractPipelineExt.java:149:in `initialize'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:22:in
`initialize'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'",
"/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'",
"/usr/local/logstash-6.4.2/logstash-core/lib/logstash/agent.rb:309:in `block in converge_state'"]
原来是host配置要加引号，即"127.0.0.1:9200"
- 插件不可用
[2019-12-27T21:58:38,822][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed.
{:exception=>#<LoadError: no such file to load -- logstash/inputs/ElasticSearch>,
:path=>"logstash/inputs/ElasticSearch", :type=>"input", :name=>"ElasticSearch"}
[2019-12-27T21:58:38,849][ERROR][logstash.agent ] Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError",
:message=>"Couldn't find any input plugin named 'ElasticSearch'. Are you sure this is correct?
Trying to load the ElasticSearch input plugin resulted in this error: no such file to load
-- logstash/inputs/ElasticSearch", :backtrace=>["/usr/local/logstash-6.4.2/logstash-core/lib/logstash/plugins/registry.rb:211:in
`lookup_pipeline_plugin'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/plugin.rb:137:in `lookup'",
"org/logstash/plugins/PluginFactoryExt.java:210:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:166:in `plugin'",
"/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:71:in `plugin'", "(eval):8:in `<eval>'",
"org/jruby/RubyKernel.java:994:in `eval'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:49:in
`initialize'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'",
"/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'",
"/usr/local/logstash-6.4.2/logstash-core/lib/logstash/agent.rb:309:in `block in converge_state'"]}
ElasticSearch要改成上述的小写elasticsearch
最后运行成功，静候佳间，如果es中的数据比较大的话，比如好几百兆百把万条数据，还是要一些时间的，一秒钟大概500条。后续重点关注以下问题：
（1）重新运行命令，数据是否会重复导入？
（2）源es新增了数据，目标es是否会同步过去？
更多迁移方法可以看看，这里