elasticsearch安装

December 19, 2019
es安装 jdk11 elasticsearch kibana 日志管理 head

1 引言

es,不必多说,基本上软件标配,即便不用其全文检索功能,作为ELK的组合之一也是常见,见此文:elk日志管理

最新的发行版是7.5.1,依赖jre 11+,本文简介安装过程,系统是ubuntu 18.04

运行端口9200/9300,常用命令:
cd /usr/local/elasticsearch-7.5.1
su elk
bin/elasticsearch -d

2 下载两个安装包

3 安装jdk

解压即可,设置环境变量:vim /etc/profile,新增配置如下:

export JAVA_HOME=/usr/local/jdk-11.0.5
export PATH=$JAVA_HOME/bin:$PATH

使配置生效:source /etc/profile

4 安装es

解压即可,尝试运行:bin/elasticsearch,报错如下:

Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-12-19T14:15:37,086][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [iZ8vbj9qj1ctpijae0bxunZ] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
  1. es启动脚本里带了一些jdk不再支持的启动参数

  2. 不能用root用户去启动es

4.1 配置es用户

ubuntu下新增用户:

groupadd elk
useradd elk -g elk -p elk # 在组elk下新增用户elk,其密码为elk
chown -R elk.elk elasticsearch-7.5.1 # 更换所有者,这样才有启动权限

4.2 启动

启动之前,看是否有需要进行一些配置vim elasticsearch-7.5.1/config/elasticsearch.yml,比如绑定的IP,是否有跨域访问的需求等,注意以下配置:

network.host: 172.26.71.241 # 绑定的IP,默认是127.0.0.1

# 跨域访问,如es head等
http.cors.enabled: true
http.cors.allow-origin: "*"

更多配置也可以修改:

# Path to directory where to store the data (separate multiple locations by comma):
path.data: /usr/local/elasticsearch-7.5.1/data

# Path to log files:
path.logs: /usr/local/elasticsearch-7.5.1/logs

cd elasticsearch-7.5.1/

su elk # 切换用户

bin/elasticsearch # 启动es,加参数-d表示守护运行

4.3 一些操作

查看索引:curl 'yidu.seaeverit.com:9200/_cat/indices?v',返回:

health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana               6GJjyPeZQ5Cko4G1YuTEqg   1   1          5            0     21.5kb         21.5kb
yellow open   nightlionsecurity.com N40sp-0RR9-HAaL0j8LjCA   1   1          0            0       283b           283b
yellow open   toc_logstash          hDxdQ9v3SWGU5DrWrp5dEw   1   1      89889            0     34.5mb         34.5mb

4.4 一些问题

[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

解决办法:在*/etc/sysctl.conf*文件最后添加一行:vm.max_map_count=262144,重启服务器(不重启可以试试:sysctl -p) ,查看命令:sysctl -a|grep vm.max_map_count,更多相关问题可以看这里

[1]: the default discovery settings are unsuitable for production use; at least one of 
[discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

打开一个,修改配置:discovery.seed_hosts: ["172.26.71.241"]

master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, 
and [cluster.initial_master_nodes] is empty on this node: have discovered [{iZ8vbj9qj1ctpijae0bxunZ}{tn8yAs33Qa6ZRf1zPp5ing}{dVK5gLrGTjSSc_SYL6o8CA}
{172.26.71.241}{172.26.71.241:9300}{dilm}{ml.machine_memory=32655335424, xpack.installed=true, ml.max_open_jobs=20}]; 
discovery will continue using [] from hosts providers and [{iZ8vbj9qj1ctpijae0bxunZ}{tn8yAs33Qa6ZRf1zPp5ing}{dVK5gLrGTjSSc_SYL6o8CA}
{172.26.71.241}{172.26.71.241:9300}{dilm}{ml.machine_memory=32655335424, xpack.installed=true, ml.max_open_jobs=20}] 
from last-known cluster state; node term 0, last-accepted version 0 in term 0

打开配置:cluster.initial_master_nodes: ["172.26.71.241"]

4.5 基本信息

es的基本信息查看,curl http://172.26.71.241:9200,如下:

{
  "name" : "iZ8vbj9qj1ctpijae0bxunZ",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "CFqItLYzTL2PNQnu-ll9rQ",
  "version" : {
    "number" : "7.5.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
    "build_date" : "2019-12-16T22:57:37.835892Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

坑:

(1)退出只读模式

curl -X PUT 'xx.xxxx.com:9200/xxxx/_settings' -H 'Content-Type: application/json' 
-d '{"index.blocks.read_only_allow_delete": null}'

返回{"acknowledged":true}

(2)重配:

curl -X PUT  'xxx.xxxxx.com:9200/_cluster/settings' -H 'Content-Type: application/json' -d 
'{"transient":{"cluster.routing.allocation.disk.watermark.low":"100gb","cluster.routing.allocation.disk.watermark.high":"50gb","cluster.routing.allocation.disk.watermark.flood_stage":"10gb","cluster.info.update.interval":"1m"}}'

返回

{
    "acknowledged": true,
    "persistent": {},
    "transient": {
        "cluster": {
            "routing": {
                "allocation": {
                    "disk": {
                        "watermark": {
                            "low": "100gb",
                            "flood_stage": "10gb",
                            "high": "50gb"
                        }
                    }
                }
            },
            "info": {
                "update": {
                    "interval": "1m"
                }
            }
        }
    }
}

4.6 head插件安装

项目开源地址在这里。克隆代码: git clone git://github.com/mobz/elasticsearch-head.git

环境nodjs v8.10.0,npm的版本是3.5.2,进入目录,运行npm install,还提示安装最新的nodejs和npm,出错:

Failed at the phantomjs-prebuilt@2.1.16 install script

安装老版本,npm install phantomjs-prebuilt@2.1.14 --ignore-scripts,过程比较漫长:

npm WARN deprecated phantomjs-prebuilt@2.1.14: this package is now deprecated
npm WARN deprecated hawk@3.1.3: This module moved to @hapi/hawk. 
Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.

npm install,OK

后台运行:nohup npm start &

4.7 kibana安装

kibana来查看日志,核心传输是logstash,另外有介绍,见此文:es_logstash;核心存储使用的是es。 kibana的中文手删链接在这里

下载页在[这里]()https://www.elastic.co/cn/downloads/kibana),下载的最新版:[7.5.1](https://artifacts.elastic.co/downloads/kibana/kibana-7.5.1-linux-x86_64.tar.gz)

解压,修改基本配置:vim config/kibana.yml,比如:

server.host: "172.26.71.241"
server.basePath: "/kibana"
elasticsearch.username: "kibana"
elasticsearch.password: "pass"

后台运行:nohup bin/kibana --allow-root --no-sandbox & # allow-root参数是允许root运行,默认不允许的

用elk运行,报错:

The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports.

4.8 es数据如何迁移

下面介绍的是通过logstash迁移数据,参考自这里 ,这个还是阿里云的英文版的技术介绍文档。

比如,新建下列logstash脚本:

input {
    ElasticSearch {
        hosts => ["********your host**********"]
        user => "*******"
        password => "*********"
        index => "logstash-2017.11.07"
        size => 1000
        scroll => "1m"
    }
}
# a note in this section indicates that filter can be selected
filter {
}
output {
    ElasticSearch {
        hosts => ["***********your host**************"]
        user => "********"
        password => "**********"
        index => "logstash-2017.11.07"
    }
}

我相应修改的版本是:

input {
    elasticsearch {
        hosts => "127.0.0.1:9200"
        index => "toc_logstash"
        size => 1000
        scroll => "1m"
    }
}
# a note in this section indicates that filter can be selected
filter {
}
output {
    elasticsearch {
        hosts => "47.xx.255.224:9200"
        index => "toc_logstash"
    }
}

然后的源机器上,运行命令:bin/logstash -f config/migration.conf,依次出现了以下几个错误:

Sending Logstash logs to /usr/local/logstash-6.4.2/logs which is now configured via log4j2.properties
[2019-12-27T21:54:06,340][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' 
file because modules or command line options are specified
[2019-12-27T21:54:06,379][FATAL][logstash.runner          ] Logstash could not be started 
because there is already another instance using the configured data directory.  
If you wish to run multiple instances, you must change the "path.data" setting.
[2019-12-27T21:54:06,390][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: 
Logstash stopped processing because of an error: (SystemExit) exit

解决办法:bin/logstash -f config/migration.conf --path.data ./logs/

上述命令,主窗口关闭程序就会结束,后台运行的版本是:nohup bin/logstash -f config/migration.conf --path.data ./logs/ > sync.out 2>&1 &,日志输出到sync.out文件, 关于这个命令的nohup的几个参数的解释,参考这里


[2019-12-27T21:56:06,229][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, 
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, } at line 3, column 23 (byte 51) after input {\n   
 ElasticSearch {\n        hosts => 127.0", :backtrace=>["/usr/local/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:41:in
  `compile_imperative'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", 
  "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", 
  "org/jruby/RubyArray.java:2486:in `map'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", 
  "org/logstash/execution/AbstractPipelineExt.java:149:in `initialize'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:22:in 
  `initialize'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'",
   "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", 
   "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/agent.rb:309:in `block in converge_state'"]

原来是host配置要加引号,即”127.0.0.1:9200”

[2019-12-27T21:58:38,822][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. 
{:exception=>#<LoadError: no such file to load -- logstash/inputs/ElasticSearch>,
 :path=>"logstash/inputs/ElasticSearch", :type=>"input", :name=>"ElasticSearch"}
[2019-12-27T21:58:38,849][ERROR][logstash.agent           ] Failed to execute action 
{:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", 
:message=>"Couldn't find any input plugin named 'ElasticSearch'. Are you sure this is correct?
 Trying to load the ElasticSearch input plugin resulted in this error: no such file to load 
 -- logstash/inputs/ElasticSearch", :backtrace=>["/usr/local/logstash-6.4.2/logstash-core/lib/logstash/plugins/registry.rb:211:in 
 `lookup_pipeline_plugin'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/plugin.rb:137:in `lookup'", 
 "org/logstash/plugins/PluginFactoryExt.java:210:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:166:in `plugin'", 
 "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:71:in `plugin'", "(eval):8:in `<eval>'", 
 "org/jruby/RubyKernel.java:994:in `eval'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:49:in 
 `initialize'", "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", 
 "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'",
  "/usr/local/logstash-6.4.2/logstash-core/lib/logstash/agent.rb:309:in `block in converge_state'"]}

ElasticSearch要改成上述的小写elasticsearch

最后运行成功,静候佳间,如果es中的数据比较大的话,比如好几百兆百把万条数据,还是要一些时间的,一秒钟大概500条。后续重点关注以下问题:

(1)重新运行命令,数据是否会重复导入?

(2)源es新增了数据,目标es是否会同步过去?

更多迁移方法可以看看,这里


loading