MENU

es+kibana+filebeat 日志收集

2021 年 06 月 14 日 • Java

elk日志收集

前言

最近搭建了elk平台 用于收集日志 发现网上的一些方案不能满足需求 翻了一下官方文章 总结了一套自己的配置 特此记录一下

lidop日志收集

一、部署方案

1.方案:

  • 组件:elasticsearch + filebeat + kibana
  • 版本:7.12.1

2.es集群部署:

机器节点名称http监听端口tcp传输端口服务器地址部署方式
b91yhow-node-192009301192.168.2.91zypper
b92yhow-node-292009302192.168.2.92zypper
b93yhow-node-392009303192.168.2.93zypper

3.kibana部署

机器http端口地址部署方式
b915601192.168.2.91zypper

4.filebeat部署

机器地址部署方式
lidop1-bj172.20.103.100yum
lidop2-bj172.20.103.99yum
lidop1-tky172.31.19.198yum
lidop2-tky172.31.25.158yum
lidop1-tw172.30.64.8yum
lidop2-tw172.30.64.9yum

二、es集群配置

配置中的xpack 见es认证说明

  • b91 /etc/elasticsearch/elasticsearch.yml
cluster.name: yhow
node.name: yhow-node-1

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 192.168.2.91
http.port: 9200
transport.tcp.port: 9301
discovery.seed_hosts: ["192.168.2.91:9301", "192.168.2.92:9302","192.168.2.93:9303"]
cluster.initial_master_nodes: ["yhow-node-1", "yhow-node-2","yhow-node-3"]

http.cors.enabled: true
http.cors.allow-origin: "*"

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
  • b91 /etc/elasticsearch/elasticsearch.yml
cluster.name: yhow
node.name: yhow-node-2

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 192.168.2.92
http.port: 9200
transport.tcp.port: 9302
discovery.seed_hosts: ["192.168.2.91:9301", "192.168.2.92:9302","192.168.2.93:9303"]
cluster.initial_master_nodes: ["yhow-node-1", "yhow-node-2","yhow-node-3"]

http.cors.enabled: true
http.cors.allow-origin: "*"

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
  • b93 /etc/elasticsearch/elasticsearch.yml
cluster.name: yhow
node.name: yhow-node-3

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 192.168.2.93
http.port: 9200
transport.tcp.port: 9303
discovery.seed_hosts: ["192.168.2.91:9301", "192.168.2.92:9302", "192.168.2.93:9303"]
cluster.initial_master_nodes: ["yhow-node-1", "yhow-node-2","yhow-node-3"]

http.cors.enabled: true
http.cors.allow-origin: "*"

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12



1.es 集群配置认证

es 7.0以后的插件xpack 有商业改为免费使用 因此采用xpack进行认证与授权

  • bin目录/usr/share/elasticsearch/bin
  1. 在es的bin目录执行./elasticsearch-certutil cert -out elastic-certificates.p12 -pass ""
  2. 将生成的证书放到所有节点的 ES_CONF下/etc/elasticsearch
  3. 在所有节点配置添加
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
  1. 生成es系统账户的账号和密码

es会将用户信息同步给所有节点

在bin目录执行./elasticsearch-setup-passwords auto

Changed password for user apm_system
PASSWORD apm_system = ***************

Changed password for user kibana_system
PASSWORD kibana_system = ***************

Changed password for user kibana
PASSWORD kibana = ***************

Changed password for user logstash_system
PASSWORD logstash_system = ***************

Changed password for user beats_system
PASSWORD beats_system = ***************

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = ***************

Changed password for user elastic
PASSWORD elastic = ***************

三、kibana配置

  • b91 /etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.2.91"
server.name: "yhow"

elasticsearch.hosts: ["http://192.168.2.91:9200","http://192.168.2.92:9200","http://192.168.2.93:9200"]
elasticsearch.username: "kibana_system"        #配置访问es的用户名 es默认提供
elasticsearch.password: "***************" #配置访问es的密码   es默认提供

i18n.locale: "en"                              #设置语言 目前版本中文不完善

四、filebeat配置

  • lidop全部机器
filebeat.config.inputs:
  enabled: true
  path: /etc/filebeat/inputs.d/*.yml  # 配置类似nginx的 config.d 方便管理
  
setup.ilm.enabled: false
setup.template.name: "whyhow"
setup.template.pattern: "whyhow-*"

output.elasticsearch:
  hosts: ["192.168.2.91:9200","192.168.2.92:9200","192.168.2.93:9200"]
  index: "%{[fields.index]:other}-%{+yyyy.MM.dd}"
  username: "filebeat_client"      #配置filebeat访问es的用户 es默认不提供 
  password: "***************" #配置filebeat访问es的密码 es默认不提供 
  • filebeat访问es所需权限:

    • Cluster privileges: all,monitor,manage
    • Privileges: auto_configure,create_index,manage,all
    • indices: 按需创建
  • filebeat采集日志配置 /etc/filebeat/inputs.d/*.yml

    - type: log
      enabled: true
      paths:
        - /var/log2/lidop/sdk/sdk.log #被采集的日志文件路径
      json.keys_under_root: true    
      json.overwrite_keys: true
      fields:
        index: 'whyhow-sdk'
      processors:
      - add_host_metadata: ~
      - script:                      
          lang: javascript
          id: creat_time_filter
          source: >                # 日志文件时间格式转换成
            function process(event) {
               var createTime = event.Get("createTime");
               var d = new Date(createTime);
               event.Put("tmp_time",d);
            }
      - decode_json_fields:
          fields: ["message"]
          target: ""
      - timestamp:
          field: tmp_time
          timezone: Asia/Shanghai
          layouts:
            - '2006-01-02 15:04:05'
          test:
            - '2019-06-22 16:33:51'
      - drop_fields:
          fields: ["log","input","agent","ecs","cloud","tmp_time"]
          ignore_missing: false


       本站的资源及素材小部分来自互联网,内容若侵犯到了您的权益,请联系我删除,下载资源仅供交流学习所用,请勿用于商业用途,并在下载后的 24 小时内删除;本站所有未注明转载的文章均为原创,并采用 CC BY-NV-SA 4.0 授权协议,转载请注明来源,谢谢!