Elasticsearch

Overview

Install browser plugin

/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head

Dump database

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{ "type": "fs", "settings": { "location": "/mount/backups/my_backup", "compress": true }}'

Configure cluster

  • Edit /etc/elasticsearch/elasticsearch.yml

  • Set cluster.name, node.name, network.host and discovery.zen.ping.unicast.hosts

  • Make sure TCP port 9300 is open

Cluster autodiscovery is not working

  • Edit /etc/elasticsearch/elasticsearch.yml

  • Make one node the master node

node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1.example.com"]
  • Let the others connect to the master node

node.master: false
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1.example.com"]

Add authentication to Elasticsearch

  • Does not work in cluster mode

bin/plugin install license
bin/plugin install shield
  • Edit elasticsearch.yml

action.auto_create_index: .security
  • Restart elasticsearch and add a user

bin/shield/esusers useradd admin -r admin

Insert data manually

curl -XPUT 'http://localhost:9200/dept/employee/1' -d '{ "empname": "emp1"}'

Configure Rsyslog to log to Logstash

  • Create file /etc/rsyslog.d/logstash.conf

*.* @127.0.0.1:5544

Configure Rsyslog to log to Fluentd

  • Create file /etc/rsyslog.d/fluentd.conf

*.* @127.0.0.1:42185

Configure Rsyslog to log directly to Elasticsearch

yum install rsyslog-elasticsearch
  • Now edit /etc/rsyslog.conf

module(load="imuxsock")             # for listening to /dev/log
module(load="omelasticsearch") # for outputting to Elasticsearch
# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
  type="list") {
    constant(value="logstash-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="on")
      constant(value="\",\"message\":\"")    property(name="msg" format="on")
    constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="logstash-index"
    dynSearchIndex="on")

Let Elasticsearch listen only on loopback

  • Edit /etc/elasticsearch/elasticsearch.yml

network.host: 127.0.0.1

Use logstash as log aggregator

  • Create /etc/logstash/conf.d/10-syslog.conf

input {
  syslog {
    type => syslog
    port => 5544
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
  • Create /etc/logstash/conf.d/30-elasticsearch-output.conf

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

Use fluentd as log aggregator

  • Can collecd and parse log from many sources (200+)

  • Is written in Ruby and needs no Java like Logstash

  • Can output to many directions including files, mongodb and of course elasticsearch

  • For installation see http://docs.fluentd.org/categories/installation

  • Install Elasticsearch plugin

gem install fluent-plugin-elasticsearch
  • If your ruby version is too old or buggy install fluentd inside rvm

curl -sSL https://get.rvm.io | bash -s stable --ruby
source /usr/local/rvm/scripts/rvm
gem install fluentd
gem install fluent-plugin-elasticsearch
 # live debugging agent
 #<source>
 #  type debug_agent
 #  bind 127.0.0.1
 #  port 24230
 #</source>

 # Listen to Syslog
 <source>
   type syslog
   port 42185
   tag system.raw
 </source>

 # Apache Access Logs
 <source>
   type tail
   format apache2
   path /var/log/httpd/access_log
   pos_file /var/log/fluentd/httpd.access.pos
   tag httpd.access
 </source>

 # Apache Error Logs
 <source>
   type tail
   format apache_error
   path /var/log/httpd/error_log
   pos_file /var/log/fluentd/httpd.error.pos
   tag httpd.error
 </source>

 # Tag kernel messages
 <match system.raw.**>
   type rewrite_tag_filter
   rewriterule1 ident ^kernel$  kernel.raw # kernel events
   rewriterule2 ident .* system.unmatched     # let all else through
 </match>

 # Identify iptables messages
 <match kernel.raw.**>
   type rewrite_tag_filter
   rewriterule1 message ^IN=.* OUT=.+$ iptables.raw  # iptables events
   rewriterule2 message .* kernel.unmatched      # let all else through
</match>

 # Parse iptables messages
 # IN=eno1 OUT= MAC=aa:bb:cc:aa:bb:cc:aa:bb:cc:aa:bb:cc:aa:00 SRC=192.168.10.42 DST=192.168.10.23 LEN=148 TOS=0x00 PREC=0x00 TTL=255 ID=53270 DF PROTO=UDP SPT=5353 DPT=5353 LEN=128
 <match iptables.raw.**>
   type parser
   key_name message # this is the field to be parsed!
   format /^IN=(?<iface>.*) OUT=(?<oface>.*) MAC=(?<mac>.*?) (SRC=(?<srcip>.*))? (DST=(?<dstip>.*))? LEN=(?<pkglen>.+) TOS=(?<pkgtos>.+) PREC=(?<pkgrec>.+) TTL=(?<pkgttl>.+) ID=(?<ipid>.+) \w{0,2}\s?PROTO=(?<pkgproto>.+)( SPT=(?<srcport>.+) DPT=(?<dstport>.+) LEN=(.*))?$/
   time_format %b %d %H:%M:%S
   tag iptables.parsed
 </match>

 # write to file
 #<match iptables.parsed>
 #  type file
 #  path /var/log/td-agent/iptables.log
 #</match>

 # Write to elasticsearch
 <match *.**>
     type elasticsearch
     host localhost
     port 9200
     include_tag_key true
     tag_key _key
     logstash_format true
     flush_interval 10s
 </match>

 # Log to stdout for debugging
 #<match *.**>
 #    type stdout
 #</match>
  • Last but not least configure your systlog to send messages to fluentd

*.* @127.0.0.1:42185
  • Start fluentd in foreground for testing purpose

fluentd -c /etc/fluent/fluent.conf -vv

Kibana Web Frontend

[Service]
ExecStart=/opt/kibana4/bin/kibana
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=kibana4
User=root
Group=root
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target