elk

tony2099 included in micro

2021-04-30 273 words 2 minutes

Contents

what is elk

roles:

logstash:
1. ingest datas;
2. transform it
3. send it to a stash
beat: data collection
elasticsearch: store data;

logstash

1. collection data configure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ingest data
input{

}
# transform  data
filter{

}


# data go to where
output{
    
}

grok: 1.translate plain text to structure data; 2. add,remove field filter example

1
2
3
4
5


filter {
   grok {
    match => { "message" => ['%{TIMESTAMP_ISO8601:time} %{LOGLEVEL:logLevel} %{GREEDYDATA:logMessage}'] }
  }
}

2. output

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


output {
  elasticsearch {
    hosts => "172.16.7.31:10200"
    index => "logstash-nginx-ingress-logs-%{+YYYY.MM.dd}"
    

    manage_template =>  true,
    template => "/data/logstash-7.4.0/config/template.json"
    template_name => "template"
  }
 }

hosts: the stash host
index: index name
template
1. manage_template: true
  1. false: only use api to create template
  2. true: create a template on es on logstah’s startup
2. template: custom template path
3. template_name: how the template is named
4. template_overwrite: overwrite the template if one exists already under the same name;

beat and input config

consume a specific source

filebeat

consume file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


filebeat.inputs:
- input_type: log
  paths:
    - /var/log/httpd/access.log

document_type: apache-access
fields_under_root: true

output.logstash:
  hosts: ["127.0.0.1:5044"]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


input {
  beats {
    port => 5044
  }
 }

filter {
  grok {
    match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
 }

output {
  elasticsearch { hosts => ["localhost:9200"] }
  }