Integration between Filebeat and logstash
5 likes1,836 views
Filebeat sends log files to Logstash. There are several cases described for integrating Filebeat and Logstash: 1) A simple configuration where one log file is sent from Filebeat to Logstash and output to one file. 2) Another simple configuration where multiple log files are sent from Filebeat to Logstash using a wildcard, and output to one file. 3) An advanced configuration where multiple log files are sent from Filebeat to Logstash, and Logstash outputs each file to a separate file based on the original file name using filtering. 4) A more advanced configuration where log files are sent from Filebeat to Logstash, Logstash parses the timestamp and uses it as the output
![Common Config : Filebeat
filebeat.prospectors:
- type: log
enabled: true
paths:
- /data/logs/reallog/2018-12-27.log
output.logstash:
hosts: ["target.aggserver.com:5044"]](https://image.slidesharecdn.com/filebeatlogstash-181227083608/85/Integration-between-Filebeat-and-logstash-3-320.jpg)
![Case #4 : Advance, multiple files to multiple
files : with log timestamp
filter {
grok {
patterns_dir => ["/usr/local/logstash-5.4.1/patterns"]
match => { "message" => "^%{TIMESTAMP_ISO8601:timestamp}" }
}
date {
match => ["timestamp", "yyyy-MM-dd"]
}
}
output {
file {
path => "/data/logstash/%{+YYYY-MM-dd}.log"
codec => line { format => "%{message}" }
}
}
Filtering timestamp and using it as filename.](https://image.slidesharecdn.com/filebeatlogstash-181227083608/85/Integration-between-Filebeat-and-logstash-9-320.jpg)
![Case #4 : Advance, multiple files to multiple
files : with log timestamp
filter {
……
date {
match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss-08:00"]
Timezone => "UTC"
}
}
Parsing timezone part as string, and set other parts
as UTC](https://image.slidesharecdn.com/filebeatlogstash-181227083608/85/Integration-between-Filebeat-and-logstash-12-320.jpg)
Integration between Filebeat and logstash
- 1. Integration between Logstash and Filebeat charsyam@naver.com
- 2. Integration between Logstash and Filebeat Filebeat Logstash Filebeat sends logs to logstash.
- 3. Common Config : Filebeat filebeat.prospectors: - type: log enabled: true paths: - /data/logs/reallog/2018-12-27.log output.logstash: hosts: ["target.aggserver.com:5044"]
- 4. Common Config : Logstash input { beats { port => 5044 } } output { file { path => "/data/logstash/2018-12-27.log" codec => line { format => "%{message}" } } }
- 5. Case #1 : Simple, one file to one file Just use common config
- 6. Case #1 : Simple, one file to one file But we don’t need this case
- 7. Case #2 : Simple, multiple files to one file filebeat.prospectors: - type: log enabled: true paths: - /data/logs/reallog/*.log Just use *.
- 8. Case #3 : Advance, multiple files to multiple files : Just move content by each file filter { grok { match => {"source" => "data/logs/%{DATA:logdate}.log"} } } output { file { path => "/data/logstash/%{logdate}.log" codec => line { format => "%{message}" } } } Filebeat sends original filename with source field
- 9. Case #4 : Advance, multiple files to multiple files : with log timestamp filter { grok { patterns_dir => ["/usr/local/logstash-5.4.1/patterns"] match => { "message" => "^%{TIMESTAMP_ISO8601:timestamp}" } } date { match => ["timestamp", "yyyy-MM-dd"] } } output { file { path => "/data/logstash/%{+YYYY-MM-dd}.log" codec => line { format => "%{message}" } } } Filtering timestamp and using it as filename.
- 10. Case #4 : Advance, multiple files to multiple files : with log timestamp Logstash Parsing timestamp as UTC, so If your log format is like below and your timezone is UTC -8(PST), 2018-12-26T23:00:00-08:00, it will be handled by 2018-12-27 not 2018-12-26, because logstash uses UTC as timestamp.
- 11. Case #4 : Advance, multiple files to multiple files : with log timestamp How to fix?
- 12. Case #4 : Advance, multiple files to multiple files : with log timestamp filter { …… date { match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss-08:00"] Timezone => "UTC" } } Parsing timezone part as string, and set other parts as UTC