Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
3 likes2,975 views
The document outlines the configuration and setup of a central log storage system using the ELK stack (Elasticsearch, Logstash, Kibana) along with Filebeat and RabbitMQ. It includes various configuration files, SSL settings, and troubleshooting steps for monitoring and managing log data. Additionally, it discusses Logstash filters for filtering log messages based on status and user authentication roles.
![FileBeat config file. YML standard
FileBeat config file. YML
4
vim /etc/filebeat/filebeat.yml
-
paths:
- ${API_APP_LOG_PATH}/file.log
encoding: plain
input_type: log
fields:
level: apico_backend
review: 1
fields_under_root: false
ignore_older: 0
document_type: api_backend_requests
scan_frequency: 2s
multiline:
pattern: ^[[:digit:]]{4}
negate: true
match: after
max_lines: 500
timeout: 2s
tail_files: false
### Logstash as output
logstash:
# The Logstash hosts
hosts: ["cls.apico.net:9999"]
# Number of workers per Logstash host.
worker: 2
# Optional TLS. By default is off.
tls: certificate_authorities:["/etc/ssl/certs/rootCA.crt"]
insecure: false](https://image.slidesharecdn.com/centrallogstorage-160430164049/85/Central-LogFile-Storage-ELK-stack-Elasticsearch-Logstash-and-Kibana-4-320.jpg)
![SSL Certificate Authority
Certificate Authority — в 5 OpenSSL команд
https://habrahabr.ru/post/192446/
Using TLS between Beats and Logstash
https://gist.github.com/andrewkroh/fdc7e5f3f0f0ed63a11c
Validate crt key:
[root@MSG-predprod lostash_pp]# pwd
/etc/ssl/certs/lostash_pp
[root@MSG-predprod lostash_pp]# curl -v --cacert rootCA.crt https://cls.apico.net:9999
* Rebuilt URL to: https://cls.apico.net:9999/
* Connected to cls.apico.net (172.31.13.49) port 9999 (#0)
* CAfile: rootCA.crt
* Server certificate:
* subject: E=kh.airat14@gmail.com,CN=cls.apico.net,OU=ITY,O=Default Company Ltd,L=Moscow,ST=Moscow,C=MT
* start date: Apr 19 11:59:50 2016 GMT
* expire date: Sep 05 11:59:50 2043 GMT
* common name: cls.apico.net
> GET / HTTP/1.1
> Host: cls.apico.net:9999
> Accept: */*
* Empty reply from server
[root@MSG-predprod lostash_pp]#
6
Security:SSL Certificate AUthority.](https://image.slidesharecdn.com/centrallogstorage-160430164049/85/Central-LogFile-Storage-ELK-stack-Elasticsearch-Logstash-and-Kibana-6-320.jpg)
![LogStash Filter config file.
LogStash Filters. Filter for “message_sid”.
Filter SmsController RadiusController
SmsSender by message_sid:
13
Source Consoler_app: [MessageSid:msgXXX] SmsSender_app:"message_sid":"msgXXX"
message_sid msgXXX msgXXX](https://image.slidesharecdn.com/centrallogstorage-160430164049/85/Central-LogFile-Storage-ELK-stack-Elasticsearch-Logstash-and-Kibana-13-320.jpg)
![LogStash Filter config file.
LogStash Filters. Filter for “message_sid”.
Filter SmsController RadiusController
SmsSender by message_sid:
14
Source Consoler_app: [MessageSid:msgXXX] SmsSender_app:"message_sid":"msgXXX"
message_sid msgXXX msgXXX](https://image.slidesharecdn.com/centrallogstorage-160430164049/85/Central-LogFile-Storage-ELK-stack-Elasticsearch-Logstash-and-Kibana-14-320.jpg)
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
- 1. Central Log Storage. ELK stack(a.k.a., Elasticsearch, Logstash, and Kibana) 1
- 2. List of Log Files. Central Log Storage. 2
- 3. Central Log Storage. Diagram. 3
- 4. FileBeat config file. YML standard FileBeat config file. YML 4 vim /etc/filebeat/filebeat.yml - paths: - ${API_APP_LOG_PATH}/file.log encoding: plain input_type: log fields: level: apico_backend review: 1 fields_under_root: false ignore_older: 0 document_type: api_backend_requests scan_frequency: 2s multiline: pattern: ^[[:digit:]]{4} negate: true match: after max_lines: 500 timeout: 2s tail_files: false ### Logstash as output logstash: # The Logstash hosts hosts: ["cls.apico.net:9999"] # Number of workers per Logstash host. worker: 2 # Optional TLS. By default is off. tls: certificate_authorities:["/etc/ssl/certs/rootCA.crt"] insecure: false
- 5. Using Environment Variblaes in Configuration Logstash config source Link environment-variables @see /etc/sysconfig/filebeat Filebeat_using_env_vars Environment Logstash config result input { tcp { port => "${TCP_PORT}" } } export TCP_PORT=12345 input { tcp { port => 12345 } } input { tcp { port => "${TCP_PORT}" } } No TCP_PORT defined Raise a logstash configuration error input { tcp { port => "${TCP_PORT:54321}" } } No TCP_PORT defined input { tcp { port => 54321 } } filter { mutate { add_field => { "my_path" => "${HOME}/file.log" } } export HOME="/path" filter { mutate { add_field => { "my_path" => "/path/file.log" } } 5
- 6. SSL Certificate Authority Certificate Authority — в 5 OpenSSL команд https://habrahabr.ru/post/192446/ Using TLS between Beats and Logstash https://gist.github.com/andrewkroh/fdc7e5f3f0f0ed63a11c Validate crt key: [root@MSG-predprod lostash_pp]# pwd /etc/ssl/certs/lostash_pp [root@MSG-predprod lostash_pp]# curl -v --cacert rootCA.crt https://cls.apico.net:9999 * Rebuilt URL to: https://cls.apico.net:9999/ * Connected to cls.apico.net (172.31.13.49) port 9999 (#0) * CAfile: rootCA.crt * Server certificate: * subject: E=kh.airat14@gmail.com,CN=cls.apico.net,OU=ITY,O=Default Company Ltd,L=Moscow,ST=Moscow,C=MT * start date: Apr 19 11:59:50 2016 GMT * expire date: Sep 05 11:59:50 2043 GMT * common name: cls.apico.net > GET / HTTP/1.1 > Host: cls.apico.net:9999 > Accept: */* * Empty reply from server [root@MSG-predprod lostash_pp]# 6 Security:SSL Certificate AUthority.
- 7. Beats Platform: Collect, Parse, and Ship Don't stop the Beat Packetbeat - the best way to understand the value of a network packet analytics system like Packetbeat is to try it on your own traffic. Topbeat - helps you monitor the availability of your website or service by providing system-wide and per-process statistics along with a disk usage overview. Filebeat - read data from log files 7
- 8. LogStash Input config file. LogStash Input config file. 8 vim /etc/logstash/conf.d/logstash.conf input { beats { #plugins port => 9999 host => "cls.apico.net" ssl_certificate => "/etc/ssl/certs/rootCA.crt" ssl_key => "/etc/pki/tls/private/rootCA.key" } rabbitmq { exchange => "event-sms-in" queue => "logstash-queue-pp" } } filter { ... } output { file{ path => "/logstash/%{+YYYY-MM-dd}-input.log" } rabbitmq { exchange => "logstash-exchange" key => "logstash-routing-key" } }
- 9. Get data from Rabbit. Get data from RabbitMq (Logstash plugin) https://www.elastic.co/guide/en/logstash/current/plugins-inputs-rabbitmq.html 9 Get data from RabbitMq.
- 10. RabbitMq configuration. Output RabbitMq configuration. 10 Hostname IP cls.apico.net 127.0.0.1 Exchange logstash-exchange Exchange_type direct Routing Key logstash-routing-key User user
- 11. LogStash Filter config file. LogStash Filters. Filter for “log_status”. WIKI: Добавление параметра message_sid в файла логирования для компонентов SmsController RadiusController. Filter messages from log file by STATUS: 11 log file message status error warning info log_status error warning info
- 12. LogStash Filter config file. LogStash Filters. Filter for “account_sid”. Filter API request by User Auth Status/Role: 12 user auth status/role User unauthorized User:: ROLE_SYSTEM User::ROLE_ACCOUNT account_sid unknown system acc01fe181e-741b-3693-88bb-3847abfb6e86 sac01fe181e-741b-3693-88bb-3847abfb6e55
- 13. LogStash Filter config file. LogStash Filters. Filter for “message_sid”. Filter SmsController RadiusController SmsSender by message_sid: 13 Source Consoler_app: [MessageSid:msgXXX] SmsSender_app:"message_sid":"msgXXX" message_sid msgXXX msgXXX
- 14. LogStash Filter config file. LogStash Filters. Filter for “message_sid”. Filter SmsController RadiusController SmsSender by message_sid: 14 Source Consoler_app: [MessageSid:msgXXX] SmsSender_app:"message_sid":"msgXXX" message_sid msgXXX msgXXX
- 15. LogStash List og plugins.. LogStash Plugins. 15 logstash-codec logstash-filter logstash-input logstash-output collectd dots edn edn_lines es_bulk fluent graphite json json_lines line msgpack multiline netflow oldlogstashjson plain rubydebug anonymize checksum clone csv date dns drop Fingerprint geoip grok json Kv metrics multiline mutate ruby sleep split syslog_pri throttle urldecode useragent uuid xm beats couchdb_changes elasticsearch eventlog exec file ganglia gelf generator graphite heartbeat http http_poller Imap irc Jdbc kafka log4j lumberjack pipe rabbitmq redis s3 snmptrap sqs stdin syslog tcp twitter udp unix xmpp zeromq cloudwatch csv elasticsearch email exec file ganglia gelf graphite hipchat http irc juggernaut kafka lumberjack nagios nagios_nsca null opentsdb pagerduty pipe rabbitmq redis s3 sns sqs statsd stdout tcp udp xmpp zeromq logstash-patterns-core
- 16. Nginx Kibana: Authentication. Nginx-to-proxy Nginx Restricting Access Nginx Reverse Proxy 16 Nginx Kibana: Authentication. Nginx-to-proxy
- 17. Kibana. Discover. Search. ElasticSearch Query. 17 Kibana: The Main Components. Discover.
- 18. Kibana. Visualize. Visualization type. 18 Kibana: The Main Components. Visualize.
- 19. Kibana. Dashboard. EL Query + Visualization types 19 Kibana: The Main Components.
- 20. ElasticSearch Query DSL. ES Query DSL ES Query DSL (Webinar Video) 20 ElasticSearch Query DSL
- 21. Debug. Troubleshooting. Filebeat: 1. filebeat -e -d "publish" -c /etc/filebeat/filebeat.yml -configtest 2. .(точка)(пробел) /etc/sysconfig/filebeat 3. filebeat -e -d "*" -c /etc/filebeat/filebeat.yml Logstash: 1. /opt/logstash/bin/logstash --config /etc/logstash/conf.d/logstash.conf --configtest 2. /opt/logstash/bin/logstash --verbose --config /etc/logstash/conf.d/logstash.conf 21 Debug. Troubleshooting.
- 22. Inspiration manual. 1.Собираем и анализируем логи с помощью Lumberjack+Logstash+Elasticsearch+RabbitMQ (The Guardian ) 2. Wiki MTT.Innovations. АPICO.CentralLogStorage. 3. Import to CvS Excel . Read Comments https://habrahabr.ru/post/236341/ 4. Logstash alert. The throttle filter is for throttling the number of events received. https://www.elastic.co/guide/en/logstash/current/plugins-filters-throttle.html Plugins outputs email https://www.elastic.co/guide/en/logstash/current/plugins-outputs-email.html 22 Inspiration manual.
- 23. End. No just the beginning. Elastic BackUp: Snapshot and restore. Backing-up-your-cluster File rotation linux BackUp & Restore(5). Elasticsearch Monitoring Java 23 Just the beginning.