Graylog for open stack 3 steps to know why
0 likes1,391 views
1. Graylog is log management software that can be used to collect and analyze logs from OpenStack. It provides centralized log management. 2. With hundreds of log files generating tens of gigabytes of logs daily across OpenStack controllers and compute nodes, effective log management is needed. Graylog can collect logs from all OpenStack services and components. 3. Using Graylog for OpenStack involves 3 simple steps - collecting logs from OpenStack services, analyzing the logs in Graylog, and using log information to understand and resolve issues. This allows incidents to be responded to quickly by finding the root cause in the logs.
Graylog for open stack 3 steps to know why
- 1. Graylog for OpenStack : 3 steps to know WHY
- 2. MediTech JSC https://meditech.vn Private Cloud Storage Monitor Logging Managed Services About me Dinh Van Manh ● System Integration Department in MediTechJSC ● Member of Hocchudong ● Interested in OpenStack, Linux, Monitoring, Logging and new technology ● Habbit : “tra da + thuoc lao” with friends
- 3. Agenda 1. Log Overview 1.1. Logs : What & Where? 1.2. Why look at Logs 1.3. How to use Logs effectively 2. Log in OpenStack 2.1. OpenStack log statistics 2.2. OpenStack Log Management : in imagionation & in fact 3. Graylog for OpenStack 3.1. Introduce about Graylog 3.2. Key features 3.3. Architecture/Mechanism/Model of Graylog 3.4. Graylog for OpenStack: 3 steps to know WHY? 4. Demo + Q.A
- 4. Log Overview What? Where? Why? How?
- 5. 1.1. Logs : What & Where What logs? (from the view of system administrator) ● System event diary ● System status records ● User activities ● Incident notify Log format
- 6. 1.1. Logs : What & Where Log come from WHERE? ● Storage devices ● Application in Linux/Windows ● Cloud Services : OpenStack ● Servers ● Firewalls ● Routers, switches
- 7. 1.2. Why look at Logs? Basically : Incident response higher Tracking system event higher Measuring security : metrics, trends… higher and higher Situational awareness New threat discovery Estimating about user habit, trends...
- 8. 1.3. How to use Logs effectively Level 1 : Just SSH and view ! ● Understanding log location ● Command to view log : tail, more, grep ● Filtering by keyword Level 2 : Use Syslog ● Collect syslog from client ● Store in log server Level 3 : Log management Software ● Collect everything ● Retain most everything ● Analyze enough ● Summarize and report ● Advance features : visualize, alert, share...
- 9. 1.3. How to use Logs effectively ● Facility ○ Application Logs ○ Event Logs ○ Service Logs ○ System Logs Log Keywords ● Severity ○ 0 - emerg ○ 1 - alert ○ 2 - crit ○ 3 - error ○ 4 - warn ○ 5 - notice ○ 6 - info ○ 7 - debug ● Rotention ○ Time to rotate log ● Retention ○ Delete, archive...log ● Syslog ○ protocol to transfer log
- 10. Log in OpenStack Which level is appropriate?
- 11. 2.1. OpenStack log statistics OpenStack System : 3 Controller + 30 Compute node ● Controller Node ○ 6 log folder per OpenStack service ○ system log : auth, dmesg, kernel… ○ application log : apache, haproxy, pacemaker… ● Compute Node ○ 2 log folder per OpenStack service ○ system log : auth, dmes, kernel… ○ application log : libvirt ○ log of instances => Total : ● ~ 220 log file ● 10 GB log = 30 million messages / day
- 12. 2.2. OpenStack log management : in imagionation & in fact Communication think Colleagues think In fact When i said : My job is OpenStack log management ! So Waste !!! What should we do?
- 13. Graylog for OpenStack: To infinity & beyond !
- 14. 3.1. Graylog Introduce ● Log centralized management software ● Released in 2010 by Lenart Koopman with name is Graylog2 ● In 1/2015 release Graylog v1., Graylog Inc was established ● Big change from Graylog version 2.0 ● Newest version is Graylog 2.3.1, stable version is Graylog 2.3.0
- 15. 3.2. Key features Various Input & Output Analyze & Search Visualize metricAlert & Trigger User management
- 16. 3.3. Architecture/Mechanism/Model of Graylog Overall architecture ● Server ○ Graylog ● Client ○ Client host ○ Graylog sidecar ○ Nxlog/Filebeat Filebeat Graylog Sidecar : Break the old path ● Configuration management system ● Config in client host only ONCE ! ● All in Web ● Secure with SSL/TLS
- 17. 3.3. Architecture/Mechanism/Model of Graylog Sidecar Work-flow : Easy config in 3 steps Step 1 : Config in client ● install sidecar ● declare : graylog ip, client hostname, tags ● start service Step 2 : Config in Graylog Web ● add tags ● chose what logs you want to collect Step 3 : Checking ● Check colleted log
- 18. 3.3. Architecture/Mechanism/Model of Graylog Deep dive in architecture Graylog Server ● receive log message ● execute log ● communicate with other components Elasticsearch ● store log message ● search engine MongoDB ● store meta infomation ● store config data
- 19. 3.3. Architecture/Mechanism/Model of Graylog Log execute processing Step 1 : ● Spooling & store in disk temporarily ● Prepare for buffer process Step 2 : ● Messages from disk go in to Input Buffer ● Mission : Filter, classify messages Step 3 : ● Messages go in to Output Buffer ● Onward to Elasticsearch or user defined output
- 20. 3.3. Architecture/Mechanism/Model of Graylog Elasticsearch & Graylog ● Clustering ● Use API to communicate ● Use unicast-discovery to recogize other nodes ● Graylog as a Master Node MongoDB & Graylog ● Client - Server mechanism ● Graylog use driver to communicate with MongoDB Internal Graylog components mechanisms
- 21. 3.3. Architecture/Mechanism/Model of Graylog None HA - Small production HA - Bigger Production
- 22. Code show you HOW ! Log show you WHY !
- 23. 3.4. Graylog for OpenStack : 3 steps to know WHY? Just 3 steps to exploiting log in OpenStack
- 24. 3.4. Graylog for OpenStack : 3 steps to know WHY? What should i do when instance spawning fail A. Try to spawn again B. Blame for customer D. Bug again! I’m quit ! C. Take a search in Graylog Incident Response Problem appear ! What should we do?
- 25. 3.4. Graylog for OpenStack : 3 steps to know WHY? Step 1 : Collect logs Take log from : ● nova log ● neutron log ● cinder log ● glance log Step 2 : Analyze Make a search in Graylog : Syntax : instance id + ERROR Step 3 : Now you know WHY Just solve the problem & Go to sleep !
- 26. 3.4. Graylog for OpenStack : 3 steps to know WHY? Tracking a event My instances was rebooted last night ??? When?
- 27. 3.4. Graylog for OpenStack : 3 steps to know WHY? Measuring metric
- 28. DEMO & Q.A
- 29. Bonus : Graylog vs ELK Graylog is coming the closest to the Splunk architecture ! VS