Log Collection and Analysis with elk Stack
- 1. ELK(Elastic Search, Logstash, Kibana) Stack (주) 투라인코드 설상민 2020.06 © TWOLINECODE Inc. All rights reserved.
- 2. TWOLINECODE 목차 © TWOLINECODE Inc. All rights reserved. ➢ ELK Stack 이란? ➢ Elasticsearch와 관계형 DB ➢ Elasticsearch RESTful API ➢ Elasticsearch Mapping ➢ LogStash ➢ LogStash filter ➢ Kibana
- 3. TWOLINECODE ELK Stack 이란? © TWOLINECODE Inc. All rights reserved. - Elasticsearch JSON 기반의 분산형 오픈 소스 RESTful 검색 및 분석 엔진 - LogStash 여러 소스에서 동시에 데이터를 수집하여 변환 stash로 전송하는 데이터 처리 파이프 라인 - Kibana 차트와 그래프를 이용한 데이터 시각화 도구 * Beats 단일 목적 데이터 수집기 제품군 ( File, Packet, Window Event Log, etc… )
- 4. TWOLINECODE Elasticsearch와 관계형 DB © TWOLINECODE Inc. All rights reserved.
- 5. TWOLINECODE Elasticsearch와 관계형 DB의 데이터 저장 © TWOLINECODE Inc. All rights reserved. - Elasticsearch의 데이터 저장 - 관계형 DB의 데이터 저장
- 6. TWOLINECODE Elasticsearch RESTful API © TWOLINECODE Inc. All rights reserved. - Elasticsearc의 Data CRUD - RESTful API Data CRUD Elasticsearch Restful INSERT POST SELECT GET UPDATE PUT DELETE DELETE
- 7. TWOLINECODE Elasticsearch RESTful API © TWOLINECODE Inc. All rights reserved. - Elasticsearc POST API : Data Insert / Create - Elasticsearc GET API : Data Read / Select
- 8. TWOLINECODE Elasticsearch RESTful API © TWOLINECODE Inc. All rights reserved. - Elasticsearc PUT API / Data Update
- 9. TWOLINECODE Elasticsearch RESTful API © TWOLINECODE Inc. All rights reserved. - Elasticsearc DELETE API / Data Delete
- 10. TWOLINECODE Elasticsearch RESTful API © TWOLINECODE Inc. All rights reserved. - Elasticsearch API 검색 – Index 전체
- 11. TWOLINECODE Elasticsearch RESTful API © TWOLINECODE Inc. All rights reserved. - Elasticsearch API 검색 – URI 검색
- 12. TWOLINECODE Elasticsearch Mapping © TWOLINECODE Inc. All rights reserved. - 관계형 Database의 스키마에 해당. - Document의 type 정보에 대한 기술. - Mapping 정보 없이 Document Handling 가능. - Mapping 정보에 따라 Kibana에서 Visualize 기능 등에 제약이 있을 수 있음
- 13. TWOLINECODE Logstash © TWOLINECODE Inc. All rights reserved. - Logstash는 Conf 파일을 기반으로 동작 - Conf 파일은 기본적으로 Input, filter, output으로 구성. Input : beats, jdbc, syslog, tcp, udp, file, stdin 등 데이터 소스 선택 output : elasticsearch, jdbc, file, stdout 등 filter : 입력받은 데이터를 가공
- 14. TWOLINECODE Logstash © TWOLINECODE Inc. All rights reserved. - Logstash 입출력 Conf 파일 예시
- 15. TWOLINECODE Logstash filter © TWOLINECODE Inc. All rights reserved. - 입력 데이터의 분해, 추가, 삭제, 변형 등의 과정 grok, mutate, date 등 - 입력 순서 대로 적용됨.
- 16. TWOLINECODE Logstash filter © TWOLINECODE Inc. All rights reserved. - Grok
- 17. TWOLINECODE Logstash logfile->elasticsearch conf © TWOLINECODE Inc. All rights reserved. - Logstash file input – log File, output – elasticsearch 설정
- 18. TWOLINECODE Kibana © TWOLINECODE Inc. All rights reserved. - Create Index Patterns Define index pattern & select Time filter field
- 19. TWOLINECODE Kibana © TWOLINECODE Inc. All rights reserved. - Discover Time filter 설정을 통한 데이터 검색 가능
- 20. TWOLINECODE Kibana © TWOLINECODE Inc. All rights reserved. - Discover Document JSON
- 21. TWOLINECODE Kibana © TWOLINECODE Inc. All rights reserved. - Visualization
- 22. TWOLINECODE Kibana © TWOLINECODE Inc. All rights reserved. - Visualization : Vertical Bar
- 23. TWOLINECODE Kibana © TWOLINECODE Inc. All rights reserved. - DashBoard
- 24. TWOLINECODE 참조 사이트 모음 • https://www.elastic.co/kr/what-is/elk-stack • https://victorydntmd.tistory.com/308 • https://www.slideshare.net/kjmorc/ss-49009522 • https://www.inflearn.com/course/elk-%EC%8A%A4%ED%83%9D-%EB%8D%B0%EC%9D%B4%ED%84%B0- %EB%B6%84%EC%84%9D# • https://victorydntmd.tistory.com/313?category=742451 : Elasticsearch Search API • https://victorydntmd.tistory.com/314?category=742451 : Elasticsearch Query DSL • https://esbook.kimjmin.net/ : Elasticsearch 가이드 북 © TWOLINECODE Inc. All rights reserved.
- 25. TWOLINECODE 마무리 하기 © TWOLINECODE Inc. All rights reserved.
- 26. Thank you © TWOLINECODE Inc. All rights reserved.