tdc2012
3 likes1,256 views
This document proposes an architecture for distributed indexing, storage, and real-time analysis of logs. It discusses challenges of scaling log collection and analysis across hundreds of servers generating terabytes of data daily. The proposed architecture uses multicast messaging and sharding to distribute indexing and querying across clusters of servers for scalability. It emphasizes low overhead indexing and real-time aggregation of results.
tdc2012
- 1. Um Case de Arquitetura Distribuída para Indexação, Armazenamento e Análise de Logs em Tempo Real Juan Lopes
- 2. COMPLEX EVENT PROCESSING
- 4. LOGS
- 8. marvin@goldenheart ~ $ ssh root@deepthought **** WELCOME TO 1 OF YOUR 38,157,987 SERVERS. TRY THE VEAL. IT'S THE BEST IN THIS FARM. **** root@deepthought ~ $ tail -f /var/log.txt COMO ACESSAR OS LOGS?
- 13. 3TB / DIA
- 14. 3TB / DIA 10.000.000.000 MSGS / DIA 36 MB / SEGUNDO
- 15. TWITTER 400.000.000 MSGS / DIA EM JUNHO/2012
- 16. LOGGLY Amplamente utilizado Primeira opção para cloud Maior plano não-custom: 12GB/dia Preço: $1,779/mês
- 17. GRAYLOG2 Open Source Self-hosted Arquitetura de partes móveis MongoDB ElasticSearch AMQP
- 18. SPLUNK Famoso na área de BigData Destinado ao mundo Enterprise Muitos gráficos e relatórios $6,000 one-time fee: 500MB/dia 500MB < 3TB :(
- 20. JAVA
- 21. HOTSPOT
- 23. VISÃO GERAL Armazenar mensagens Interpretar Indexar
- 24. RFC 3164: SYSLOG <34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8 <priority = facility*8+severity> <date/time> <host> <process> <message>
- 25. CHAVE: VALOR message <34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8 text su, root, failed, for, lonvick, on, /dev/pts/8 facility AUTH severity CRITICAL date 20121011 time 221415 host mymachine process su
- 26. ?
- 27. MG4J Egothor Nutch Oxyus BDDBot Zilverline YaCy Compass Lius Regain Piscator Hounder HSearch
- 28. <FIELD:CONTENT, DOC*> TEXT:ABACAXI ➜ 1, 3, 9 TEXT:BANANA ➜ 2, 3, 10, 42 TEXT:CAJU ➜ 3, 11, 50
- 29. BAIXA ENTROPIA
- 30. <10% de termos únicos menor overhead por mensagem MESSAGE BAG
- 31. VISÃO GERAL Armazenar Interpretar Bufferizar Indexar
- 32. <DOC, FREQ, POSITION*> 1, 4 ➜ 5, 6, 10, 20 3, 1 ➜ 40 9, 4 ➜ 6, 7, 8, 9
- 33. SCORES NÃO IMPORTAM
- 34. NORMAL INDEXAÇÃO BUSCA Field QueryParser Document Query IndexWriter IndexSearcher
- 35. HARDCORE INDEXAÇÃO BUSCA TokenStream TermPositions Document FieldCache IndexWriter IndexReader
- 36. CULPA DA WIDESCREEN CULPA DA WIDESCREEN
- 37. WEB INTERFACE Jersey (REST API) Backbone.js CometD
- 38. WEB INTERFACE Jersey (REST API) Backbone.js CometD "app:apache http 404"? engine browser "OK. listen: /comet/1234568790abcdef"
- 39. CULPA DA WIDESCREEN CULPA DA WIDESCREEN
- 40. COMMAND-LINE INTERFACE Cara e coragem HttpClient CometD /intelie/lognit-cli
- 41. REALTIME (AKA TAIL -F) EVENTS subscriber
- 42. LIGHTWEIGHT TERM TRIE ABRAÇO ABRIGO CHOCOLATE <RAIZ> ABR CHOCOLATE AÇO IGO
- 43. AGREGAÇÃO (AKA WC -L) EVENTS
- 44. ~10.000 eventos / segundo http
- 45. 1 evento / segundo http => count()
- 46. ~100 eventos / segundo http => count() by host
- 47. ~100 eventos / 30 segundos http => count() by host every 30 seconds
- 48. ~100 eventos / 30 segundos http => avg(cputime#) by host every 30 seconds
- 49. CULPA DA WIDESCREEN CULPA DA WIDESCREEN
- 51. taxa de leitura MODERADA taxa de escrita ALTÍSSIMA dependência entre os dados BAIXA
- 52. SHARDING engine Load UDP/TCP 514 engine Balancer engine
- 53. Cluster engine engine engine
- 54. Cluster engine engine engine
- 55. Cluster engine Web HTTP Server engine engine usuário Broker
- 56. Cluster engine Web HTTP Server engine engine usuário
- 57. Cluster engine Multicast engine HTTP engine usuário
- 59. MULTICAST JChannel channel = new JChannel(); channel.setReceiver(new ReceiverAdapter() { public void receive(Message msg) { System.out.println( msg.getSrc() + ": " + msg.getObject()); } }); channel.connect("meuCanalDeChat"); BufferedReader reader = new BufferedReader( new InputStreamReader(System.in)); while(true) { String line = reader.readLine(); channel.send(null, line); }
- 62. BUSCA engine 10 mergesort, take 10 10 last 10 "http_status: 10 engine 404" usuário 10 engine
- 63. BUSCA engine 10 mergesort, take 10 10 last 10 "http_status: 10 engine 404" before {id:84324814} usuário 10 engine
- 64. AGREGAÇÃO http 200 => count() by host host count foo 1234 bar 2345 baz 3456
- 65. AGREGAÇÃO count() + count() + count() engine engine engine
- 66. AGREGAÇÃO http 200 => avg(time) by host host avg_time foo 0.888889 bar 0.224568 baz 5.623424
- 67. AGREGAÇÃO avg(time) + avg(time) + avg(time) ? engine engine engine
- 68. AGREGAÇÃO sum(time) + sum(time) + sum(time) count(time) + count(time) + count(time) engine engine engine
- 69. AGREGAÇÃO sum(time) + sum(time) + sum(time) count(time) + count(time) + count(time) engine engine engine
- 70. E EM PRODUÇÃO?
- 71. 3.5BI DE MENSAGENS 1TB DE DADOS ORIGINAIS 180GB DE ÍNDICE 3 SERVIDORES (LOAD < 0.2)
- 72. UMA ÚLTIMA COISA
- 73. 1700+ TESTS 99% LINES COVERED
- 74. OBRIGADO! /juanplopes @juanplopes intelie.com.br