MongoDB dla administratora
0 likes693 views
This document discusses using MongoDB as a log collector. It provides examples of storing log data from syslog-ng in MongoDB collections, including filtering and parsing logs. It also gives examples of analyzing the log data through map-reduce to find top IP addresses and provides ideas for other uses like CAPTCHAs, error localization, and analytics.
![afmongodb
Przykładowy log z HAProxy:
20120402T21:10:25+02:00 10.x.y.z haproxy[2913]:
Connect from 80.55.238.74:3935 to 195.8.99.31:80
(dom.gratka.pl/HTTP)
HAProxy -> syslog-ng(local) -> syslog-ng(remote) -> mongodb](https://image.slidesharecdn.com/3-ukaszjagieo-mongodbdlaadministratora-wersjaztech-3campa-120404063340-phpapp01/85/MongoDB-dla-administratora-10-320.jpg)
![afmongodb
destination d_mongodb {
mongodb(
valuepairs(
scope("selectedmacros" "nvpairs")
exclude("DATE")
exclude("HOST")
exclude("LEGACY_MSGHDR")
[...]};
rewrite r_rewrite_subst{subst("^Connect from (.*):
[[:digit:]]{1,5} to.*", "${1}", value("MESSAGE"));};](https://image.slidesharecdn.com/3-ukaszjagieo-mongodbdlaadministratora-wersjaztech-3campa-120404063340-phpapp01/85/MongoDB-dla-administratora-11-320.jpg)
![Pomysły – Top10 IP
m=function () {
emit(this.MESSAGE, {count:1});
}
r=function (key, values) {
var total = 0;
for (var i = 0; i < values.length; i++) {
total += values[i].count;
}
return {count:total};
}
db.messages.mapReduce(m, r, { out : "myoutput" } );
db.myoutput.find().sort({value:1});](https://image.slidesharecdn.com/3-ukaszjagieo-mongodbdlaadministratora-wersjaztech-3campa-120404063340-phpapp01/85/MongoDB-dla-administratora-19-320.jpg)
![Pomysły – Top10 IP
> db.messages.mapReduce(m, r, { out : "myoutput" } );
db.myoutput.find().sort({value:1});
{ "_id" : "195.8.99.115", "value" : { "count" : 20771 } }
{ "_id" : "86.111.244.239", "value" : { "count" : 6816 } }
{ "_id" : "94.75.116.206", "value" : { "count" : 6716 } }
{ "_id" : "195.137.162.149", "value" : { "count" : 5638 } }
{ "_id" : "178.63.0.2", "value" : { "count" : 4902 } }
{ "_id" : "31.187.3.80", "value" : { "count" : 3573 } }
{ "_id" : "176.9.73.105", "value" : { "count" : 2734 } }
{ "_id" : "62.129.228.159", "value" : { "count" : 2092 } }
{ "_id" : "209.123.8.176", "value" : { "count" : 2016 } }
{ "_id" : "107.20.40.240", "value" : { "count" : 1927 } }
[...]](https://image.slidesharecdn.com/3-ukaszjagieo-mongodbdlaadministratora-wersjaztech-3campa-120404063340-phpapp01/85/MongoDB-dla-administratora-20-320.jpg)
MongoDB dla administratora
- 2. MongoDB dla administratora ● ● Łukasz Jagiełło – l.jagiello@gratka-technologie.pl http://geekandpoke.typepad.com/geekandpoke/2011/01/nosql.html
- 3. Agenda ● Kto ja i co ja ● Jak to się robi dzisiaj ● Jak to można zrobić jutro ● Pomysły...
- 4. Kto ja i co ja ? Ukradzione z wyszukiwania w Grafika google.com
- 5. Jak to się robi dzisiaj
- 6. Jak to się robi dzisiaj filter f_kern { facility(kern); }; filter f_cron { facility(cron); }; destination d_kern { file("/var/log/$YEAR$MONTH$DAY/kern"); }; destination d_cron { file("/var/log/$YEAR$MONTH$DAY/cron"); }; log { source(src); filter(f_kern); destination(d_kern); }; log { source(src); filter(f_cron); destination(d_cron); };
- 7. Jak to się robi dzisiaj
- 8. Jak to się robi dzisiaj filter f_kern { facility(kern); }; filter f_cron { facility(cron); }; destination d_collector { udp("10.X.Y.Z" port(514)); }; log { source(src); destination(d_collector); };
- 9. Syslog-ng 3.4.0alpha1 Moduły: ● File ● JSON ● MongoDB ● Stream ● SQL ● CSV ● Socket ● Network ● Console
- 10. afmongodb Przykładowy log z HAProxy: 20120402T21:10:25+02:00 10.x.y.z haproxy[2913]: Connect from 80.55.238.74:3935 to 195.8.99.31:80 (dom.gratka.pl/HTTP) HAProxy -> syslog-ng(local) -> syslog-ng(remote) -> mongodb
- 13. MongoDB „Capped collections are fixed sized collections that have a very high performance auto-FIFO age-out feature (age out is based on insertion order). They are a bit like the "RRD" concept if you are familiar with that.” „In addition, capped collections automatically, with high performance, maintain insertion order for the documents in the collection; this is very powerful for certain use cases such as logging.” > db.createCollection("mycoll", {capped:true, size:100000}) > db.createCollection("messages", {capped: true, size:10000000, autoIndexId:true})
- 15. MongoDB Puchnące mongo ? 17772 root 15 0 1463m 474m 276m S 0.0 8.0 2011 147:00 ./mongod fork logpath /d0/db/mongod.log ● ponad 5 miesięcy działania ● średnio co 30min wszystkie dane są wymienione ● 10mln rekordów w bazie – zawsze ! ● 4GB na dysku !! Wesje 2.x działają DUŻO lepiej niż wcześniejsze !!
- 19. Pomysły – Top10 IP m=function () { emit(this.MESSAGE, {count:1}); } r=function (key, values) { var total = 0; for (var i = 0; i < values.length; i++) { total += values[i].count; } return {count:total}; } db.messages.mapReduce(m, r, { out : "myoutput" } ); db.myoutput.find().sort({value:1});
- 20. Pomysły – Top10 IP > db.messages.mapReduce(m, r, { out : "myoutput" } ); db.myoutput.find().sort({value:1}); { "_id" : "195.8.99.115", "value" : { "count" : 20771 } } { "_id" : "86.111.244.239", "value" : { "count" : 6816 } } { "_id" : "94.75.116.206", "value" : { "count" : 6716 } } { "_id" : "195.137.162.149", "value" : { "count" : 5638 } } { "_id" : "178.63.0.2", "value" : { "count" : 4902 } } { "_id" : "31.187.3.80", "value" : { "count" : 3573 } } { "_id" : "176.9.73.105", "value" : { "count" : 2734 } } { "_id" : "62.129.228.159", "value" : { "count" : 2092 } } { "_id" : "209.123.8.176", "value" : { "count" : 2016 } } { "_id" : "107.20.40.240", "value" : { "count" : 1927 } } [...]
- 21. Pomysły cd. http://www.slideshare.net/octplane/mongofr-mongodb-as-a-log-collector
- 22. Pomysły cd. http://www.slideshare.net/octplane/mongofr-mongodb-as-a-log-collector
- 23. Pomysły cd. ● CAPTCHA ● Lokalizowanie błędów ● Kontrola ● Liczniki (youporn.com 8GB-15GB/godzinę) ● Itd. Do statystyk mamy ciekawsze narzędzia