Noit ocon-2010
0 likes1,674 views
This document provides an overview and instructions for installing and configuring Reconnoiter, a system for large-scale trending and fault detection. It describes the goals of the system, its architectural design including main components, and steps for installing the code, database, and web console. It also provides information on setting up SSL, configuring the noitd and stratcond components, and contact information for support and hiring.
Noit ocon-2010
- 1. Reconnoiter: Large-scale trending and fault-detection / another product built from pain Sunday, August 1, 2010
- 2. Goals • make checks cheap: 10000+ checks on cheap hardware • centralized configuration management. • decentralized configuration manipulation. • decouple data collections from • visualization/trending • fault-detection • make life suck just a little less. Sunday, August 1, 2010
- 3. Architectural Design Sunday, August 1, 2010
- 4. System Components • noitd • C, hybrid thread/event model, async I/O, small, fast, efficient, • extensible in lua. • stratcond • C, brokers and aggregates data, feeds PostgreSQL, • feeds Esper complex event processing system for fault-detection, • has a comet-style webserver built in for feeding web clients. • postgres • webconsole • PHP, almost entirely AJAX based, • uses canvas to draw. Sunday, August 1, 2010
- 5. Some basics on the architecture • Everything important happens over SSL: • Services exposed over SSL with certificates, • Client connects using client certificates as well. • Designed so that data collection is journalled and replayed • prevents data loss due to transient networking issues between NOC an data center. • There are a lot of moving parts... • designed to work when the parts don’t get along. Sunday, August 1, 2010
- 6. Installing the code • svn co https://labs.omniti.com/reconnoiter/tags/wangle • cd trunk • autoconf • ./configure • make • make install Sunday, August 1, 2010
- 7. Installing the database • createdb reconnoiter • createlang -d reconnoiter plpgsql • createuser reconnoiter • createuser stratcon • createuser prism • psql -U postgres reconnoiter • BEGIN; • i sql/reconnoiter_ddl_dump.sql • COMMIT; • install the crontab in sql/crontab Sunday, August 1, 2010
- 8. Installing the web console • Web web UI lives in trunk/web/ui • Setup Apache (left as an exercise for the reader) Sunday, August 1, 2010
- 9. SSL is underneath everything • Setup SSL certs: • Need a CA (even if a dummy CA) • need a cert (signed) for each noitd server • need a client cert (signed) for the stratcond server • need a web cert (signed) for the stratcond server (future**) • More details by running make test and looking into the • test-noit.conf • test-stratcon.conf Sunday, August 1, 2010
- 10. Configure noitd • noit.conf • pretty good as a starting point, just clear out all the checks • module loading is at boot time, so make sure all the modules you want are loaded. • checks can be added at run time via the text console. • noitd must run as root, it drops privileges after module initialization. • /usr/local/sbin/noitd -c /usr/local/etc/noit.conf Sunday, August 1, 2010
- 11. Configure stratcond • stratcon.conf • pretty good as a starting point, just clear out all the noit addresses • each noit in the field requires a line in the stratcon.conf file: • <noit address="10.225.209.25" port="43191" /> • database passwords must be configured • hostname and document_domain must be configured • /usr/local/sbin/stratcond -c /usr/local/etc/stratcon.conf Sunday, August 1, 2010
- 12. Thank You • Documentation at https://labs.omniti.com/trac/reconnoiter • Commercial support available from OmniTI. • Please join in. • OmniTI is hiring: http://omniti.com/is/hiring • Thanks! Sunday, August 1, 2010