NoSQL - "simple" web monitoring
Download as PPT, PDF1 like745 views
The document discusses monitoring systems for NoSQL in a web computing environment, highlighting existing monitoring tools and their limitations. It explores the advantages and challenges of using NoSQL databases like CouchDB and MongoDB, addressing issues related to deployment, flexibility, and query management. Additionally, it outlines concepts like MapReduce, document structures, views, and the integration of web applications with CouchDB.
NoSQL - "simple" web monitoring
- 2. HEP Computing environment • Big datacenters • Existent monitoring o Nagios/Cacti/similars monitoring o Site Availability Monitoring/Hammercloud o High level tests (Grid services) o General Monitoring of grid services • Missing/incomplete monitoring o Specific/detailed information about specific Grid services o Transfer Management tool (PhEDEx for CMS) o Batch system (Condor,PBS,LSF,Torque, etc) o Quota report based on Experiment's characteristic organization (Physics groups, T2 federation, etc)
- 3. Options • SGDB's & SQL - Might be too much o More costs for deployment o Less flexibility o Increase application's complexity/time spent o Black sheep - SQLite - however, scaling problems • Scripting - Might be too bad o Database is too much, query the info from CLI, format and print (text file, HTML) o Parse logs, format and print (text file, HTML) • Bottom line is evaluate – Effort x Result x Quality you need
- 4. NoSQL It doesn't mean SQL is a bad idea =) CouchDB,MongoDB, etc...
- 5. JSON My apologies to the ones that know it... Format that represents associative arrays (ke value pairs) { "Name": "Smith", "age": 25, "address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": "10021" }, }
- 6. For the curious... MapReduce Trivial parallelization First Map (filter, get documents subset) Then reduce (count, sum, etc)
- 7. Overview Server Databases Documents Unique ID, revision All the rest is free Key/Value – datatype for many languages Views Static sets of what you want to show View code scans all documents in the database Caching For Your Information – “CouchApps”
- 8. { Documents "_id": "osgce@osgce.hepgrid.uerj.br#1146503.0#1335286041", "_rev": "10-12a26ecf9bc0fecf5ed9073c374a620c", "data": { "NumJobStarts": { "i": "1" }, "EnteredCurrentStatus": { "i": "1335415262" }, "CurrentHosts": { http://<MYSERVER>/<MYDATABASE>/<DOCUMENT_ID> "i": "0" } } }
- 9. Views Extract from the documents, only what you want to see Only documents you want to see function(doc) { if (doc.Type == "customer") { emit(doc._id, {LastName: doc.LastName, FirstName: doc.FirstName, Address: doc.Address}); } } http://<MYSERVER>/<MYDATABASE>/_design/<MYDESIGN>/ _view/<MYVIEW>
- 10. CouchApps Webserver CouchDB hosts your HTML(5...)+Javascript application Upload to the given “DB”, has access to it's views, and docs. DB is replicated, everything goes together http://<MYSERVER>/<MYDATABASE>/_design/<MYCOUCHAPPAME>/index.html
- 11. GlobalMonitor