Monitoring at scale - Sensu Kafka Kafka-connect Cassandra PrestoDB
- 1. Sensu data monitoring system at scale LEANDRO TOTINO PEREIRA SYSTEM ENGINEER Sensu Kafka Kafka-connect Cassandra PrestoDB
- 2. Agenda Problem? Sensu Monitoring system Taking a little about the componentes/concepts used PrestoDB vs Circuit breaker Taking about the full architecture built Questions?
- 3. Problem? Opensource/comercial doesn´t scale very well (poller, proxy and databases) These system often do have SQL database which doesn´t scale well (sharding, master/slave, no TS database for metrics) These systems are hard to customize for our needs. (integrations with other systems and dashboards) Don´t provide any queue layer to avoid overload.
- 4. Sensu Monitoring System Provide nice API (good for customizations) Suport Nagios commands pattern (0,1,2,3 return for OK, WARNING, CRITICAL,UNKNOWN) Scaling and distribuited Mutator support (Modify output events) Handler Support No support for perfdata (alternatives, we will show later how) Support no metrics ( alternatives, we will show later how) Checks should be implemented inside machine and executed by sensu-agent Pros Cons
- 5. Cassandra Database Elastic Scalability Peer to peer architecture High Availability and Fault Tolerance Modeling tables restriction ( cqlsh informs if your modeling model is right/good to run your queries to improve performance) High performance (customize tables to perform your queries even more) TTL support for inserted rows Tunable Consistency
- 6. Kakfa? Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. (kafka website) Elastic highly scalable fault-tolerant Equally viable for small, medium, & large use cases Exactly-once processing semantics No separate processing cluster required Develop on Mac, Linux, Windows
- 7. Stream Data Platform: A central hub for all your data A stream data platform captures streams of events or data changes and feeds them to other data systems such as relational databases, key-value stores, Hadoop, or the data warehouse.In our case, we are going to use Kafka Cassandra sink to input data into Cassandra database
- 8. Kafka Connect /Cassandra-Sink The DataStax Certified Connector, developed by DataMountaineer, simplifies writing data from Kafka into Cassandra. The connector converts the value from the Kafka Connect SinkRecords to Json. A fail fast thread pool is then used to insert the records asynchronously into Cassandra. Kafka Connect is a framework included in Apache Kafka that integrates Kafka with other systems. Its purpose is to make it easy to add new systems to your scalable and secure stream data pipelines. To copy data between Kafka and another system, users instantiate Kafka Connectors for the systems they want to pull data from or push data to. Source Connectors import data from another system (e.g. a relational database into Kafka) and Sink Connectors export data (e.g. the contents of a Kafka topic to an HDFS file).
- 9. PrestoDB vs Circuit Breaker Presto allows querying data where it lives, including Hive, Cassandra, relational databases or even proprietary data stores. A single Presto query can combine data from multiple sources, allowing for analytics across your entire organization. Another option is to use circuit breaker pattern in your application through consul/etcd to check the services is up and notify when something is down avoiding impacting another resources.
- 10. Taking about architecture OBS: if you don´t need to get data from all Cassandra and correlate them. I really recommend to use circuit breaker concept at dashboard in the case some database get down. Kafka have two topics: alarm topics to receive all output with status different 0 from nagios command and it´s not silenced and another one to receive all output to save in the metrics database
- 11. Sensu data Sensu servers got all output from checks (because type metrics) and Kafka-send through validate/normalize the data and specify which topic to send (alarms or metrics). Kafka connect is going to sink the data to Cassandra database. Handler configurationCheck configuration Sensu Output (all the information) when get on Kafka_send command Check output from the server (monitoring client) Kafka-send is going to check if output contains “status != 0” or “silenced != 0” to send to alarm Kafka topic, all the output is send to metrics topic in a normalize database pattern.
- 12. Sensu – Deleting servers Even you delete a server from Sensu, you must get still keepalived checks, you should delete on sensu registry as well to take away keepalived checks.
- 13. Sensu – Silence checks
- 14. Cassandra Database and testing sink demo For modeling your table for TS database we strong recomend use composite key to avoid to get partition limitations (2 bilhões rows) and clustering order as DESC in the timestamp column. We shouldn´t consider use materalize to join tables. You should model the table to perform even more your queries. Sending formated data to kafka Showing the data saved on Cassandra table through Kafka connect
- 15. Presto demo Create another Cassandra db and associate them on presto as cassandra2 Quering both tables on diferentes Cassandra servers Quering both tables and union both of them.
- 16. Presto UI