Cloud Foundry CF LOGS stack
0 likes521 views
This document discusses a Cloud Foundry logging stack Bosh release that replaces the default syslog-based logging with Kafka. It notes that Kafka can ingest millions of logs per second, provides persistence, and is easier to consume than syslog. The release includes Kafka, a Firehose nozzle to send logs to Kafka, Kafka Manager, and a syslog forwarder. It will create Kafka topics for each event type and forward logs and app info to any syslog endpoint. The full release is available on GitHub.
1 of 10
Download to read offline
Cloud Foundry CF LOGS stack
- 1. CFLOGS bosh release (LT) Etourneau Gwenn @The_Shinji62
- 2. Me …. Etourneau Gwenn Senior Solution Architect @Pivotal (Cloud Foundry Superman) https://twitter.com/the_shinji62 L,,, https://github.com/shinji62
- 3. Cloud Foundry logging stack 3
- 4. 4 Almost no buffer In memory buffer If not consume fast enough you lose logs. You lose LOGS Cloud Foundry logging stack Syslog
- 5. 5 KAFKA Syslog Cloud Foundry logging stack
- 6. Why Kafka ? 6 ● Kafka ‒ Can ingest million of logs by second ‒ Persistency ‒ More easy to consume ‒ A lot of community around Kafka Losing logs is still possible No real 100% no loss solution but this should help
- 7. CF logs 7 ● One command deploy bosh deploy ● Release including ‒ KAFKA ‒ Firehose kafka nozzle (fork from the Rakuten one) ‒ Kafka-manager ‒ CF syslog forwarder
- 8. CF logs 8 ● Create a kafka topic for every events type ‒ LogMessage, ValueMetrics and so on .. ● Forwarder to syslog is included ‒ Can forward to any syslog endpoint ‒ Include Apps info. Org etc ..
- 9. CF logs Bosh release 9 https://github.com/shinji62/cflogs-boshrelease