Fluentd meetup logging infrastructure in paa s
9 likes2,309 views
This document discusses using Fluentd for logging infrastructure in Platform as a Service (PaaS). It describes how Fluentd can be used to collect and persist logs from applications running on Cloud Foundry PaaS in a way that logs are not lost when applications are updated or restarted. A Fluentd daemon is attached to each application instance to monitor and forward logs to log storage servers.
![Current logging issue in Cloud Foundry
$ vmc push yssk22-myapp!
!
$ curl http://yssk22-myapp.cloudfoundry.com/!
!
$ vmc logs yssk22-myapp!
!
[Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !
!
$ vmc update yssk22-myapp!
!
$ vmc logs yssk22-myapp!
7](https://image.slidesharecdn.com/fluentdmeetup-logginginfrastructureinpaas-120827014544-phpapp01/85/Fluentd-meetup-logging-infrastructure-in-paa-s-7-320.jpg)
![Current logging issue in Cloud Foundry
$ vmc push yssk22-myapp!
!
$ curl http://yssk22-myapp.cloudfoundry.com/!
!
$ vmc logs yssk22-myapp!
!
[Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !
!
$ vmc update yssk22-myapp!
!
$ vmc logs yssk22-myapp!
Oops, logs are gone.
8](https://image.slidesharecdn.com/fluentdmeetup-logginginfrastructureinpaas-120827014544-phpapp01/85/Fluentd-meetup-logging-infrastructure-in-paa-s-8-320.jpg)
![Misc: Adding new forwarding nodes
dynamically through NATS?
# DEA
@droplets.each_value do |instance|
begin
instance_collector = Fluent::Logger::FluentLogger.new('logs', instance[:logging_ip], instance[:logging_port])
instance_collector.post('storage', {'type': 'serverConf', 'host': '127.0.0.1', 'port': 23938 })
rescue => e
@logger.error "Error: #{e} -- #{e.backtrace}"
end
end
# Extended Forward plugin
module Fluent
class CloudfoundryAppForwardOutput < ForwardOutput
Plugin.register_output('cf_app_forward', self)
def emit(tag, es, chain)
es.each do |emit_time, record|
# Add a new forwarding server
if record.has_key?('type') and record['type'] == 'serverConf’
add_node(record['host'], record['port’])
next
end
28](https://image.slidesharecdn.com/fluentdmeetup-logginginfrastructureinpaas-120827014544-phpapp01/85/Fluentd-meetup-logging-infrastructure-in-paa-s-28-320.jpg)
Fluentd meetup logging infrastructure in paa s
- 1. Logging infrastructure in PaaS with Fluentd Yohei Sasaki / Waldemar Quevedo Paas Dev&Ops team 1 Rakuten. Inc, 22/08/2012
- 2. About us § ささきと申します。日本人です。 § ワリと申します。メキシコ人です。 (Sasaki, from Japan) (Wally, from Mexico) § Living in Japan since 1982. § Living in Tokyo since 2010. § Tech Lead of the PaaS Dev&Ops § Member of the PaaS Dev&Ops team team at Rakuten, Architect of at Rakuten, programming in Ruby. Rakuten PaaS. § Emacs user (org-mode rocks, try .org § Node.js user / node-fluentd-logger markup in Github) § CouchDB, …etc § OpenStreetMap (Mapnik rocks) 2
- 3. Agenda § Cloud Foundry § Cloud Foundry + Fluentd § Fluentd Pros & Cons § Misc 3
- 4. Cloud Foundry • Open Application Platform as a Service project • By VMware • Written in Ruby • OpenSource http://github.com/cloudfoundry/vcap • Hosted version http://www.cloudfoundry.com/ 4
- 5. Why Cloud Foundry? • Last year it was decided to introduce Cloud Foundry to build a PaaS in Rakuten http://techtarget.itmedia.co.jp/tt/news/1111/28/news02.html http://www.slideshare.net/rakutentech/cloud-foundry-10423477 • Main points: – Open Source – Separation of concerns from IaaS – Scalable architecture – Multi language 5
- 6. What Cloud Foundry does? • Easy to deploy applications!! $ mkdir app.js $ vi app.js $ vmc push yssk22-myapp $ vmc instances yssk22-myapp 3 • Not easy to deploy business ready applications yet. – We are now solving our own issues by modifying Cloud Foundry source – One critical thing is Logging! – Now we’ve solved this matter basically, let’s share our case study. 6
- 7. Current logging issue in Cloud Foundry $ vmc push yssk22-myapp! ! $ curl http://yssk22-myapp.cloudfoundry.com/! ! $ vmc logs yssk22-myapp! ! [Wed, 22 Aug 2012 08:03:48 GMT] Hello World! ! ! $ vmc update yssk22-myapp! ! $ vmc logs yssk22-myapp! 7
- 8. Current logging issue in Cloud Foundry $ vmc push yssk22-myapp! ! $ curl http://yssk22-myapp.cloudfoundry.com/! ! $ vmc logs yssk22-myapp! ! [Wed, 22 Aug 2012 08:03:48 GMT] Hello World! ! ! $ vmc update yssk22-myapp! ! $ vmc logs yssk22-myapp! Oops, logs are gone. 8
- 9. Persisting logs is a must for business applications! • Monitoring • Analysis • Visualization Ask a data scientist => Sorry we lost everything :P http://www.slideshare.net/doryokujin/hadoop-and-the-data-scientist 9
- 10. Cloud Foundry does not persists the logs from apps… • Enter Fluentd! 10
- 11. Brief overview of Cloud Foundry architecture Droplet Execution Agent Source: http://www.slideshare.net/derekcollison/design-of-cloud-foundry 11
- 12. Approach Attach a Fluentd collector to one application instance upon dispatch. DEA app fluentd app fluentd Log Storage DEA app fluentd app fluentd 12
- 13. Collecting the logs from the apps DEA • Log collector monitors changes in the files app fluentd – Stdout, stderr logs in general – Catalina, manager, host-manager logs for Java – error.log for PHP apps app fluentd app fluentd • Also makes available: – VCAP_LOG_PORT env variable to flush logs directly app fluentd – VCAP_LOG_STREAM_PORT to inspect the logs in realtime app fluentd 13
- 14. Collecting the logs from the apps DEA app fluentd Log Storage Fluent::Logger => VCAP_LOG_PORT Sends logs directly to Fluentd making unnecessary having to write to file. VCAP_LOG_STREAM_PORT Streams the logs to a client {‘type’: ‘serverLog’, ‘text’: ‘Hello World’, ‘level’: ‘INFO’} 14
- 15. Small setup for collecting the logs DEA CloudController app fluentd Log Storage Server app fluentd /logs/:path fluentd Rack app Collected log files DEA Log Storage Server CloudController app fluentd Rack app app fluentd Collected log files /logs/:path 15
- 16. How is it setup? app/ ! fluentd/! Attached to an application instance on |-- stderr.stat (1)! startup by DEA |-- stdout.stat (2)! • (1) and (2) is a “stat file” used by the input plugin, |-- fluentd.conf (3)! which allows it to pick up from the last position it |-- startup (4)! had monitored. |-- stop (5)! fluentd_child.pid (6)! • (3) is the configuration of the plugins dynamically generated by the DEA fluentd.pid (7)! logs/! • (4) and (5) are the scripts executed to manage the |-- fluentd.log (8)! fluentd process |-- stdout.log! • (6) and (7) are the supervisor and main process |-- stderr.log! ids from Fluentd ruby! run.pid! • (8) is where any logs produced by the logging daemon are stored startup! stop! 16
- 17. fluentd.conf :: Input configuration ! Set up VCAP_LOG_PORT to flush the logs <source>! directly type forward! port 12345! </source>! ! <source>! type cf_app_logs! instance_dir /var/vcap.local/dea/apps/appname-0-3e9676485562c64d81a61de2df05903a! runtime ruby19! framework sinatra! tag logs.storage! </source>! ! ! Specify runtime and dir to monitor files 17
- 18. fluentd.conf :: Output configuration <match logs.**>! type copy! Settings for when forwarding <store>! to the log storage, we include user type cf_app_forward! information here so that it cannot be flush_interval 5s! overrided by a logging library. buffer_type file! buffer_path fluentd/fluentd_buffer! ! cf_app_user foo@example.com! cf_app_name appname! cf_app_instance_id 3e9676485562c64d81a61de2df05903a! cf_app_instance_index 0! 18
- 19. fluentd.conf :: Output configuration <server>! name logserver! Location of the log storage server: host 192.168.2.1! - We use a active-standby setup at the port 4224! moment. </server>! - When primary Logserver is down, logs <server>! are forwarded to the one on standby. name backup_logserver! host 192.168.2.2! port 4224! When both are down, fluentd buffers to standby! files the logs locally until one of them is </server>! available. </store>! <store>! type cf_app_logs_streaming! port 43208! </store>! </match>! Set up VCAP_LOG_STREAM port to enable tail –f like support. 19
- 20. …And now that we managed to collect the logs, we also can support the following 2 commands: 20
- 21. vmc tail 21
- 22. vmc log-storage 22
- 23. What we could do • Persisting the logs in Cloud Foundry is a must! • Support to persist the logs is a key issue in adoption of Cloud Foundry for developing application services. • Thanks to Fluentd we could enable users to have log archives in a straightforward way. • Cloud Foundry and Fluentd themselves are both written in Ruby so integration was relatively easy. 23
- 24. What we could not do • It was difficult to let the user have access to the logs from Nginx (used as a Router in CF) # Sample release process! vmc push app-v1 --runtime ruby19 => app-v1.cf.rakuten.co.jp! vmc map app-v1 olympics.rakuten.co.jp => Released!! vmc push app-v2 --runtime ruby19 => app-v2.cf.rakuten.co.jp! vmc map app-v2 olympics.rakuten.co.jp => Load balancing between apps! vmc unmap app-v1 olympics.rakuten.co.jp => Release 2 complete.! • Then the app-v1.cf.rakuten.co.jp subdomain becomes available to other user. • Need to keep track that user X had Y domain at Z time for application named A in real time while storing the logs. 24
- 25. Fluentd: Pros and Cons • Pros – Easy to include in Cloud Foundry compared to other solutions • Possible to include it like gem 'fluentd' in a Ruby component – Easy to change storage technology depending on scale – Easy to extend – Growing community o/ • Cons – Fluentd needs better testing tools – Better materials about Fluentd from the community? • Learning curve is already low • Just we need to share more about how we are using Fluentd • e.g. 'The Little Fluentd Book', 'Advanced Fluentd Recipes’ – Cool.io is not maintained 25
- 26. Misc: Playing with the message bus • Cloud Foundry is built with Eventmachine. • Fluentd is built with Cool.io. • Cool.io and Eventmachine do not play along very well. • The message bus from Cloud Foundry named NATS is also built using Eventmachine. 26
- 27. Misc: NATS input plugin # This works, though a warning is displayed from Cool.io overriding Eventmachine! require 'nats/client'! ! module Fluent! class NatsInput < Input! Plugin.register_input('nats', self)! def run! NATS.start(:uri => @nats_connection_uri) do! ! # Triggered when an url changing event occurs (remap, change)! NATS.subscribe('dea.update') do |msg|! Engine.emit('logs.dea_update', Time.now.to_i, msg)! end! ! # Triggered when an application is started or dea is started! NATS.subscribe('dea.*.start') do |msg|! Engine.emit('logs.dea_start', Time.now.to_i, msg)! end! end! end! 27
- 28. Misc: Adding new forwarding nodes dynamically through NATS? # DEA @droplets.each_value do |instance| begin instance_collector = Fluent::Logger::FluentLogger.new('logs', instance[:logging_ip], instance[:logging_port]) instance_collector.post('storage', {'type': 'serverConf', 'host': '127.0.0.1', 'port': 23938 }) rescue => e @logger.error "Error: #{e} -- #{e.backtrace}" end end # Extended Forward plugin module Fluent class CloudfoundryAppForwardOutput < ForwardOutput Plugin.register_output('cf_app_forward', self) def emit(tag, es, chain) es.each do |emit_time, record| # Add a new forwarding server if record.has_key?('type') and record['type'] == 'serverConf’ add_node(record['host'], record['port’]) next end 28
- 29. FIN You can enable log collection to your Cloud Foundry by merging this patch: https://github.com/rakutentech/dea/commit/ 64d271904de1e195cc90ca1958eadf704f530d94 Checkout our github repo for more Cloud Foundry related things: https://github.com/rakutentech/ That’s all, thanks! ご清聴ありがとうございました！ 29