Firehose
Download as PPTX, PDF3 likes2,077 views
The document summarizes how to build a real-time data stream application with NodeJS. It discusses using NodeJS, CoffeeScript, and ZeroMQ to build a resilient, scalable, and accurate streaming application. It notes some challenges in building an asynchronous tail library for NodeJS, including errors only manifesting at high throughput and existing implementations being wrong. Performance is limited by IO beyond 5MB, but throughput is consistent otherwise. The stack, architecture, challenges, learnings, performance, and references are outlined.
Firehose
- 1. Firehose Building a real-time data stream with NodeJS Luca Grulla@Forward London, 2/8/2011
- 2. Introduction Or what we wanted to build
- 3. We like to have a (near) real-time view of the data stream
- 4. Resilient
- 5. Scalable
- 6. Accurate (but not necessarily precise)
- 8. Verse and Chorus Or how we built it
- 9. Stack • NodeJS (0.4.9) • CoffeeScript • ZeroMQ
- 10. Architecture
- 11. Interlude Or joys and pitfalls of building a tail library for node
- 12. Living the asynchronous dream
- 13. Living the asynchronous dream • Error manifests itself only at high throughput (7MB/secs) • The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)
- 14. Breathing in an asynchronous way
- 15. Learnings • You need to think asynchronous to write asynchronous code • Dispatch internal events if you need to keep control of the flow • “Internal queues” to scale(up to a certain limit)
- 16. Performance • Beyond 5MB we start to fill the internal queue • We are limited by IO, not from CPU/memory • Consistent throughput
- 17. Questions ?
- 18. References • npm install tail • https://github.com/forward/node-tail
Editor's Notes
- #4: Because you can do cool stuff with this !!Previous implementation was with Flume+zeroMQ..flume was unraliable