Analysis big data by use php with storm
0 likes372 views
The document discusses different technologies for real-time data collection and analysis, including Kafka for collecting and distributing streaming data, Storm for distributed real-time computation, and using PHP and FastCGI to parse real-time logs from Kafka in a Storm topology. It provides an overview of these technologies and their features, and proposes an architecture to collect logs with Kafka, process them with Storm and PHP, and output results through FastCGI.
Analysis big data by use php with storm
- 1. PHP |
- 3. Pla$orm Services Group Sam AWS S3 CDN ATS Biwing 1k 30+ 10+ Monika G o Rambone PHP Front-End Services Panel Skylee C++ Go
- 4. Content Summary " # PHP PHP $ % Case
- 5. Real 7me data collec7on "
- 6. How to collect data in real 7me ? …… ……
- 7. Ka<a —— a high-throughput distributed messaging system source from LinkedIn Producer KaLa LinkedIn Apache KaLa Producer & " ' ( Model Two models: queuing and publish-subscribe. the la<er is more commonly used. high-level abstracPon Each Topic corresponds to one or more log files. Each Topic has one or more par@@ons. Feature Scalability & Durability & Reliability & Performance & Fault- tolerant Ecosystem Stream processing systems, Hadoop integraPon, monitoring, and deployment tools.
- 8. Ka<a should be no7ced when using Ka<a ) KaLa offset Consumer Buffer KaLa gzip snappy lz4 CPU KaLa KaLa KaLa Topic Par@@on Par@@on Topic Par@@on Zookeeper KaLa Zookeeper ZK ZK
- 10. BackType nathanmarz Storm BackType TwiXer Storm TwiXer TwiXer Apache ' Feature Storm —— a distributed real-7me computa7on system source from TwiGer Highly scalable Fault-tolerant Guarantees processing Language agnosPc * Topology of one Spout * Topology of three Spout
- 11. * UI / Nimbus / ZK / Supervisor Storm Storm cluster organiza7on structure Storm UI ZookeeperStorm Nimbus Storm Supervisor Topology
- 12. Storm Storm mul7 language support + Python PHP JS Shell , Storm use STDIN & STDOUT support multi language
- 13. PHP Connect Pool Storm PHP Connect Storm to PHP + PHP STDIN STDOUT + PHP PHP-FPM PHP PHP FastCGI
- 15. PHP PHP parse a real 7me log * -> Kafka -> Storm -> Spout Bolt -> Bolt FCGI
- 16. PHP PHP program for parse log PHP - PHP-FPM dynamic PHP pm.max_children PHP DB Cache Storm web RD PHP Input PHP PHP
- 17. Storm Bolt PHP Storm bolt send a request by FastCgi protocol
- 20. Performance data
- 21. CaseShow Case %
- 24. Real-7me log analyse service
- 25. Q & A ?