Khanh-Nguyen - Gearman - distributed process solution
12 likes10,885 views
The document presents a Gearman distributed processing solution, highlighting its capabilities such as multi-language API support and high job throughput. It outlines installation steps, use cases like asynchronous queues and scatter/gather processing, and provides examples of how Gearman can be utilized for tasks like email sending and data crawling. The author, a student and part-timer, concludes by inviting engagement from the audience and providing contact information.
![Installation
1.
2.
Download source code [ https://launchpad.net/gearmand ].
Compile and install from tarball.
tar xzf gearmand-X.Y.tar.gz
cd gearmand-X.Y
./configure
make
make install
3.
Starting gearman.
gearman -d](https://image.slidesharecdn.com/khanhhnguyen-gearman-distributedprocesssolution-131017035003-phpapp01/85/Khanh-Nguyen-Gearman-distributed-process-solution-8-320.jpg)
![Hello world !!!
●
Python and C/C++ and a good choices for workers.
●
PHP, Node.js should be client than worker. [ PHP Worker often crash after
1-2 hours because memory leak ]](https://image.slidesharecdn.com/khanhhnguyen-gearman-distributedprocesssolution-131017035003-phpapp01/85/Khanh-Nguyen-Gearman-distributed-process-solution-25-320.jpg)
![Gearman Admin GUI
●
Command Line Tool : http://gearman.info/bin/gearadmin.html
●
Gearman-Monitor [ PHP ] : https://github.com/yugene/Gearman-Monitor
●
GearmanUI [ PHP ] : http://rripado.info/gearmanui/](https://image.slidesharecdn.com/khanhhnguyen-gearman-distributedprocesssolution-131017035003-phpapp01/85/Khanh-Nguyen-Gearman-distributed-process-solution-36-320.jpg)
Khanh-Nguyen - Gearman - distributed process solution
- 1. Gearman Distributed Process Solution Created by Google Drive Presentation.
- 2. Introduction $ whoami ######################################################## #### Name : Nguyen Duy Khanh #### #### Job : Student of HCMUS, Part-timer at PNC #### #### My skills : PHP, MySQL, JS #### #### Love : New technology #### ######################################################## $|
- 3. Problems HEAVY WORKLOAD function main() { ... doSomeThing(); doSomeThingBig(); doSomeThingBigBig(); doSomeThingVeryBig(); … return result; }
- 6. Solution Give me a job, my workers will do it for you. Worker 1 Worker 2 Worker... Worker n
- 7. Gearman ❏ GEARMAN → MANAGER ❏ Open Source. ❏ Multi-language API. ❏ Multi-threaded ( 50k jobs / second ). ❏ Ultra fast ( written in C/C++ ). ❏ No limits on message size. ❏ Failover
- 8. Installation 1. 2. Download source code [ https://launchpad.net/gearmand ]. Compile and install from tarball. tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install 3. Starting gearman. gearman -d
- 10. How does it work ? Client Hey, I can send email. Let me do it !!! Worker Python Worker Node.JS Job Manager Worker PHP
- 11. How does it work ? Client Worker Python (send_email) Worker Node.JS Job Manager Worker PHP
- 12. How does it work ? Client And I can push notification. Let me try !!! Worker Python Worker Node.JS Job Manager Worker PHP
- 13. How does it work ? Client Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP
- 14. How does it work ? Client Haha, I can do them all !!! Worker Python Worker Node.JS Job Manager Worker PHP
- 15. How does it work ? Client Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
- 16. How does it work ? Client I need send an email to abc@xyz.com , please help me ! Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
- 17. How does it work ? Client OK, Let me see. Aaah … We have 2 workers can do it for you. But, Worker Python is busy, let Worker PHP do it. Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
- 18. How does it work ? Client Hey Worker PHP, please send an email to abc@xyz.com Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
- 19. How does it work ? Client OK Boss, email has sent to abc@xyz.com Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
- 20. How does it work ? Client Hi client, your email has sent to abc@xyz.com. Worker Python (send_email) Worker Node.JS (push_notify) Job Manager Worker PHP (send_email, push_notify)
- 21. Hello world !!! Install Node.js API $ npm install node-gearman Install PHP API $ pecl install gearman Install Python API $ pip install gearman or $ easy_install gearman
- 25. Hello world !!! ● Python and C/C++ and a good choices for workers. ● PHP, Node.js should be client than worker. [ PHP Worker often crash after 1-2 hours because memory leak ]
- 26. Use-cases ❏ Scatter/Gather ❏ Map - Reduce ❏ Asynchronous Queues ❏ Pipeline Processing
- 27. Scatter / Gather ● Perform a number of task concurrently. ○ Ex : Db Query, Image Processing, Location Search, Full-text Search… ● Take advantage of many tier. ● Speed up your web applications. ● Tasks don’t need to be related.
- 28. Scatter / Gather Db Query Client Resize Image Search
- 29. Map / Reduce
- 30. Asynchronous Queues ● Most popular use-case. ● Do something needn’t immediate processing. ○ Ex : send emails, push notification, index data, crawling, ... ● Allows for batch operations.
- 31. Asynchronous Queues function tweet(msg) { // Insert to database db.insert(msg); // Background tasks gearman.push_notify(); // Return return true; }
- 32. Pipeline Processing ● Do related thing. ○ Ex : search engine, data analytic, ... ● Chain workers to send data to next step.
- 33. Pipeline Processing Main Client Worker Step 1 Client Worker Step 2 Client Worker Last Step Output
- 34. Examples GEARMAN Auto News Crawling System Reload Request MongoDB ● Manager : GEARMAN ● Workers “parse_rss” Gearman Client Webserver : Node.JS No Workers : Python ● RSS Sources ● Web Framework : Express ● Database : MongoDB ● Client Script : jQuery Have new items ? Crawl new items Workers “crawling” End
- 35. Persistent Queues ❏ By default, jobs are only stored in memory. ❏ Various contributions from community : ❏ MySQL / Drizzle ❏ PostgreSQL ❏ SQLite ❏ Tokyo Cabinet ❏ memcached ❏ Hope MongoDB (or an NOSQL DB) coming soon.
- 36. Gearman Admin GUI ● Command Line Tool : http://gearman.info/bin/gearadmin.html ● Gearman-Monitor [ PHP ] : https://github.com/yugene/Gearman-Monitor ● GearmanUI [ PHP ] : http://rripado.info/gearmanui/
- 39. About Us Author : Khanh Nguyen Duy Find me at : admin@ndksolution.com Presentation made for “Javascript HoChiMinh City Meetup” You can find us at : ● http://meetup.com/JavaScript-Ho-Chi-Minh-City/ ● https://www.facebook.com/JavaScriptHCMC ● https://plus.google.com/u/0/communities/116105314977285194967