Slideshare - Magento Imagine - Do You Queue
Download as PPTX, PDF0 likes510 views
The document discusses using queues to improve the scalability of PHP applications. It describes how queues allow asynchronous and distributed processing of tasks to improve performance and allow applications to handle more traffic. Specifically, it promotes using Zend Server's job queue to offload long-running tasks like payments processing so the frontend can scale independently of backend processing. Examples show building jobs that communicate with the queue to asynchronously execute tasks like payments.
Slideshare - Magento Imagine - Do You Queue
- 1. Do you queue? Characteristics of scalabilityKevin SchroederTechnology EvangelistZend Technologies
- 2. About me Past: Programming/Sys Admin Current: Technology Evangelist/Author/Composer @kpschrade
- 3. I blog at eschrade.comFollow us!(good things will happen to you)Zend Technologieshttp://twitter.com/zendhttp://twitter.com/kpschrade (me!)
- 4. Twtpoll resultsCould your PHP apps benefit from being able to process data or execute asynchronously?
- 5. Why would you want to queue?PerformanceExecute logic apart from the main request (asynchronicity)ScalabilityThe ability to handle non-immediate logic as resources are available
- 6. Typical anatomy of a PHP ApplicationBad for scala-bility!PresentationApplication ControlDatabase AccessBusiness LogicPresentationApplication ControlBusiness LogicPresentation| 6
- 7. Good for ScalabilityPresentationSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlApplication ControlSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpBusiness LogicSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpDatabase AccessSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.php| 7 | 8-Feb-11
- 8. What helps make software scalable?Defined tasksLoose couplingResource discovery
- 9. The Golden Rule of Scalability“It can probably wait”
- 10. Asynchronous execution usesPre-caching dataData analysisData processingPre-calculating (preparing data for the next request)Data is “out of date” once it leaves the web serverImmediacy is seldom necessary
- 11. CharacteristicsA sledgehammer can hit a machineScalability and High Availability are yin and yangA site that can’t keep running is not scalableA site that can’t scale will fail (if it gets really popular)Machines can be added and removed at willNot “cloudy” necessarilyNo single point of failureData exists in at least two, preferably at least three, places
- 12. ConsiderationsWaste disk spaceControl usage (don’t let users do anything they want)Pre-calculate as much as possibleCalculate and cache/storeKeep data processing off the front end serversDon’t just cacheDon’t let it substitute for thoughtCache hit rates can be meaningless if you have hundreds of cache hits for a request
- 13. ConsiderationsBuild a deployment mechanism with NO hardcoded values like directory or resource locations
- 14. Make as much as possible configurable/discoverableDecouple/PartitionDon’t tie everything (relationships and such) into the databaseUse queues/messagingStomp interfaces are really good for PHP – Can also use Java BridgeZend_Queue has several interfacesTry to use stateless interfaces(polling is more scalable than idle connections)
- 15. OptionsUse Cron /w PHP CLI (not the best idea)People mostly use cron simply due to availabilityIt’s an available option, not necessarily the best option.Use GearmanUse home-grown (don’t do this)Use pcntl_fork() (NEVER do this)Use Zend Server Job Queue
- 16. Your only real optionsVery cloud friendly* I am not an expert on Gearman. Corrections will be taken in the spirit that they are given.For obvious reasons, I will focus on Zend Server
- 17. Zend FrameworkZend ServerPHPApplicationPerformanceReliability &ManagementScale-OutBusiness-gradePHPAccelerationMonitoringClusteringHot FixesOptimizationRoot-CauseJob QueueSupportCachingDownloadsJava BridgeConfigurationLinux(rpm/web repositories)IBM i(PTF)Windows(MSI) What the heck is Zend Server?16Zend Server
- 18. Job Queue Architecture – Elastic BackendWeb Server /w JQWeb ServerUsers!Web Server /w JQWeb ServerLoad BalancerWeb Server /w JQWeb ServerProsScale the backend as necessaryDefault (easy) mechanismConsGetting the job status requires using a DB
- 19. Job Queue Architecture – Elastic FrontendWeb ServerWeb Server /w JQUsers!Web ServerWeb Server /w JQLoad BalancerWeb ServerWeb Server /w JQProsEasy to communicate with the Job Queue server handling the jobConsRequires you to build your own interface (or just use mine!)
- 20. Kevin’s favorite way to implement itCreate a task-handling controllerCreate an abstract task classUnderstands the Job QueueSelf containedIf Elastic Backend: connects to localhostIf Elastic Frontend: connects to load balancer (my preferred), load balanced JQ servers manage themselvesExecute the task, have it serialize itself and send it to send to the task handler
- 21. Classes involved in the demoManagerHandles connecting to the queue and passing results back and forthJobAbstractAbstract class that a job would be based off ofResponseThe response from the manager when a job is queued. Contains the server name and job numberGetRemoteLinksScans a given web page and reports back a list of all the links on the page
- 22. Execution FlowCreate job and set dataExecute jobJob passes itself to the queue managerManager serializes jobManager uses HTTP call through a load balancer to queue the jobThe queue on the other end returns the job id and server nameJob ID and server name is passed to the clientClient polls the manager to get a completed jobWhen the job is returned pass the serialized version of the executed job
- 23. Demo
- 24. Neato, but how in the world does this pertain to Magento?
- 25. How?Long running front-end requests kill front-end responsiveness which kills the user experience40% will wait no more than 4 seconds for a website (Forrester)Calculating cache data on-the-fly can lead to bad user experiences (abruptly slow page load times) and consistency problemsLong running page requests can push up against max_execution_timeFor example, handling payments from a slow CC APISimply setting it to run longer is a band-aid on a broken bone
- 26. Solution?Build Zend Server integration extensionsZendserver_JobqueueZendserver_MonitorUtilize the event system to intercept actionsAsync_Payment (for payments, in this case)Profit! (scalability AND performance, actually)
- 27. Building a jobInstall the Zend Server Magento extensionCreate a class that extends Zendserver_Jobqueue_JobAbstractCall $job->execute();
- 28. Payment as an exampleHooks an observer into controller_action_predispatch_checkout_onepage_saveOrder, core_block_abstract_to_html_afterChecks if a defined template is being renderedInjects JavaScript that changes the review.save() methodInjects a new URL for submitting the transaction which creates a jobPasses POSTed data to a jobJob executes by submitting the payment URL on behalf of the browser, storing the resultBrowser pings the server to see if the job has completed executing yet
- 29. Scaling the solutionWebserverJob Queue/ WebserverWebserverLoad BalancerJob Queue/ WebserverWebserverJob Queue/ Webserver* Duplicate deployments on ALL machinesDatabase or Session Clustering
- 30. DownloadsZend Serverhttp://www.zend.com/serverZendserver_Jobqueuehttps://github.com/kschroeder/Magento-ZendServer-JobQueueZendserver_Monitorhttps://github.com/kschroeder/Magento-ZendServer-MonitorAsync_Paymenthttps://github.com/kschroeder/Magento-Async-PaymentJob Queue library & demohttps://github.com/kschroeder/ZendServer-JobQueue-Job-APIhttp://bit.ly/hWJBYw
- 31. Magento/Zend Case Study - http://bit.ly/horpFFFollow us!Zend Technologieshttp://twitter.com/zendhttp://twitter.com/kpschrade (me!)
- 32. Get this information and all the examples at eschrade.com…
Editor's Notes
- #17: We released Zend Server and Zend Server Community Edition in April this year Products written from scratch based on our very extensive experience with Platform technologies Full integrated stack, native installer, ZF and Studio integration, software updates, all new UI, … Both editions have been very well received by users (love performance boost, ease of use, deployment) Great fit to our partners – we’re working with Varien/Magento, KnowledgeTree, MCS, … Next step will round up web app server offering to support high availability and scalability – more on this in the next few months