Turbocharging Drupal Syndication with Node.js
4 likes2,405 views
The document discusses optimizing Drupal content delivery using Node.js, focusing on its integration with a video content management system for a Finnish TV broadcaster. It addresses challenges faced with Drupal's performance and storage limitations and proposes a solution of using Node.js and MongoDB to improve speed and efficiency in content syndication. The author, Kalle Varisvirta, outlines technical prerequisites, the architecture of the system, and future enhancements to be made in the platform.
Turbocharging Drupal Syndication with Node.js
- 1. Turbocharging Drupal syndication with Node.JS Kalle Varisvirta Technology Director
- 2. Pre-requisites Moderate understanding of PHP and Javascript Moderate understanding of Drupal’s internals Some idea of how MySQL, NoSQL, SOLR and Varnish work
- 3. What you’ll learn How to speed up Drupal content delivery with Node.JS in real life Why to speed up Drupal content delivery with Node.JS How does Node.JS scale compared to Drupal/PHP
- 4. About me Kalle Varisvirta Technology Director at Exove Exove is a technology company based in Finland, Estonia and UK with over 70 professionals
- 5. The project The customer is Nelonen, a Finnish TV broadcaster owned by Sanoma The platform is used by their subscription video and catchup video on-demand service
- 7. The architecture A video content management system It gets fed by linear TV programming as well as uploaded content It’s supposed to deliver all content to downstream clients, multiple websites, mobile and TV apps and other CMSs
- 8. The architecture Of videos, the system only handles metadata Any uploaded videos are moved away from the CMS, to a separate stream controlled by the CMS Videos are streamed from multiple streaming locations as directed by the CMS
- 9. Video content management system (Drupal 7) Linear television data (ERP) Drupal 7 site Wordpress site
- 10. Drupal Focusing solely in content management, not delivering any web pages outside the admin There are custom modules for Integrating to linear TV ERP system Controlling the video binary management system Marking videos ready to display
- 11. Video content management system (Drupal 7) Linear television data (ERP) Drupal 7 site Wordpress site
- 12. Video content management system (Drupal 7) Linear television data (ERP) Drupal 7 site Drupal 7 site iOS app Wordpress site Samsung SmartTV app Android app
- 13. Drupal optimizations It’s not like we didn’t think of the performance early on The Drupal 7 was built on MongoDB field storage and thus standing on a fast database The Views feeds (JSON) were coming from SOLR backend
- 14. Drupal optimizations MongoDB field storage Is faster Isn’t compatible with Views, unless used with Entity Field Queries EFQ’s didn’t originally work well with MongoDB, e.g. booleans just didn’t work Eventually, isn’t worth the trouble
- 15. Syndicating content Big feeds are always too slow if coming from Drupal Storage doesn’t make a difference (MySQL, MongoDB, SOLR) Field API is extensible and flexible and thus - slow
- 16. Fields API 40 fields per item, 1000 items in feed page 40 000 calls to every hook per page load
- 17. Hey, can’t you just cache it?
- 18. Caching feeds Downstream clients want integration feeds limited by time Time attribute by seconds of the last fetch To deal with existing content To deal with changed content Caching just doesn’t cut it well
- 19. Indexing outside
- 20. We need a new approach Indexing outside of Drupal felt like the only way out We decided we’ll go with SOLR via ApacheSOLR integration Distribution by a simple REST API
- 21. We need a new approach Due to very frequent updates in popularity data and the need the order by popularity, SOLR indexing was found too slow All data was moved to MongoDB except full-text search backend
- 22. Video content management system (Drupal 7) Linear television data (ERP) Drupal 7 site Drupal 7 site MongoDB, SOLR and Node.JS REST API iOS app Wordpress site Samsung SmartTV app Android app
- 23. Video content management system (Drupal 7) Linear television data (ERP) Drupal 7 site Drupal 7 site Indexing is done using a Drupal module, MongoDB indexer MongoDB, SOLR and Node.JS REST API iOS app Wordpress site Samsung SmartTV app Android app
- 24. MongoDB indexer MongoDB indexer uses a straight connection to the MongoDB We have an indexing API on the roadmap, but currently don’t have a need for it MongoDB indexer also de-normalizes the data for optimized distribution by the Node.JS REST API It’s a contributed module, currently waiting approval
- 26. Delivery We have optimized data in the MongoDB We want to deliver it out, fast We want to be able to do some logic while delivering it
- 27. Node.JS
- 28. Node.JS Node.JS is Javascript running on the server The Javascript is ran by Chrome’s Javascript engine, the V8 It’s non-blocking, event-based and when used correctly, blazing fast
- 29. RPS Avg response time Drupal Node.JS
- 30. Node.JS No Node.JS framework used No fronting Nginx used, requests are passed from the main program to separate sub-programs manually Running on three nodes, sharing a MongoDB replica set and fronted by a F5 load balancer
- 31. Node.JS The code is very simple as it’s mostly just passing information out from the MongoDB There are filters in the API that are validated and used for filtering, other than that, very little processing is done The backend has also some special services
- 32. API
- 33. Node.JS Cluster (npm module) Forever (npm module)
- 34. Node.JS For a PHP programmer, it’s quite a change Due to the asynchronous nature, parallel programming understanding is needed There are npm modules there to help During the lifetime of the project (18 months) both Node.JS and MongoDB have evolved quite a bit
- 35. What’s next?
- 36. Node.JS optimizations Separating different Node.JS services Moving MongoDB’s out from the Node.JS servers MongoDB’s are very, very IO intensive
- 37. Drupal optimizations Moving integrations outside Drupal for clarity Using a fake Drupal 8 REST API to connect the integrations to the Drupal To be able to upgrade some time soon We released out ‘fake Drupal 8 REST API’ module, it’s in sandbox pending approval