Giuliana Benedetti - Can Magento handle 1M products?
4 likes1,147 views
The document details a project involving the migration of an e-commerce platform to Magento, including the integration of management software and data handling for a growing catalog of products, evolving from 150k to 700k items. It outlines the challenges faced, such as catalog updates, reindexing issues, and server load, and describes the strategies implemented to optimize performance and manage disk space. Additionally, it discusses the planning for a potential migration to Magento 2 to improve search functionality and handle expected growth in product numbers.
Giuliana Benedetti - Can Magento handle 1M products?
- 2. About me • Project Manager @ Webformat • Magento and TYPO3 projects • Requirements analysis • Planning of development and support activities
- 3. What’s in the menu? • Huge catalog • What we did • What we are doing
- 4. Once upon a time.. • The project began as a migration from a proprietary platform to Magento 1 Community • Shoes and accessories E-commerce • We developed the integration between their management software, that was handling products anagraphic, warehouse and orders anagraphic • Integration with Amazon e Ebay
- 5. Products Database • The original products database counted around 150k products • Configurable products • On average, 10 simple products for each configurable • Virtual products
- 6. Continued Growth • In one year only we reached the amount of 700K products stored in Magento • 66k configurable products
- 7. Challenges faced Alignment between catalog and management software Updating warehouse Reindexing Generating images Server response time Backoffice operations Third parts modules integration Export of feed for Google shopping & Co. Marketplace synchronization Disk space
- 8. Updating the catalog (1/2) • Initially 150k products, this is what we planned: • Massive initial import • Frequent update during the day via webservice • When the catalog started growing, the data exchange volumes via webservice began unsustainable. The exchange procedure needed a redesign.
- 9. Updating the catalog (2/2) • Today we have700k products • Based on Magmi and CSV file exchange (product anagraphic) • Nighttime update – the DIFF • Exceptional whole catalog update • The client accepted that the new products will be published with a delay of 1 day
- 10. Warehouse update (1/2) • No warehouse fully dedicated to web • Shared with the offline shops • It’s not possible to update the warehouse nighttime only and use that stock during the day • Frequent updates
- 11. Warehouse update (2/2) • Every 15 minutes update from management software by loading the DIFF • Only stock update • Via CSV file writing directly on database (Magmi)
- 12. Reindex (1/2) • The bigger the catalog, the slower the reindex • Initially, the reidex was lauched after each update (15 min) • After a while, the reindex started being too much time demanding: the update cycle was starting when the previous update reindex cycle was still running.
- 13. Reindex (2/2) • Solution: • All the reindexes have been disabled, except for the stock reindex • All reindexes are now performed after the nighttime import • Today a full reindex takes around 75 minutes and generates a heavy load on the database
- 14. Catalog_url_rewrite (1/2) • Magento 1 has a critical point with URL rewrite process: • All product URLs are rewritten, also simple products that are «Not visible individually» and exist only to be associated to a configurable. • With 700k products catalog, this meant: • Creating millions of rows in the catalog_url_rewrite table • An URL rewrite process that takes hours to be completed
- 15. Catalog_url_rewrite (2/2) • A patch has been installed, to avoid the simple and not visible individually products url generation • Module Dnd_Patchindexurl: https://www.magentocommerce.com/magento-connect/dn-d-patch- index-url-1.html • Now the reindex process takes around 20 minutes
- 16. Images generation (1/2) • One of the main problems that we had to face was the product thumbnails generation, done by Imagemagik • Every day hundreds of products are published We verified that the frontend CPUs were often stressed because of Imagemagik process and the writing operations on database
- 17. Images generation (2/2) • We found a solution in generating the thumbnails during the massive import, so Imagemagik could work together with the import procedure • Nighttime, the images are generated and saved in a dedicated server, without interfering with user navigation • Today we have around 881K images saved
- 18. Server response time • With such a huge catalog, some categories hold even hundreds of products • The first loading time (if they are not cached) is indeed high • We activated caching on Redis and Varnish • Not enough, the first loading time was anyway too heavy
- 19. Solutions 1/2 • Moving the cache clearing process during the night • At 8 in the morning, the website navigation was starting to suffer • We planned a job to pre-cache all the critical pages • Minimized cache invalidation • Clear cache only for products for which the stock quantity was updated via WS
- 20. Solutions 2/2 • Client training to better handle the cache erasing • Minimized the number of filters in layered navigation • Each filter increases the reindex time and the pages combinations not cached
- 21. Backoffice operations • Initially all the catalog update activities were performed from Magento backoffice • Problems: • Frequent reindexes • Frequent cache updates • Server load (the backoffice product list filters are CPU demanding and they charge MySql) • Common operations were slown • Several BE users ended to be concurrent
- 22. Solutions • Initially a new backoffice server have been introduced • MySql load problem was not solved. Reindex re-caching as well. • We introduced a new process to handle the catalog, using an excel file • This improved the efficiency of who was managing the anagraphic data • Massive excel file import performed each 3 days via FTP • Categories still handled from backoffice
- 23. Third party modules integration • Critical point • Not all the modules found in the Marketplace are developed in an optimal way • They «simply» load the products collection without pagination • They execute nested query • There are cycles on collections that initialize all products unnecessarily • … • A big profiling and optimization work was needed
- 24. Feed export (Google Shopping & Co.) 1/2 • While the catalog was growing, the feed time export was encreasing as well • In the very beginning, the exports were handled by a Magento module
- 25. Feed export (Google Shopping & Co.) 2/2 • Solution steps: • The module have been replaced with ad-hoc procedures, with high level of optimization • The exportation jobs are executed on backoffice server during the night, to not load the frontend • It have been introduced a MySql slave as data source, to not load the master and the website as a consequence
- 26. Marketplace synch • We are using M2E Pro • Client side: EAN code full check • Tech side: handling the automatic synchronization process • An automatic full synchronization is too heavy. When synchronize? • What synchronize? • Magmi
- 27. Disk space (1/2) • Well, here we are: even if disk space is quite cheap, using too much of it it’s not convenient.. • Data exchange logs very heavy • Frequent data exchange and huge amount of data • Log files were growing fast • Log rotate was activated hourly • Log are archived after few days
- 28. Disk space (2/2) • High image quantity, continuously growing • Huge feed export • Huge CSV import files • … • Solutions applied: • Constant monitoring activated • Activated automatic procedures to clean log, old images, expired feed, etc.
- 29. Challenges to be faced Elasticsearch integration Growing catalog, until 1M products More sells, more page views Magento 2 migration
- 30. Elasticsearch • For two reasons: • Improve the search functionality offered to the client • Minimize the load produced by the Magento internal search engine • Critical issues to be faced: • Catalog index time • Only configurable products? • What about the sizes?
- 31. 1M products • Expected growth: in 1 year we’ll have 1M products • At the moment we are performing tests with fake products • We didn’t detect other critical aspects • At the moment, we had to develop some more data exchange and feed generation procedures optimization
- 32. More sells, more page views • Sessions are increasing the number of not cached pages views is increasing • Pre – caching extension • Increasing Varnish cache TTL • Minimize products in categories and filters used • Sales are increasing increasing also frequency of out-of stock products • To be evaluated: the impact of new reindex and re-caching politics on client
- 33. What if..? • We’re planning with the client a Magento 2 migration • We started our tests by migrating the actual Magento 1 environment (700K products) to a Magento 2 installation • We collected the results and still performing some other tests
- 34. HW specs All tests were run on a VirtualBox VM with Linux Ubuntu 16.04.1 LTS, 8 GB RAM, 1 x 2,60 GHz cpu Lamp configuration was featuring PHP version 5.6, Apache 2.4.18, MySQL 14.14 Migration was performed from Magento version 1.9.2.2 through 2.1.3
- 35. Magento 2 migration (1/4) • DB migration times: 1h 20‘ • BE performances: BE Operation Magento 1 with cache Magento 2 with cache Access to catalog almost 5' 7'' Access to product 3'' 10'' Access to categories 7'' 6'' Product searching 1'5'' 3''
- 36. Magento 2 migration (2/4) • FE performaces for catalog browsing: FE Operation Magento 1with cache Magento 2with cache Catalog browsing / categories 30'' 7''
- 37. Magento 2 migration – Reindex Times (3/4) M1 M2 Total: 2h 55‘’11’’ Total: 2h 53‘ 47’’
- 38. Magento 2 migration (4/4) • We had some issues with the Catalog Fullsearch reindex (Magento 2) • we had to apply a patch https://github.com/magento/magento2/issues/5146 • Catalog Fullsearch reindex without patch takes around 2 hours with patch applied took around 1 hour, so the times are quite comparable 02:12:37 02:12:37
- 39. Catalog URL rewrite • M1 with Dnd_Patchindexurl module: 00:14:34 • M1 without Dnd_Patchindexurl module: 01:03:50 • M2: no catalog URL rewrite. URL Rewrite is handled at the product saving
- 41. Conclusions
- 42. Yes, we can! • It’s possible, but not without effort • Large initial analysis • Special attention to optimization processes • What about Magento 2?
- 43. Q & A • Giuliana Benedetti – giuliana.benedetti@webformat.com • WEBFORMAT srl - www.webformat.com