Scaling symfony apps
3 likes4,632 views
This document discusses scaling Symfony applications. It begins by introducing the speaker and their experience scaling large applications. It then covers scaling different aspects of an application including the web server, sessions, database, and filesystem. For each area, it provides recommendations such as using PHP opcode caching, storing sessions in Redis or Memcached, considering database sharding for very large databases, and using an abstraction layer like FlysystemBundle to store files in cloud storage like Amazon S3. The overall message is that scaling can be achieved through configuration changes and decoupling services rather than code changes.
Scaling symfony apps
- 1. Rome, 28 October 2016 Scaling Symfony apps
- 2. WHO AM I?
- 4. CTO @ website: madisoft.it tech blog: labs.madisoft.it
- 5. Scalability
- 6. It’s from experience. There are no lessons.
- 7. Nuvola ● > 3M HTTP requests / day ● ~ 1000 databases ● ~ 350GB mysql data ● ~ 180M query / day ● ~ 25M of media files ● ~ 4.50TB of medis files ● From ~5k to ~200k sessions in 5 minutes
- 8. Scalability Your app is scalable if it can adapt to support an increasing amount of data or a growing number of users.
- 9. “But… I don’t have an increasing load” (http://www.freepik.com/free-photos-vectors/smile - Smile vector designed by Freepik)
- 10. “Scalability doesn’t matter to you.” (http://www.freepik.com/free-photos-vectors/smile - Smile vector designed by Freepik)
- 12. “I do have an increasing load” (http://www.freepik.com/free-photos-vectors/smile - Smile vector designed by Freepik)
- 13. Your app is growing
- 14. But… suddenly…
- 19. Ok, we need to scale
- 21. Everything?
- 24. Can PHP scale?
- 25. Scaling is about app architecture
- 26. App architecture How can you scale your web server if you put everything inside? Database, user files, sessions, ...
- 27. App architecture / Decouple ● Decouple services ● Service: do one thing and do it well
- 28. App architecture / Decouple
- 29. 4 main areas 1. web server 2. sessions 3. database 4. filesystem There are some more (http caching, frontend, queue systems, etc): next talk!
- 30. Web server Many small webservers (scale up vs scale out)
- 31. Web server NGINX + php-fpm PHP 7 Symfony 3
- 32. Web server / Cache PHP CACHE SYMFONY CACHE DOCTRINE CACHE
- 33. Web server / PHP Cache Opcache Bytecode caching opcache.enable = On opcache.validate_timestamps = 0 https://tideways.io/profiler/blog/fine-tune-your-opcache-configuration-to-avoid-caching-suprises
- 34. PHP code / Symfony cache ● Put Symfony cache in ram ● Use cache warmers during deploy releaseN/var/cache -> /var/www/project/cache/releaseN “/etc/fstab” tmpfs /var/www/project/cache tmpfs size=512m
- 35. PHP code / Doctrine cache ● Configure Doctrine to use cache ● Disable Doctrine logging and profiling on prod doctrine.orm.default_metadata_cache: type: apc doctrine.orm.default_query_cache: type: apc doctrine.orm.default_result_cache: type: apc
- 36. PHP code / Cache DISK I/O ~ 0%
- 38. PHP code / Profiling XHProf Blackfire New Relic
- 41. PHP code / Recap ● Easy ● No need to change your PHP code ● It’s most configuration and tuning ● You can do one by one and measure how it affects performance ● Need to monitor and profile: New Relic for PHP ● Don’t waste time on micro-optimization Take away: use cache!
- 42. Sessions ● Think session management as a service ● Use centralized Memcached or Redis (Ec2 or ElasticCache on AWS) ● Avoid sticky sessions (load balancer set up)
- 43. Session / Memcached No bundle required https://labs.madisoft.it/scaling-symfony-sessions-with-memcached/
- 45. Session / Redis config.yml framework: session: handler_id: snc_redis.session.handler
- 46. Session / Redis Bundle config snc_redis: clients: session_client: dsn: '%redis_dsn_session%' logging: false # https://github.com/snc/SncRedisBundle/issues/161 type: phpredis session: client: session_client locking: false prefix: session_prefix_ ttl: '%session_ttl%'
- 47. Session / Redis parameters.yml redis_db: 3 redis_dsn_session: 'redis://%redis_ip%/%redis_db%' redis_ip: redis-cluster.my-lan.com session_ttl: 86400
- 48. Session / Recap ● Very easy ● No need to change your PHP code ● Redis better than Memcached: it has persistence and many other features ● Let AWS scale for you and deal with failover and sysadmin stuff Take away: use Redis
- 52. Database If you need data integrity do not replace your SQL db with NOSQL to scale
- 53. Database How to scale SQL db?
- 55. Database If dbsize < 10 GB dont_worry();
- 56. Database / Big db problems ● Very slow backup. High lock time ● If mysql crashes, restart takes time ● It takes time to download and restore in dev ● You need expensive hardware (mostly RAM)
- 57. Database / Short-term solutions Use a managed db service like AWS RDS ● It scales for you ● It handles failover and backup for you But: ● It’s expensive for big db ● Problems are only mitigated but they are still there
- 58. Database / Long-term solutions Sharding
- 59. Database / Sharding Split a single big db into many small dbs (multi-tenant)
- 60. Database / Sharding ● Very fast backuo. Low lock time ● If mysql crashes, restart takes little time ● Fast to download and restore in dev ● No need of expensive hardware ● You arrange your dbs on many machines
- 61. Database / Sharding ● How can Symfony deal with them? ● How to execute a cli command on one of them? ● How to apply a migration (ie: add column) to 1000 dbs? ● …...
- 62. Database / Sharding Doctrine DBAL & ORM
- 63. Database / Sharding Define a DBAL connection and a ORM entity manager for each db https://symfony.com/doc/current/doctrine/multiple_entity_managers.html
- 64. Database / Sharding doctrine: orm: entity_managers: global: connection: global shard1: connection: shard1 shard2: connection: shard2 doctrine: dbal: connections: global: ….. shard1: …… shard2: …... default_connection: global
- 65. Database / Sharding This works for few dbs (~ <5)
- 66. Database / Sharding Doctrine sharding http://docs.doctrine-project.org/projects/doctrine-dbal/en/latest/reference/sharding.html
- 67. Database / Doctrine sharding ● Suited for multi-tenant applications ● Global database to store shared data (ie: user data) ● Need to use uuid http://docs.doctrine-project.org/projects/doctrine-dbal/en/latest/reference/sharding.html
- 68. Database / Sharding Configuration doctrine: dbal: default_connection: global connections: default: shard_choser_service: vendor.app.shard_choser shards: shard1: id: 1 host / user / dbname shard2: id: 2 host / user / dbname
- 69. Database / Sharding ShardManager Interface $shardManager = new PoolingShardManager(); $currentCustomerId = 3; $shardManager->selectShard($currentCustomerId); // all queries after this call hit the shard where customer // with id 3 is on $shardManager->selectGlobal(); // the global db is selected
- 70. Database / Sharding ● It works but it’s complex to be managed ● No documentation everywhere ● Need to manage shard configuration: adding a new shard? ● Need to parallelize shard migrations: Gearman? ● Deal with sharding in test environment
- 71. Database / Recap ● NOSQL is not used to scale SQL: they have different purposes. You can use both. ● Sharding is difficult to implement ● Need to change your code ● Short-term solution is to use AWS to leverage some maintenance ● Doctrine ORM sharding works well but you need to write code and wrappers. Best suited for multi-tenant apps ● When it’s done, you can scale without any limit Take away: do sharding if your REALLY need it
- 72. Filesystem Users upload files: documents, media, etc How to handle them?
- 73. Filesystem ● Need of filesystem abstraction ● Use external object storage like S3 ● Avoid using NAS: it’s tricky to be set-up correctly
- 74. Filesystem / Abstraction ● FlysystemBundle ● KnpGaufretteBundle https://github.com/1up-lab/OneupFlysystemBundle
- 75. Filesystem / Abstraction https://github.com/1up-lab/OneupFlysystemBundle ● AWS S3 ● Dropbox ● FTP ● Local filesystem ● ...
- 76. Filesystem / Abstraction Configuration oneup_flysystem: adapters: s3_adapter: awss3v3: client: s3_client bucket: "%s3_bucket%" oneup_flysystem: adapters: local_adapater: local: directory: ‘myLocalDir’
- 77. Filesystem / Abstraction Configuration prod.yml oneup_flysystem: filesystems: my_filesystem: adapter: s3_adapter dev.yml oneup_flysystem: filesystems: my_filesystem: adapter: local_adapter
- 78. Filesystem / Abstraction Usage // LeagueFlysystemFilesystemInterface $filesystem = $container->get(‘oneup_flysystem.my_filesystem’); $path = ‘myFilePath’; $filesystem->has($path); $filesystem->read($path); $filesystem->write($path, $contents);
- 79. Filesystem / Recap ● Easy ● Need to change your PHP code ● Ready-made bundles ● Avoid local filesystem and NAS Take away: use FlystemBundle with S3
- 80. Scaling / Recap ● Sessions and filesystem: easy. Do it ● PHP code: not difficult. Think of it. Save money. ● Database: very hard. Think a lot ● Queue systems, http cache and some other stuff: next talk but think of them as services
- 81. THANK YOU
- 82. WE ARE HIRING! (wanna join? ask us or visit our website)
- 83. QUESTIONS?