Building a scalable online backup system in python
Download as PPTX, PDF4 likes1,377 views
This document discusses building a scalable online backup system in Python. It describes the architecture including using a load balancer, app servers to handle uploads and deliver web pages, and task servers to do heavy lifting like file processing. It discusses scaling the system to upload over 100k files in a day by having clients upload in chunks and using technologies like Django, PostgreSQL, Memcached, Amazon S3 storage and EC2 servers. It also reflects on lessons learned including that Python was flexible but PostgreSQL could be replaced with MySQL and a native Windows client may not have been needed.
Building a scalable online backup system in python
- 1. Building a Scalable Online Backup System in PythonJoe Drumgoolehttp://twitter/jdrumgoole
- 2. ScalingYou probably shouldn’t careThroughput vs response timeScaling is a fractal problemThe database is what will get ya!Amazing what a well tuned DB will supporthttp://twitter.com/jdrumgoole2
- 4. Online BackupNot really a Web 2.0 playMore like client serverLarger vision of PutPlaceMap of your Digital Worldhttp://twitter.com/jdrumgoole4
- 5. Online Backup : ClientInstallation and support of Windows 20**Mac SupportOpen file/locked file handlingBandwidth throttlingCPU ThrottlingUpload restartsFeedbackhttp://twitter.com/jdrumgoole5
- 6. Online Backup : ServerDon’t loose any filesDe-duplicationThumbnail generation for imagesFlickr BackupClient FeedbackBulk downloadFile relationshipshttp://twitter.com/jdrumgoole6
- 7. Online Backup - SecretsPeople Don’t backupCompute dominatesRestores represent 0.01% of bandwidth and loadWriting web clones of Windows Explorer is hardThe browser sucks as a client side app container (for now)http://twitter.com/jdrumgoole7
- 8. ScalingFor online backup the challenge is to receive shed loads of data from lots of clientsClients upload in 1MB chunksChunks must be stored coalesced and push to stable backup (S3)Clients must get acknowledgementWeb page must updateQuota managementhttp://twitter.com/jdrumgoole8
- 9. Load BalancerLoad Balancer : Perlbalhttp://www.danga.com/perlbal/Can handle 100 x millions of requests per dayEvent based(sshhh : Don’t tell anyone, but its Perl!)It does fall over occasionallyOtherwise works perfectlyhttp://twitter.com/jdrumgoole9
- 10. App ServerOur app servers:Handle loginDeliver web pagesHandle uploads from clientsHand off heavy duty processing to task serversThumbnail generationFile coalescingChecksum generationHand off is via a database queuehttp://twitter.com/jdrumgoole10
- 11. App ServerJust Django InstancesTemplates deliver web pagesViews handle chunks/login etc.Models update the databaseTask Servers do the heavy liftinghttp://twitter.com/jdrumgoole11
- 12. Task ServerRun off a database queue (table)Four main task servers:Assemble completed file uploadsCreate thumbnailsRemove deleted filesGenerate user statisticsServers are multi-threadedhttp://twitter.com/jdrumgoole12
- 13. RefactoringOriginally N blacknight servers writing to NFSThen N blacknight servers writing to S3Then N EC2 servers writing to S3The N EC2 servers writing to MogileFS/S3Lots of uploading optimisations along the wayhttp://twitter.com/jdrumgoole13
- 14. ResultsSystem has successfully uploaded over 100k files in a single dayRegularily does 50k files a dayHave about 2k registered usersContinues to get registrationsRuns in lights out mode (no daily/weekly/monthly housekeeping)http://twitter.com/jdrumgoole14
- 15. What workedPython proved extremely flexibleStandard library saved us lots of workDjango provided a lot of glueEasy to migrate from dedicated host on NFS to Cloud Hosting and S3 storageNagios/Monitis monitoringhttp://twitter.com/jdrumgoole15
- 16. What Didn’t WorkWould use MySQL rather than PostgresEasier to cluster, more knowledge availableNative Windows ClientUnecessary, Python client was good enoughWould use an off the shelf queueing systemRabbitMQ, ActiveMQ, SQSKludgey client side APIThreading The Clienthttp://twitter.com/jdrumgoole16
- 17. Tool ChainWush.net : Subversion and TracDynDNS: Dynamic DNSPython/Django: Dev StackPostgres: DatabaseHudson : Build ServerPerlbal: Load BalancingMogileFS : Distributed File SystemMemcached : CachingNagios, Monitis: MonitoringHamachi : VPN through FirewallGoogle Apps : Email, Calendar, Docs, WikiAuthSMTP : Validated SMTPZendesk: Support DeskAmazon : Storage, Compute, BandwidthPaypal : Billinghttp://twitter.com/jdrumgoole17
- 18. CostsCapital ExpenditureOne server 5k euroOne laptop per developer 2.5k (7 devs)One Linksys WIFI/Firewall (won at Raffle)Two 24 port switches 1.6kTotal: ~24kRunning Costs for Grid and Storage~1800 euro a month (8 instances)http://twitter.com/jdrumgoole18
- 19. If I Were Doing it AgainStick with native python clientLook at eventing ala Node.js for serverUse MySQLUse Google App Engine as Front End/Load BalancerUse a commercial queueing packagehttp://twitter.com/jdrumgoole19