Move out from AppEngine, and Python PaaS alternatives
Download as KEY, PDF14 likes3,359 views
This document discusses moving a podcast hosting application called MyAudioCast off of Google App Engine (GAE) and onto other Python platforms as a result of high costs and limitations. Some key points: - MyAudioCast was running on GAE for over a year but costs were rising to $120/month due to high storage, bandwidth, and processing usage. - Performance on GAE was poor with high error rates for operations like inserting logs and updating counters. - Development was slowed by GAE limitations like long deployment times and inability to easily use common Python packages. - The author chose to migrate MyAudioCast to the Linode VPS and Heroku PaaS for better pricing,
Move out from AppEngine, and Python PaaS alternatives
- 1. Move out from AppEngine and Python PaaS
- 2. 小海 StreetVoice / CTO twitter.com / tzangms github.com / tzangms
- 5. Pros you don’t worry about the server enviroment
- 6. Cons you worry about money
- 7. MyAudioCast running on app engine over One year
- 8. Just a simple website that host Podcasts
- 10. What it use • Google AppEngine • Django 1.1 • S3 • CDN - CDNetworks (sponsored by pixnet)
- 11. Current stats • 600+ podcasts • 900+ episodes • 220GB storage usage / month • 60TB bandwidth usage / month • 6000,000 plays / month
- 12. Audience 80% • China: 80% • Taiwan: 3% • Other: 17% 16% 4% China Taiwan Other
- 13. Page views
- 15. Why I moved out from AppEngine • Pricing • Performance • Deployment • Reinvent the wheel
- 16. Pricing
- 17. Expensive
- 18. $4 a day? $120/month, you can rent Linode 1024 for 3
- 19. One Linode 1024 for now
- 20. One Linode 1024 Cost you $39.95 you save $80
- 21. Performance
- 23. High error rate
- 24. What problem? • INSERT logs, 2000+ ops • Counter (20 shards), 8000+ ops
- 25. 1000+ errors per day
- 26. For now • INSERT logs, 35000+ ops • Counter (no shards), 60000+ ops
- 27. 0 errors
- 28. Slow Deployment
- 29. After click deploy button
- 30. I have to wait for 60 seconds
- 31. and 60 seconds
- 32. more 60 seconds
- 33. zzz ...
- 36. I know it’s better now
- 38. Slow development
- 39. `pip` is useless
- 40. You can not do lots of things in easier way
- 41. For example
- 42. Shard Counter?
- 43. Why waste your time?
- 44. Just use Redis it’s damn easy and stable
- 45. Cache add cache / invalidation
- 46. Why waste your time?
- 47. use `jonny-cache` or `cache-machine` just add a few lines of code
- 48. It's a lot more easier for you to run on Linode or other PaaS.
- 49. How to move out and keep your data.
- 50. Dump and Prepare your data
- 51. Generate bulkloader.yaml appcfg.py create_bulkloader_config --filename=bulkloader.yaml myaudiocast
- 52. Dump as csv appcfg.py download_data --config_file=bulkloader.yaml --filename=users.csv --kind=User myaudiocast
- 53. write a script to read CSV and insert your data into database
- 54. The hardest problem Foreign Key / relation
- 55. Foreign Key with parent User Podcast Episode url tzangms 1 1001 /tzangms/1001/ lomeer 1 1001 /lomeer/1001/ william 1 1001 /william/1001/
- 56. What? All the key id of Podcast kind is 1
- 57. Some data can’t be dumped
- 58. Write it down before you dump your data.
- 59. How about Auth? OMG, I don’t have password for user!
- 60. Implement Google OAuth2
- 61. The problem
- 62. Poor document I can’t find `scope` at that time
- 63. You need this. weird scope https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
- 64. After almost half year I still can’t find those scope!
- 65. Just use `email` to identify user forget the `id` in DataStore
- 66. How about Heroku and other python PaaS
- 67. Python PaaS • heroku • dotcloud • gondor.io • ep.io • ...
- 68. Advantage against AppEngine • Pricing • Performance • Deployment • Dynamic configuration
- 70. AppEngine: $120 / mo you can run ... • heroku: 4 dynos • gondor.io: 4 wsgi server • dotcloud: 4 services
- 71. But, how about the performance?
- 72. StickerAction for example • Running on Heroku • One dyno (it means $0) • 30 reqs / sec • No errors • Smooth
- 73. New Relic
- 74. One Dyno is good enough on heroku
- 75. Deployment
- 76. Faster than AppEngine! Life is short
- 77. You can use any python packages
- 79. You dont’ have to reinvent the wheel
- 80. pip install -r requirements.txt mysql-python django-bootstrap-form==0.2 django==1.3.1 werkzeug django-guardian==1.0.3 python-memcached django-taggit==0.9.3 johnny-cache django-userena==1.0.2 django-debug-toolbar django-pagination south django-annoying django-nextpage==0.4 simples3 tweepy pillow requests python-dateutil<2.0 django-extensions boto django-storages django-admin-tools
- 82. you can switch config online
- 84. That’s why I left AppEngine
- 85. Live Demo
- 86. AppEngine is still good but not for me
- 89. Any Questions?
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: \n
- #61: \n
- #62: \n
- #63: \n
- #64: \n
- #65: \n
- #66: \n
- #67: \n
- #68: \n
- #69: \n
- #70: \n
- #71: \n
- #72: \n
- #73: \n
- #74: \n
- #75: \n
- #76: \n
- #77: \n
- #78: \n
- #79: \n
- #80: \n
- #81: \n
- #82: \n
- #83: \n
- #84: \n
- #85: \n
- #86: \n
- #87: \n
- #88: \n
- #89: \n
- #90: \n