WebWorkersCamp 2010
1 like543 views
The document discusses recent changes in architectural concepts and implementations in technology, highlighting the importance of understanding various frameworks and their trade-offs. It emphasizes the evolution of client applications, the impact of cloud computing, and the necessity for scalability and automation while managing consistent performance. It concludes by mentioning the necessity of revisiting assumptions around databases and consistency models in the context of rapidly evolving tech stacks.
![Concepts | Context | Implementations
“The Google File System” / Google 2003
“MapReduce: Simplified Data Processing on Large
Clusters” / Google 2004
“Life beyond Distributed Transactions: an
Apostate’s Opinion” / Pat Helland, Amazon 2007
[ZFS, Dynamo, PNUTS, CAP, REST,...] papers](https://image.slidesharecdn.com/webworkerscamp2010slides-100703095514-phpapp01/85/WebWorkersCamp-2010-12-320.jpg)
![Concepts | Context | Implementations
Client and apps evolution:
html5, native apps, COMET, web of [docs,apps], ...
Modeling problems:
tags, social graphs, walls, feeds, ...
“Like the big boys” - brought legitimacy](https://image.slidesharecdn.com/webworkerscamp2010slides-100703095514-phpapp01/85/WebWorkersCamp-2010-13-320.jpg)
WebWorkersCamp 2010
- 1. Click & Tune What’s new in our architectures ? Olivier Gutknecht olg@no-distance.net / twitter.com/olg
- 2. Me Apple alumnus Fotopedia co-founder & alumnus Author, french ebook - introduction to REST http://izibook.eyrolles.com/
- 3. Me Apple alumnus Fotopedia co-founder & alumnus Author, french ebook - introduction to REST http://izibook.eyrolles.com/
- 4. Click & Tune ?
- 6. START
- 9. Getting in tune We have a lot of sliders, faders and buttons to tweak. And we often ignore most of them. Especially at architecture level.
- 10. What changed ?
- 11. What changed ? Concepts | Context | Implementations
- 12. Concepts | Context | Implementations “The Google File System” / Google 2003 “MapReduce: Simplified Data Processing on Large Clusters” / Google 2004 “Life beyond Distributed Transactions: an Apostate’s Opinion” / Pat Helland, Amazon 2007 [ZFS, Dynamo, PNUTS, CAP, REST,...] papers
- 13. Concepts | Context | Implementations Client and apps evolution: html5, native apps, COMET, web of [docs,apps], ... Modeling problems: tags, social graphs, walls, feeds, ... “Like the big boys” - brought legitimacy
- 14. Concepts | Context | Implementations Nutch - 2003 Memcached - 2003 Rails - 2004 CouchDB - 2005 S3 & EC2 - 2006 Redis, RabbitMQ, MongoDB, Node.JS...
- 16. Sliders ? # of backend servers cache size fsync / checkpointing invalidation delay
- 18. Sliders Consistency level Sync/Async mix R/W actions balance Model heterogeneity
- 19. Trade-offs
- 21. It’s the app, really. And use cases. And business cases.
- 22. Latency “Disk is the new tape” Cache ? invalidation & cold starts ...
- 23. Read Reads are a memory & locality problem. Caches. Partitions. Dupes. Smart shards.
- 24. Write Write now. Work later. (maybe not on reads) Divide and Conquer. And cheat. Our frameworks are not ready yet.
- 25. Trade-offs
- 26. Acceptable Latency Mandatory consistency (and transactions) Synchronous actions Maintainability (Cost)
- 27. NoSQL is not about No SQL. It’s about revisiting assumptions. CAP ? ACID ? Choose your poisons. (and enjoy your sliders, yay for Cassandra)
- 28. Consistency Poison #1
- 29. Partition Tolerance Poison #2
- 30. Availability
- 31. Ditto for async.
- 32. The state of art is not good enough. (we’ve been stuck in the same local minima for several years) Ilya Grigorik
- 33. Yay for node.js - and Grand Central - and EM - and Goliath - and JRuby - and ...
- 35. Hadoop. Pig. Hive. Sqoop. Flume. HBase.
- 36. So ?
- 37. Full Stack
- 40. HTTP everywhere ? & AMQP ? & Thrift ? ProtoBufs ? Avro ?
- 41. AUT OMA
- 42. Because our stacks are state of the art (or bleeding edge) Beyond Packages Global config Change Do you really want to ssh to your servers ?
- 43. “But we have very specific needs”
- 44. You’re 10% unique Jesse Robbins, Opscode CEO
- 45. You’re 10% unique (and it’s probably the things you did wrong) Jesse Robbins, Opscode CEO
- 46. The Cloud What’s great about it ? Scalability ? On-demand capacity ? Reduced CapEx ?
- 47. “Everything fails all the time” Werner Vogels (Amazon CTO)
- 48. The Cloud The best way to discover new problems. (and it’s great)
- 49. The Cloud Think stateless. Think automation. Think latency. Think crappy machines. Think lost servers. Thinks lost disks. Think offline datacenters. Think ACL. Think security. Think autoconf.
- 50. “MyPAAS”
- 51. Think Holistic
- 53. Know the limits of your tools
- 54. Episode IV: a New Hope C h e f , P u p p e t , D r i z z l e , C a s s a n d ra , MongoDB, Redis, HTTP, Rails, Node.js, EventMachine, Twisted, Heroku, SimpleDB, EC2, GAE, REST, Rack, CouchDB, Redis, Bitcask, Voldemort, Varnish, Pig, Hadoop, HBase, Thrift, Protobufs, Avro, ESI, Mahout, SolR, AMQP, Grand Central Dispatch, WebSockets, LINQ, Reconnoiter, Nginx, ...
- 55. Thanks. Creative Contents Images by Big Red Button by thirtyfootscrew - http://www.flickr.com/photos/thirtyfootscrew/3324101122/ The Steam Pipeline by stuck in customs - http://www.flickr.com/photos/stuckincustoms/3780481716 Digidesign Icon Faders by CJ Sorg - http://www.flickr.com/photos/cjsorg/3692124539/ Pro Control 24 by aud1073ch - qhttp://www.flickr.com/photos/aud1073ch/34438156/ Pipe Dream by Rishi Menon - http://www.flickr.com/photos/rxmflickr/4102530508/