Play Architecture, Implementation, Shiny Objects, and a Proposal
1 like755 views
The document discusses Mike Slinn's insights and experiences from developing the Cadenza web application that underpins Scalacourses.com, emphasizing the challenges and successes of using the Play Framework for multimedia scalability and course delivery. It highlights key architectural features, the need for standards in the ecosystem, and proposes the establishment of a Play Framework foundation to enhance open-source collaboration. Additionally, it reflects on the historical evolution of the Play Framework and advocates for improved integration and standards to foster customer adoption and lower operational costs.
Play Architecture, Implementation, Shiny Objects, and a Proposal
- 1. Lessons Learned from a 2-year-old Play App Aug 14, 2015 Scala By The Bay Oakland CA Play Architecture, Implementation, Shiny Objects, and Proposal
- 2. • About me • About the Cadenza web app that powers ScalaCourses.com • Brief History of Play Framework • Architectural Features for Multimedia Scalability • Standards and Processes Outline
- 3. Who is this guy, and why should I care? About Mike Slinn
- 4. About Mike Slinn – Today • Architect and lead developer of Cadenza, the webapp that powers ScalaCourses.com • Co-ordinates the ScalaCourses.com instructors • Contributor to the Scala / Play courses provided through ScalaCourses.com
- 5. About Mike Slinn – Selected History • Co-founder of Vancouver AutoCAD User Society (“Oldest and Most Dangerous AutoCAD User Group”, world’s largest) • Co-Chair of Silicon Valley Ruby Conference (world’s largest) • Former primary organizer Bay Area Scala Enthusiasts (grew to world’s largest) • Former organizer of SF Scala , Scala for Startups
- 6. About Mike Slinn – Author & Expert • Wrote 3 books, including “Composable Futures With Akka 2.0” • Wrote over 5 dozen technical articles for computer magazines • Former product marketing manager at Borland for C, C++, .NET & Ruby on Rails IDEs • Traveled the world for decades teaching computer software commercially • Now travel much less and uses video conferencing instead • Often retained as a software expert for court cases and arbitrations involving software
- 9. ScalaCourses.com • Best learning experience for Scala and Play Framework • Most cost-effective option • Material is maintained up-to-date • Students have access for a year at a time • We are looking for course authors
- 10. "Better to train outstanding people into new areas than to accept marginal people with some skills" – NASA
- 11. Cadenza – Powers ScalaCourses.com • ScalaCourses.com has been serving online Scala and Play training material for 2+ years • Cadenza is: • Online content editor and multimedia delivery mechanism • Play Framework 2 application • Written in Scala, uses Akka, Slick, AWS and Postgres • Originally deployed app to Heroku, now on Digital Ocean (1GB droplet) • Still using Heroku database • Some functionality is broken out into microservices • View free Cadenza course at ScalaCourses.com to learn more
- 12. Cadenza Features – A&A • Sophisticated and fine-grained authentication and authorization (A&A) • Hierarchical permissions control access to hierarchy of entities • Entities: Lecture, Section, Course, Group, PublishingConfiguration, User, Roles • Next release adds: StudyGuide, Exam • Roles: Visitor, Student, TA, Instructor, Videographer, CopyEditor, Publisher, SuperUser • Deep integration of webapp framework, A&A dependencies and persistence mechanism • Breaking changes in Play Framework seriously impacted plugin developers, which locked us in Play 2.2.6 – more on this later
- 13. Architectural Features for Multimedia Scalability • Cadenza serves large amounts of multimedia data quickly and economically • Play app does not handle any data • Play app only handles authorization and authentication • Use AWS S3 and CloudFront to serve data • Signed URLs • Avoid enumeratees, very slow; perhaps Akka-HTTP for Play will eventually be an alternative?
- 14. Cadenza Features – Online Editors • Quick demo of lecture editor and section organizer
- 15. Cadenza Features – Live Content Caching • Content is parsed, summarized and cached • Lots of lazy vals, memoization and usage of caches • Decision of which caching mechanism and cache type to use depends on artifact purpose • Tabs in web pages are demand-loaded to increase responsiveness and decrease system load • Students can download a course notes by printing to PDF on demand • Current content is printed • Content is aggregated by course
- 16. Cadenza Features – White Label Branding • Designed to be branded for third parties who use Cadenza for course authoring and delivery of their own courses
- 17. Cadenza Features – Asset Controller • Videos and images are delivered by AWS CloudFront • Cadenza’s asset manager only delivers assets when in development mode • AWS signed URLs provide time-limited access to remote assets • App is optimized for data retrieval (read access), so students do not present a significant load • A separate instance for course authoring is recommended, with more memory
- 18. Google Guava to the rescue! Concurrency, Persistence & Caching
- 19. Cadenza Features – Database • No joined tables; domain model is created in memory • Cadenza’s AbstractTrainingEntity trait provides write-through caching support • Most database requests are served from local in-memory domain model cache • Domain model is immutable; stale entities are flushed from cache and replaced as required • Fast and scalable • “Immutable domain model with write-through caching” (IMWC) pattern is easy to program to • IMWC requires a deep integration of webapp framework and database
- 20. Cadenza Features – Immutable / Lazy Domain Model
- 21. Cadenza Features – Immutable / Lazy Domain Model • A lazily instantiated, immutable domain model starts fast • Naturally optimized for read access • Naturally supports concurrency • Stateless so it scales horizontally • Best when coupled with or implemented by local cache • Must differentiate between: • Core domain model, running out of local cache • User data, run from a distributed cache or loaded from a database • Transient data, not cached
- 22. Caches and Concurrency • The Play cache interface is not threadsafe (affects all implementations, such as EHCache). • This does not matter if multiple threads attempt to set the same value, however. • Better to implement a threadsafe cache using Google Guava’s CacheBuilder.
- 23. Cadenza Front Page (Cached) def apply(pubConf: PublisherConfig, referrer: String="") (implicit request: RequestHeader): Html = views.html.main(Model.frontPageTitle, enableGoogleAnalytics=true) { Html.empty } { Html(s"""<div class="row-fluid"> | <div class="span6"> | ${pubConf.cache("courseBlurbs", courseBlurbs)} | </div> | <div class="span6"> | ${ pubConf.comments.getOrElse("") } | </div> |</div> |${ request.flash.get("signedUp").map { _ => signedUp }.mkString("n") } |""".stripMargin) }
- 24. Those ignorant of history are doomed to repeat it Brief History of Play Framework
- 25. Play Framework is Popular
- 26. Play versions • 1.0 Oct 19 2009 • 1.1 Nov 01 2010 • 1.2 Apr 13 2011 • 2.0 Mar 13 2012 • 2.1.0 Feb 06 2013 • 2.2.0 Sep 20 2013 • 2.3.0 May 30 2014 • 2.4.0 May 26 2015 • 3.0 May 2016 Huge API churn
- 29. 2012-04-13 - Play 2.0 announced
- 34. Next year: Play 3.0 • Biggest feature: Reworked dependency injection • No details yet, even though ship date is likely 7 months away. • No indication if CDI 2.0 (Contexts and Dependency Injection), which implements JSR 365, will be supported • Initial reaction to my suggestion was that nothing could be learned from standards
- 35. Scala Ecosystem generally lacks S&P Standards and Processes
- 36. Hmm… • Go-Karts - Feel fast, are slow • 747s - Feel slow, are fast • “Ship it!” - Feels fast, is slow • “Test + Document it!” - Feels slow, is fast • “Define standards and engage community” – Feels slow, is powerful • Build the right thing (product management, standards) vs. build the thing right (agile) • High agile velocity without adult supervision => API churn
- 38. Standards Engage Ecosystems • “Lack of standards threatens to derail the big data innovation train” – Jonas Bonér • Standards also important for Scala ecosystem in general and Play in particular • Jonas only recently has begun to profess the standards religion • The Akka project is the only one within Typesafe that shows evidence of interest in standards
- 39. Standards and Process • Standards bring people together: core developers, customers, 3rd party developers • Result (of a healthy standards process): • Well thought out specifications • Quality reference implementations • This drives: • Customer adoption • Lower cost of operation • Less API churn
- 40. Play Framework is Special • Play Framework requires integrating language features, persistence, security, A&A, etc • Integration success metrics are hugely affected by availability of good standards • Change in mindset required: • What can be learned from existing standards? • How can customer usage patterns be codified into standards with competing implementations? • Result: • Huge reduction in API churn • Much greater adoption by enterprises • Much higher quality product
- 41. Standards and Process Can Be Powerful • Sun was bureaucratic but that allowed them to foster standards • Java is still hugely important decades later as a result • Ruby was initiated by young developers • Disdained standards (our projects are so different that standards do not apply) • Ruby became merely the language that powered Ruby on Rails • 40% drop in demand for Rubyists in 2014, downward trend is continuing • Scala’s ecosystem also lacks standards and a vehicle to provide process
- 42. Just Enough Bureaucracy • Bureaucracy is boring, but when used appropriately it allows many participants to interact effectively • Project management requires bureaucracy • Release planning • Experimentation prior to making design decisions • Communicating and engaging users • Integration with other products
- 43. “The bug wasn't ours. It was in an open source project we use, but do not fund or contribute to in any way.” Proposal: Play Framework Foundation
- 44. Set Play Free! • Open source is becoming even more open – because it has to • Joyent gave away Node.js to the Node.js Foundation • Google gave away all rights for Kubernetes to the Cloud Native Computing Foundation • Backed by Cisco, eBay, Goldman Sachs, IBM, Intel, Joyent, Twitter, Google and Docker • VMware gave away Cloud Foundry to the Cloud Foundry Foundation • All of these foundations are managed by the Linux Foundation, which is now a meta-foundation • Typesafe should give Play a boost by doing the same: The Play Framework Foundation
- 48. Thank you! • Mike Slinn • mslinn@scalacourses.com • @scalacourses
Editor's Notes
- #21: This immutable domain model is lazily instantiated. Thus, the app starts very quickly, is optimized for read access and supports concurrency without locking. Updating the model takes considerably more computational resources. Lecture.upsert() invokes Section.upsert() which invokes Course.upsert() which invokes Group.upsert() which invokes PublisherConfiguration.upsert()
- #26: http://blog.takipi.com/the-top-100-scala-libraries-in-2015-based-on-64562-github-libraries/ Play (@ # 3) much more popular than Spark (@ #35)
- #27: Play started as an internal project at Zenexity in 2007 Open source project in 2009 Python build scripts Groovy template language Java based Used archive.org WayBack Machine
- #28: 2010-11-01 v1.1 RC3 Fix the bug and hit reload Stateless model ("share nothing") Ready for REST, horizontal scaling. Efficient template system Resolve errors quickly Provides integration with Hibernate, OpenID, Memcached Plugin system. Pure Java Code. Integrates nicely with eclipse or netbeans. Starts fast and runs fast! Plugin system was heavily used, lots of plugins were created, playframework.org hosted them
- #29: Top of page says "Play framework is joining the Typesafe Stack“ V1.x users were thrown under the bus – v2 was completely incompatible, there was no migration strategy except a total rewrite using a different language and build system V1 productivity was due to: Off-the-shelf modules that could be customized Quick turnaround Expressive language V2+ essentially forgot about modules/plugins
- #30: Complete rewrite. Huge number of features, many of which were never completed (for example, asset pipeline) Released too early, chaos has yet to subside. Front page: lightweight, stateless, web-friendly architecture and features predictable and minimal resource consumption (CPU, memory, threads) for highly-scalable applications - thanks to its reactive model, based on Iteratee IO Iteratees are a producer/consumer implementation, with very poor performance and are difficult to work with. No mention of plugins, and modules page says: "These modules are for the Play 1.x series only. Play 2.0 modules can be hosted anywhere on any Ivy, Maven or Git repository. We will add a directory for them here shortly." ... never happened to this day. Ready to use with latest Web technologies like Less CSS, Google Closure and CoffeeScript. Philosophy page said: Built for asynchronous programming Focused on type safety provides native Akka support for highly-distributed systems templates and route files will be easier to package and reuse (not true yet) Native support for Java and Scala new Anorm SQL component container-less deployment (traded in Tomcat for Docker / Heroku / etc) SBT build system Datastore and model integration: built-in relational database access libraries such as Ebean, JPA and Anorm Docs said: “A type safe DSL to generate SQL is a mistake” https://web.archive.org/web/20120317020635/http://www.playframework.org/documentation/2.0/ScalaAnorm Zenexity’s DB philosophy was at odds with Typesafe’s. Today, the result is a confusing set of poorly supported mechanisms. Start of yearly major release schedule Meet the team: Guillaume Bort, co-founder of Zenexity Nicolas Leroux, technical director at Lunatech Research Erwan Loisant, Senior Architect at Zenexity Peter Hilton, Operations Director at Lunatech Research Morten Kjetland, consultant Peter Hausel, Typesafe Sadek Drobi, CTO of Zenexity (told me he was principal mover behind Scala support in v2) Maxime Dantec, Zenexity UI director
- #31: 2013-05-18 .org now redirects to .com v2.1.1 "The High Velocity Web Framework For Java and Scala" Main message: Play Framework makes it easy to build web applications with Java & Scala. Play is lightweight, stateless, web-friendly architecture, minimal resource consumption for highly-scalable applications. JSON is a first class citizen Websockets, Comet, EventSource Hype: RESTful by default Asset Compiler for CoffeeScript, LESS, etc Extensive NoSQL & Big Data Support Numerous Play plugins (for Play 1.x, docs for Play 2 plugins were incomplete, few people bothered to reverse engineer Play plugins) Zenexity logo is still shown at bottom of page next to Typesafe logo.
- #33: play command was replaced by activator Most changes were poorly communicated in advance and surprised users, not well documented. Play 2.3 release did not mesh well with nearly concurrent Scala 2.11 release Some important 3rd party developers gave up when this version was released; their enhancements still have no replacements. Contributors, in the order listed: Guillaume Bort Alexandre Chatiron Rich Dougherty Sadek Drobi Arthur Gautier Peter Hausel Peter Hilton Christopher Hunt, Typesafe Takafumi Ikeda Morten Kjetland Nicolas Leroux Erwan Loisant, Senior Architect at Zenexity Ben McCann Greg Methvin Nilanjan Raychaudhuri James Roper, tech lead for the Play team at Typesafe Will Sargent Josh Suereth Peter Vlugter Pascal Voitot Cédric Chantepie
- #34: Current Temporary Modules directory in 2.4.0 docs lists a bunch of stuff that is mostly incompatible with 2.x, and almost nothing that works with 2.4.x Roadmap effectively abandoned, communication with 3rd party developers at all-time low