Platform as reflection of values: Joyent, node.js, and beyond
- 1. Platform as a reflection of values Joyent, node.js, and beyond CTO bryan@joyent.com Bryan Cantrill @bcantrill
- 2. Platform and values • Software platform decisions are big, important decisions with lasting consequences • One doesn’t merely select component infrastructure for where it is, but also where it’s going — and what it represents • In the post-open source world of software infrastructure, this is reflected and guided by a community’s values • Beyond right tool for the job, it is the right values for the job… • …and then the right software for the values
- 3. Some platform values • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 4. Platform core values • All of these values are important — but they are in tension • Platforms aspire to balance all of them — but for every platform some small subset represents its core values • These core values attract a like-minded community — and become self-reinforcing…
- 5. Platform core values: Seventh Edition Unix • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 6. Platform core values: OpenBSD • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 7. Platform values: K • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 8. Platform values: Awk • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 9. Platform values: ZFS • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 10. Platform values: DTrace • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 11. Platform values: node.js circa 2010 • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 12. Joyent’s core values (historically): • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 13. Joyent’s core values (now): • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 14. Growing core values • While the core values can mutate over time, it usually isn’t with the same people; core values often track people • Ideally, a community is able to grow its core values by attracting and welcoming people with those new values • In 2010, there was reason for optimism for node.js, as it was enjoying an influx of technologists bearing the scars from failed experiments in Twisted Python, Event Machine, C, etc.
- 15. The node.js demographic circa 2010
- 16. Growing core values: Debuggability • As we at Joyent built our own production infrastructure on node.js, we began to need to assert some of our own values… • It started with debuggability as we added: • DTrace probes in node.js core (2010) • DTrace probes in arbitrary node.js (2011) • MDB support (postmortem!) for node.js with mdb_v8 (2011) • DTrace ustack() helper for node.js (2011)
- 17. Growing core values: Rigor • As we hit performance problems in production, we made heavy use of the DTrace ustack() helper for profiling • The need for rigor in understanding this output inspired us to invent flame graphs (2012) to visualize node.js stacks:
- 18. Growing core values: Maintainability • As we had more and more services in node.js, we found the need for a unified logging library… • We developed Bunyan in 2012, reflecting not only our values of maintainability but also debuggability (DTrace probes built in!) • Bunyan also reflects our value of stability: we needed abstractions that we could build upon!
- 19. Growing core values: Robustness • We found that the de facto modules in several critical areas didn’t entirely reflect our own values — especially around robustness and debuggability • For building REST-based services we developed Restify (2011), with an emphasis on RFC compliance and debuggability • For constructing robust asynchronous pipelines, we developed vasync (2012), with a focus on error handling and debuggability
- 20. Growing core values: Robustness • An early problem — and one with many dynamic languages — is increasing memory usage over time • It’s easy to errantly hold a reference to an entire object graph! • We added “::findjsobjects” to mdb_v8 (2012) and have used to find many thorny production problems • We later added the ability to iterate over all functions with “::jsfunctions” (!) — and then the ability to actually print the variables in a closure with “::jsclosure” (!!)
- 21. Growing core values: Stability • We maintained quite a few binary add-ons — and immediately saw the need for stability in the interface • Stability does not mean intransigence or stasis! • Stability means developing the right abstractions that can survive future change on both sides of the interface boundary • C++ can make this difficult task much more challenging… • We developed v8plus (2012), a C interface for add-ons that we continue to use in lieu of NAN (or N-API)
- 22. Growing core values: Resiliency • Our experience trying to incorporate resiliency in our node.js- based systems led us to taxonomize node.js errors • The resulting document quickly became authoritative; searching “node.js error handling” + “I’m feeling lucky” still yields it:
- 23. The challenge of changing values • Just because we believed in production-oriented values like debuggability, robustness, and stability doesn’t mean these values were widely shared! • Values aren’t necessarily contagious! • The node.js community broadly identified more closely with growth-oriented values like approachability, expressiveness, and velocity • By 2014, these values were increasingly coming into conflict, and a fracture was (in retrospect) inevitable…
- 24. Farewell Node.js
- 26. Farewell to TJ • TJ Holowaychuk was the most prolific node module contributor; if nothing else, probably wrote more node.js than anyone else • TJ wasn’t the first; there had been foreshocks:
- 27. The challenge of large communities • The departures highlight a challenge with larger communities… • When communities are large, values can easily conflict or become diluted • Smaller communities identify more strongly with their values — but larger communities are more likely to fracture over them • Sometimes this is a fork, but more often, people just leave • Sometimes the values change because people leave, but more often people leave because the values change (or fail to!)
- 28. Divergent values • We erred in either not recognizing or not articulating that the io.js fork in late 2014 represented divergent values • Even to the degree that we recognized it and articulated it, not clear that it mattered: the problem was the divergence itself • The problem was even more basic at root: we held aspirations for the values of node.js that it didn’t hold for itself • The values of node.js are (and probably have long been) the values of JavaScript itself…
- 29. Platform values: node.js circa 2015 • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
- 30. node.js • The values for JavaScript/node.js and our values didn’t align — and it was clear that they weren’t going to • We transitioned node.js to a foundation in 2015 • But many in the community putatively believed in our values (or wanted to), so we remained engaged • This hasn’t been easy, and one issue in particular highlights the challenge in accommodating divergent values…
- 31. Promises • Promises are a different way of asynchronous programming, dating back to Argus, a CLU derivative from the mid-1980s: • node.js was originally implemented in terms of promises, but removed them entirely in February 2010 (0.2!)
- 32. Promises • Because of their use in broader JavaScript, demand grew to reintroduce promise support in node.js… • But JavaScript promises conflate operational errors with programmatic errors in that they both become exceptions • Unlike thrown exceptions absent promises, we cannot know if a thrown exception in a promise is going to be later caught or not • This makes it challenging to abort a process on, say, a type error with enough information to debug it post-mortem • Promises make writing robust software more difficult
- 33. Promises • Promises embody the divergence of values in modern node.js: approachability, compatibility and expressiveness in tension with robustness, debuggability, and simplicity • If node.js must do whatever JavaScript does in the name of compatibility, is node.js permitted its own values? • This conundrum was on vivid display in the util.promisify() PR: https://github.com/nodejs/node/pull/12442
- 36. Promises • This is not an issue of someone being right or wrong! • These are two different value systems — both admirable and reasonable — coming into conflict • This issue was put to a vote: 9 in favor, 1 against, 4 abstained • But elections do not resolve differences in values • Democracy does not necessarily imply leadership!
- 37. The challenge for Joyent • We at Joyent are resigned to the divergence in values • This doesn’t mean we’re abandoning node.js… • …but it does mean that we are actively considering other platforms when making future software platform decisions • In the coming year(s), you will certainly see us in the Go community — and perhaps Rust or others • These communities aren’t necessarily a perfect reflection of our values either — but we feel better prepared for any divergence
- 38. The challenge for node.js • node.js should define what its values are — and aren’t • Growth may be an objective, but growth is not a value! • Claptrap around values that node.js doesn’t in fact have only plants the seeds of disappointment and discord • node.js should be upfront that compatibility with JavaScript will trump robustness, debuggability, stability, etc. • We remain hopeful that node.js will be a big enough tent to not run so counter to our values that we abandon it entirely!
- 39. Platform as a reflection of values • When picking a platform, values matter • Get consensus on values before selecting a platform! • With values in hand, the challenge becomes: how does one determine the values of a community? • How a community deals with an open issue reflects much on its values around robustness, thoroughness, rigor, etc. • It’s a balance between your values and that of a community… • …that may require you to create your own platform and attract your own community bound by like values!