Community and Code: Lessons from NESCent Hackathons
- 1. Community & Code Lessons from NESCent hackathons Arlin Stoltzfus, Michael S. Rosenberg, Hilmar Lapp, Aidan Budd, Karen Cranston, Enrico Pontelli, Shann Oliver, and Rutger A. Vos https://nescent.github.io/community-and-code/
- 2. A lot of questions What practices make hackathons effective or ineffective? • What makes a good scope or theme? • How best to advertise and recruit? • How to engage before the event? • What logistics are preferable? • What supportive technology to use? • What modes of team formation? • What methods of target selection? What do people expect hackathons to deliver? Why do • sponsors underwrite hackathons? • organizers plan and execute hackathons? • participants participate?
- 3. What do hackathons do? Why are hackathons a thing?
- 4. Approach • Retrospective analysis of a series of 9 events • “Open Science” practices left a record of – 9 events with 17 event outcomes – 54 projects with 133 project outcomes – 207 participants • Guiding questions – What were the outcomes and impacts? – What practices or conditions favored positive outcomes?
- 5. NESCent hackathons are distinctive • Open and community-oriented – Sponsors have broad community-oriented goals – Everything is open-source – Diversity is a priority • Domain: evolutionary informatics – a dispersed community of thousands worldwide • Participants (25 to 30) – post-docs, faculty, grads (mostly PhDs) – offered full travel support – most apply in response to open call
- 6. Schedule Day 1 Days 2 to 4 Day 5 Work Lunch Wrap-up Work Lunch Work Stand-ups
- 7. Why the events happened Why did NESCent sponsor hackathons? • to address interop issues involving collaboration or building community resources • to foster and energize a community of practice – Spread awareness of best practices – Grow a professional network – Catalyze awareness of domain challenges & opportunities
- 8. Outcomes for • 9 events – 17 event-level products • 54 projects (teams) – 133 project products (mostly repos and team reports)
- 9. The most common products • new code repositories – most remain inactive • additions to existing codebases – E.g., open-source toolboxes (BioJava, BioPerl, etc) atypical: group keeps working, writes paper spike of 238 commits peaking at hackathon
- 10. Less common products • documentation – CRAN task view – Report on file format and parser compliance • designs, standards and schemata – Phylorefs – Skelesim • data products – Machine-readable tree annotations • community infrastructure – r-sig-phylo mailing list Details: see speaker notes
- 11. Follow-on products • demonstrations and production code – DendroPy • Blogs, conference presentations • manuscripts for publication – 2 event publications – rotl – skelesim • proposals for funding – GSOC proposals – phyloreferences – Phylotastic Link is not via actual code, but schemes, community buy-in and working relationships Project builds on code
- 12. Note • End of stuff we counted objectively • Everything from here is subjective
- 13. Intangible outcomes • Technology learning • Exposure to best practices • Awareness of challenges and opportunities • Team programming experience Credit: Randall Munroe https://xkcd.com/1425/
- 14. Lessons: scope a theme "must have the capacity to inspire participation by being specific enough to indicate the direction, while possessing sufficient openness to allow for the imagination of the group to take over” (OpenSpace philosophy)
- 15. Lessons: assistive technology • At least – source code repository – event-wide communication channel • provide training before or on day 1
- 16. Lessons: diversity • open call response rate: ~ 1/100 • personal appeal response rate: ~ 1/2 earlier events later Details: see speaker notes 2- to 3-fold increased diversity
- 17. Lessons: being welcoming • reach out to non-networked participants – “Thanks again for applying to the hackathon. <sentence that shows I’m a human who read your application>. I look forward to meeting you.” • do pre-event engagement • model positive communication – e.g., not "Isn’t that idea out of scope?" but "What are some ways that could align with our goals?"
- 18. Lesson: be wary of remote participation Works better when • Remote participant – is in similar time zone – Is already networked, experienced – commits 100 % of attention to event • On-site participants – stick to schedule – Establish 2-way channel (buddy system)
- 19. Lesson: manage team formation • Things work better when – Facilitators model asking questions to ensure plan is • Relevant (in scope) • Feasible given skills of participants • Aimed at getting tangible outcomes by the end of the event – A team is 3 to 7 people – A project requires collaboration for success Explore & sift Inform pitch Team up WorkCarefully managed, facilitated steps
- 20. Lessons: pre-event engagement • introduction in online space or telecon • chance to ask questions and discuss ideas • chance for organizers to gauge – how well theme stimulates ideas – where participants need training or info • don’t expect full participation preparation for "invested participation" (Briscoe & Mulligan)
- 21. Concluding thoughts Why participate? • learning new skills • networking • awareness • experience valued for its own sake Details: see speaker notes
- 23. Acknowledgements One of 2 working groups Some of the 207 NESCent hackathon participants
Editor's Notes
- #3: If you are interested in organizing hackathon for effectiveness, then you will have a lot of questions like these. (examples). This leads to other questions, however.
- #4: And ultimately to the question of what exactly are hackathons doing that people like so much. To the extent that I have something to offer on this issue, it is due to (describe experience, report)
- #8: As someone who worked directly with the instigators and organizers of NESCent hackathons, I know what we were trying to do initially. Primarily we were trying to address interop issues by bringing together people who wouldn’t normally be on the same team. But we recognized very quickly that we were also doing other things, like creating a shared awareness of technical challenges and best practices, and we began to think about those more consciously.
- #10: A typical case: code repo not used after Sept 2014 OpenTree hackathon (my taxon sampling repo) An atypical case: repo initiated at hackathon, much further activity, now a publication (rotl) A somewhat typical case: project from 2008, spike of activity from one week before hackathon to two weeks after (dendropy), 238 commits, 20K lines of code!!
- #11: designs, standards and schemata phyloreferencing - a way of specify coordinates in tree-space that greatly widens useability of trees * "skelesim" design for unified interface to multiple simulation tools in R * team wanted to do more but it was too daunting * got funding for a team meeting at another center * ended up meeting regularly for at least 8 months after the hackathon * r-sig-phylo mailing list 30 to 60 msg per month, 1155 users (from 28 original)
- #12: Most teams stop work when the hackathon ends. However, sometimes there are follow-on products that emerge months or years after the event. (skelesim, rotl) 2 event publications - you all have seen this kind of thing. Multi-author paper just describes the outcomes. Having written two of these myself, I don't see what purpose they serve other than giving people a paper to cite for the work they did.
- #14: Best practices: versioning, tests, docs Tech: github Awareness: people who are part of an active, exploring technology community have a sense of what is do-able and interesting; what is a challenge; what has been done many times without catching on; this kind of domain-tech awareness is IMHO one of the main intangible benefits.
- #15: * VoCamp -- too loose. combining biodiversity with ontologies & controlled vocabularies. Too vague. Few tangible products at the end. * second phylotastic -- too narrow. pressure to go out of scope, force non-ideal choices of projects. lesson. hackathons are about prototyping. if the prototype is done, the next step is to organize a sustained project.
- #16: pre-select assistive tech * source code repository * communication * provide training before or on day 1
- #17: how do we get over 40 % women and minority participation? * how do you find qualified women and minorities? ask other women and minorities (so, good to have diverse organizing team). * in your advertising & recruitment materials, don't equate participants with "programmers" (or especially, "gurus" or "experts"). Describe other roles such as "domain expert" and "use-case consultant" (not "non-programmer"). Don't go too far or it will backfire. If you recruit people with meager technical skill or knowledge AND they are not networked, the risk is much increased that they will not be able to engage productively in spontaneous team-formation.
- #18: A shy person who is surrounded by better informed and more technically competent peers, some of whom already know each other, needs encouragement. I just emailed everyone on the roster whom I didn’t know, with a simple message like this.
- #19: There are 12 remote people shown here. About 5 of them didn’t participate or participated at a very low level. Most of the ones who participated effectively were all together at a kind of satellite hackathon 3 time-zones away.
- #20: Pitches are the crux of the process. If you don’t have at least 6 good pitches going into the team-up process, you can’t have a good hackathon. A spontaneously assembled set of peers larger than 7 will tend to break out into subgroups.
- #21: \graphic: person in telecon
- #22: You are really giving people a gift when you say, "You are free from all of your ordinary responsibilities. You have no boss, no meetings to attend, and no calls to take. We want you to go into a room full of other talented, motivated people, and be part of a team that brainstorms, plans, and works hard for a day or more on a project of mutual interest." That's a gift. Hackathons exist partly because people will go out of their way to take that gift. For some participants, it is a gift that keeps on giving. Last month I was on a videoconference with a dispersed team of collaborators, and I happened to ask a junior scientist how NESCent hackathons had affected her. She said something along the lines of "I've basically built my career on them". The hackathons brought her into a network, where she became known for having energy along with a particular set of interests and skills, and she has followed that up with productive collaborations with other participants over a 5-year period.