How Cyverse.org enables scalable data discoverability and re-use

Editor's Notes

• #3: (Brief) History and Context In the mid-2000s, realization inside the NSF that biology had some unique CI challenges not being met Plant Genome was already spending on full-genome characterization projects (Arabidopsis 2010, etc). Big data was on horizon - NGS just emergent BIO-specific CI. Chose plant sciences due to strong communities and sharing culture.  Funded iPlant in 2008 Project spend its first 18 months assessing the immediate and future needs for plant science, began developing CI Renewed in 2013, with broadened mandate to cover BIO in general excepting human disease Rebranded in 2016 as part of a strategy to operate sustainably after initial program is over.
• #4: What is Cyberinfrastructure? Before diving in to specifics, define Cyberinfrastructure This is remarkably similar to the definition of a Commons So, our charge was: Blend data storage + computing capability, reproducible analysis, and human expertise
• #5: Platform Overview Vertically integrated set of offerings that serve a variety of users (technical skill, science use case, geographic location, etc) Data Storage is centralized, sharing is easy. Tied to ability to analyze in situ.  Ease of use <-> Ease of Re-use Everything below the consumer-facing layer: LEGO building blocks At the bottom: Federation is baked in. We own almost no hardware! This is key. Hard to sustain!  
• #6: Adoption and Outputs (END 6:00) So, what if you build and they don’t come? Luckily, they did. On average, we serve as many users as other major CI investments like leadership class clusters or the XSEDE project. But different users! Home to lots of training and consulting (~25% effort) Cyverse has spun out at least three successful open data ecosystem products
• #7: Finding and re-using Data 1 EARLY DESIGN DECISION: Availability of a scalable “Data Store” OPTIONAL: You don’t have to keep all your data there, but we hope to add sufficient value that you do.  Tech Stack: There was nothing ready to go. Combines iRODS + ElasticSearch + Agave APIs Currently 2+ PB of user files. At UA this is purchased as needed. At TACC, sliced from our Corral storage offering. CSHL and Plant.UK federating in. Agave APIs give us access to other storage protocols like S3, SFTP, FTP, Azure, etc.
• #8: Finding and re-using Data 2 Why don’t you just give us Google, Dropbox, Box?  Data Store APIs let us implement Data Window GUI. Here’s an example from Cyverse’s DE workbench Comprehensive, easy Data Management, but petascale Aside: Provenance under the hood, but we don’t expose via UI yet
• #9: Finding and re-using Data 3 Google Drive for Scientific Big Data Can do local caching as well but hard to do native support well This has been our story to date on Data.. more in a  minute
• #10: Finding and re-using Software (1) Reagents (Data) and Protocols (Apps) both must be sharable and reusable Software -> Application Catalog Each front-end GUI has its own concept and implementation but share common infrastructure or are interoperable Here’s the DE, our flagship workbench application Deploying apps to these catalogs involves Docker or VM image GUI specifications (written in some DSL or metadata form) About half of applications in Cyverse are community contributed
• #11: Finding and re-using Software (2) Here’s ATMOSPHERE image catalog Mandate provision of help docs and examples data/usage
• #12: Finding and re-using Software (3) Here’s an example of Cyverse App and Data available in a 3rd PARTY APPLICATION Give credit and attribution to App contributor as well as primary software author (if different)
• #13: Turning back to dataCyverse Data Commons (1) (12:00) To date, Cyverse strategy around data has been "Bring it in, use and discover within the platform, we won’t lock you in" This was not selfish - adopters needed a clear path and we wanted to be sure our CI was externally reliable We’ve been working to broaden this approach as our technology has matured under a banner called “Cyverse Data Commons" We hold a lot of 1’ data. Some of it has a natural home, like NCBI. We have taken responsibility to help that happen. In other cases, it makes sense to publish in place No natural repository Data is too large to move There is an expectation that re-users will perform extensive re-analysis on it Accomplish this now with “Community Data” and deep-linking  Improving offerings over course of 2016 …
• #14: Cyverse Data Commons (2) Here are two example workflow being implemented Both result in a persistent, resolvable identifier Note: The VM workflow is already implemented in our sister project Jetstream. Images can be exported and are being published at IU Scholarworks Uses DataCite schema. Indexed by public search engines Feeds into our ElasticSearch-based metadata service to allow easy search and retrieve Search API will be publicly accessible later this year
• #15: Bullet points