Open Innovation: The Secret to Unlocking Cross Industry Innovation - Steve Rader - CoECI - NTL
Download as PPTX, PDF2 likes1,237 views
The document discusses NASA's use of open innovation and crowdsourcing to access new skills, expertise, and solutions from a global community. It describes NASA's Center of Excellence for Collaborative Innovation which helps NASA pose challenges to crowdsourcing platforms representing over 70 million people. Well-designed challenges have proven effective at discovering new solutions, including solutions from outside the domain of the challenge. The accelerating rate of technological change makes it difficult for organizations to keep up internally, so leveraging diverse external crowds is an effective strategy for innovation.
Open Innovation: The Secret to Unlocking Cross Industry Innovation - Steve Rader - CoECI - NTL
- 1. - Open Innovation - The Secret to Unlocking Cross Industry Innovation NASA’s Toolkit for Accessing Innovation to Build Upon Center of Excellence for Collaborative Innovation NASA Tournament Lab Steve Rader steven.n.rader@nasa.gov 713.447.7867 Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration.
- 2. NASA’s Center of Excellence for Collaborative Innovation (CoECI) 2 CoECI works across NASA and other U.S. Federal Government Agencies to assist organizations in understanding and using open innovation tools.
- 3. 90% of All Scientists that Ever Lived are Alive Today As 2017: 3.12 million patents 3 The World has Changed Significantly over the Past 20 Years
- 4. Processors/GPUs CRISPR Additive Manufacturing Materials/Nano-Coatings Robotics Components Sensors Machine Learning Components Open Software Components Autonomous Vehicles Complex Manufacturing Cloud/Quantum Computing Blockchain Technology Building Blocks Accelerating the Rate of Change Photo Credit: Betatype The accelerating rate of change is driven by low cost access to powerful technology building blocks and tools. 4
- 5. Depth/Complexity Breadth of Domain Expertise and Technologies Domain-Discipline-Industry The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting
- 6. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry Machine Learning
- 7. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry Machine Learning
- 8. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry Robotics/ Automation
- 9. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry Additive Manufacturing (3D Printing)
- 10. Depth/Complexity Breadth of Domain Expertise and Technologies Domain-Discipline-Industry The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Drones
- 11. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry
- 12. PRO Amazing new technologies that could result in significant gains towards solving hard problems CON Finding these technologies & solutions across the growing number of possible sources is hard 12
- 13. Hard to Find Skills and Expertise The Rate of Change for Knowledge and Technology is Increasing High Risk to Remaining Competitive/Relevant Hard to Keep Up with Tech Advances
- 16. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry Traditional Organization Hiring of Skills and Expertise
- 17. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry
- 18. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry Individualz
- 19. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry Team
- 20. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Used by permission: Crowd Resources Consulting Domain-Discipline-Industry The Crowd
- 21. “Open Innovation” is an Effective Tool to Keep Pace with Accelerating Change Open Innovation or crowdsourcing platforms are bringing together and curating large numbers of people from all over the world with all types of backgrounds, skills, and expertise to provide valuable products and services.
- 22. Crowd Platforms are Providing Low Friction Matching at Scale
- 23. Curated Communities Freelancers Software Coders Film-Makers Businesses, Universities, Individuals Problem Solvers Engineers & Designers 400,000 2,000,000 39,000,000 1,500,000 100,000 5,000,000 23 Crowd Facing: Company draws in members from around the world around a passion and/or interest. Customer Facing: Company mobilizes the skills and expertise of their curated community members to provide value to customers. Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration.
- 24. “Most of the bright people don’t work for you – no matter who you are.” - Bill Joy, Co-Founder of Sun Microsystems People Level of Expertise/Skills General Population Our Organization Actual Relative Size High Value Expertise/Skills Crowds Provide Access to High Value Expertise/Skills 24 Curated Community
- 25. 25 Accessing Crowd VALUE Using Challenges Formulate the Problem Statement Design the Challenge Execute the Challenge Pick the Winner(s) Evaluating Get Your Solution IP licensing and/or transfer Solution Filtering (optional) Well designed challenges posed to a curated community have proven very effective for discovering new and existing (but unknown) solutions and technologies. A well formulated problem statement (with good success criteria) Solution filtering mechanisms are A well designed challenge (including setting the right prize amount) offered by some platforms
- 26. Depth/Complexity Breadth of Domain Expertise and Technologies The Breadth and Depth of Skills, Expertise, and Technology Domain-Discipline-Industry *Jeppesen, Lars Bo and Karim R. Lakhani. Forthcoming. Marginality and problem solving effectiveness in broadcast search. Organization Science 20. Published Version http://orgsci.journal.informs. Study data of successful InnoCentive challenge solutions showed… 70% of successful challenge solutions are solved by individuals outside of the challenge’s specific technical domain. “75% of successful solvers already knew the solution to the problem.” Dr. Karim Lakhani *
- 27. NASA’s Center of Excellence for Collaborative Innovation Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. HQ ARC GRC GSFC KSC LaRC MSFC JPL JSC SSC AFRC 27 Across NASA Centers Public Facing - Worldwide Across US Federal Agencies • Part of STMD’s Prizes and Challenges Program • Works across NASA & other Federal Agencies • Virtual Office hosted within JSCHH&P • Provides Access to Open Innovation Platforms Worldwide • Contracts/Mechanisms • Processes/Support • Education/Outreach • Research
- 28. Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. Innovation & Problem Solving Algorithms & Software Micro- Purchase Design & Multimedia Global Technology Search Internal The CoECI Toolkit Curated Crowds Available for NASA Challenges & Tasks 28 Research 18 Crowds Representing over 70 Million People Worldwide
- 29. CoECICoECI NASA Crowdsourcing Landscape Months $1K to $250K Ideas, design, software Years $100K+ to $Ms Technology demos Days/Weeks Recognition Software apps/tech concepts Duration Awards Products Weeks Recognition Ideas, info Months Varies Design Months Recognition Scientific observations and analysis Worldwide Space Act Worldwide Amer. Innov. and Competitiveness Act Students (US) Space Act; grants & cooperative agreements Who Authority NASA N/A Worldwide; US- led (COMPETES) Procurement; COMPETES Act US-led (to win prize) NASA prize authority Public prize/challenge programs Other public crowdsourcing initiatives (no program lines) Internal crowdsourcing pr STUDENT CHALLENGES These opportunities can be accessed by the public on the NASA Solve website: www.nasa.gov/solve 29
- 30. Crowds can access solutions outside the “bubble” that makes technical teams blind to certain solutions
- 31. Generalize the Problem & Leverage Diverse Solvers Remove a viscous fluid from a delicate wafer Remove grease from potato chips 31Solver Solution: Acoustic Vibration at the natural frequency of the grease. Problem: Remove grease from potato chips
- 32. Strain Measurement of Kevlar Challenge 32 Total Cost: $40,000 NTL’s Innocentive Challenge Results: 3 workable solutions “So simple, so elegant how could we NOT have thought of this ourselves” NASA’s Inflatable Structures Research team needed a way to measure strain in Kevlar and vectran webbing used for inflatable space habitats. Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration.
- 33. Crowds can cut across different silos, industries, and disciplines to translate expertise and find technologies
- 34. Improving Pipeline Bundle Inspection Subsea7 was seeking to improve pipeline inspections. Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. 34 NineSigma Challenge Result: Found a technology a fraction of the size that performs inspections over 100x faster that will be much less expensive to operate. This new technology was poised to transform the industry.
- 35. Data Driven Forecasting of Solar Flares 35 Total Cost: $48,000 Innocentive Challenge Results: An algorithm capable of an 8 hour prediction that borrowed techniques from RF signal processing to extract signal from noise. NASA sought to increase NASA’s ability to predict solar flares from 2 hours to 4 hours. Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration.
- 36. Determining Urine Volume in Microgravity 36 Results: Found an working prototype developed in a lab 200 yards away saving $1.3M & 3-5 years of development. NASA was seeking better solutions for measuring urine volume in microgravity.
- 37. Crowds can provide lots of “shots on goal” to provide the failures necessary for innovation in an affordable way
- 38. An Case for Augmenting the Traditional R&D Approach 15 years of proprietary research replicated in a single 60-day public challenge Case Study: Julian Birkinshaw, MLabnotes, University of London Business School Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. 38 Results: Existing technology found to solve the problem LovePik.com
- 39. Results - Space Poop Challenge Results HeroX Challenge Results: Discovered an approach that used laparoscopic surgical mechanisms for manipulation and passing material across a pressure differential. Also discovered zero-power approach to air circulation based on medical self-inflating ventilator technology. 5000 Submissions Total Cost: $58,000 39 Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. Winning Solutions NASA needed to find new and innovative concepts to address a significant Orion contingency problem with LEA suits.
- 40. Crowds can access the power of passion with crowd-based communities of practice
- 41. Case Study CODERS SUBMITTED SOLUTIONS DIFFERENT APPROACHES TO SOLVE PROBLEM IDENTIFIED WINNING COUNTRIES RUSSIA, FRANCE, EGYPT, BELGIUM & US 122 654 89 5 Coders Submitted Solutions Different Approaches Identified Winning Countries US, Russia, France, Egypt, Belgium Improve on NIH MegaBlast algorithm for nucleotide sequence alignment Winning solution performs 120x faster ANTIBODY SEQUENCE ANNOTATION The Challenge Improve on NIH MegaBlast algorithm for nucleotide sequence alignment Source: 41 47 min. $120K 1 year Development 4.3 hours $2M+ Multi-year Development 16 sec. $6K Prize 14 Day Challenge Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration.
- 42. 43 Passenger Screening Challenge 43 Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. Result: 8 winning algorithms developed with outstanding demonstrated performance including one with a 98% detection rate. DHS Challenge Owner: "This will be the best $2.5M we (DHS) have spent in a long time." James Grove from the DHS Science and Technology Directorate needed to improve the ability of mm-wave detection systems to accurately detect threats. DHS ran a Kaggle challenge with a $1.5M prize (under the COMPETES act) to find a better algorithm to use with next generation mm-wave scanning systems.
- 43. Crowds can access individuals with skills and expertise in the latest and emerging technologies
- 44. RFID Auto Logistics Management (REALM) 45 Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. Topcoder Challenge Results: A new hybridization approach that moved the project from instrumented modules to non-instrumented modules and may be a game changer if it pans out. 56 Participants 444 Solutions Submitted Total Cost: $89,700 NASA needs to track items more precisely on ISS using REALM RFID sensor data.
- 45. Robonaut Vision Algorithms 46 Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. Total Cost: $51,000 & $60,000 NASA needed computer vision algorithms to support the Robonaut project. Topcoder Challenge Results: Two algorithms that provided the basis for computer vision and accelerated development.
- 46. DARPA Grand Challenge: Autonomous Robotic Ground Vehicles Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. Crowds can be mobilized around tangible goals for technology development that stimulates advancement and investment
- 47. 48
- 48. NASA Tournament Lab Challenges by the Numbers 94+% Successful 350+ Challenges 80% Result in Cost Savings
- 49. NTL Challenge Experience 350 Challenges Total Completed or in Progress Trade names, trademarks, and logos are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. 40 Software/Apps 31 3 3 2 1 39 Technology Survey 29 10 23 Algorithms 19 1 1 1 1 36 CAD Modeling/ Mechanical Design 31 4 1 27 Graphics/Designs 15 8 4 18 Videos 6 5 5 1 1 2 Consulting Services 1 1 62 Technical Solutions 35 12 5 2 2 2 1 2 36 Other 32 2 2 76 Ideation 57 7 5 4 2 1 1 50$15M total spent to date: 43 Challenge for other Federal Agencies ($10.5M)
- 50. Open Innovation is Key to Cross-Industry Innovation 51 If we are serious about leading in Space Exploration, we must use new open in novation tools to access the cross- industry innovations we require.
Editor's Notes
- #3: Purchased images from Getty Images account under NASA HHPC contract: Rocket-GettyImages-1015481008.jpg Gov-GettyImages-1091162402.jpg
- #4: Eric Gastfriend – Harvard Institute of Life article/paper https://futureoflife.org/2015/11/05/90-of-all-the-scientists-that-ever-lived-are-alive-today/ Image above from: https://en.wikipedia.org/wiki/Scientist World Intellectual Property Organization – 2017 the total was 3.12 M patents https://www.wipo.int/edocs/pubdocs/en/wipo_pub_943_2018.pdf Reference Credit to HeroX: Christian Cotichini https://get2growth.com/how-many-startups/ 472M entrepreneurs worldwide 100M startups opening each year 1.35M tech startups… 185 accelerator programs In 2018, Asia accounted for 2/3 of all patents https://www.wipo.int/pressroom/en/articles/2019/article_0012.html Number of people with advanced degrees has doubled since 2000 https://www.census.gov/library/stories/2019/02/number-of-people-with-masters-and-phd-degrees-double-since-2000.html
- #5: More than ever, these building blocks are not exclusive to large orgs, but to individuals and small teams Enables innovation to happen at a broader level, outside of traditional research orgs Precision tools 3d print//Robotics/automation enabling manufacture of things not previously possible Innovation possibilities increasing in the RECOMBINATION of new building blocks Purchased images from Getty Images account under NASA HHPC contract: Baxter Robot Image – Wikimedia Commons: https://commons.wikimedia.org/wiki/File:Baxter_1.jpg Nano-Tech Image: Getty Images: NanoTech-GettyImages-160380750 Machine Learning image: Getty Images: Machine Learning - GettyImages-1072325472 Drone Image: Getty Images: Drone-GettyImages-911190112 Growth Curve Image: Getty Images: Growth Curve - GettyImages-1045529064.jpg Robotic Arm: Getty Images: Robot arm - GettyImages-695586212 Additive Mfg: AMAERO’s demo of Jurg’s engine design and the original baby engine for scale. Photo via Betatype (http://www.betaty.pe) CRISPR: Getty Images: CRISPR-GettyImages-959053706 Processors GPU: Getty Images: GPUs-GettyImages-916553110 Blockchain: Getty Images: Blockchain - GettyImages-953499010 API: Getty Images: API-GettyImages-653928394 Cloud: Getty Images: Cloud-GettyImages-962404026 Sensors: Getty Images: Sensors - GettyImages-927062088 Building Blocks: Getty Images: Legos-GettyImages-469195554 (Editorial License) But we have another problem… an invisible problem The world - and particularly the technology world - is expanding and speeding up 90% of scientists that have ever lived are alive and working today Tech advances are changing the fundamentals… quickly Drones, voice interactions, machine learning, early AI, robotics, materials & nano coatings, 3D printing, sharing/gig economy Hiring for the full range and depth of expertise is becoming increasingly difficult (even for NASA) AMAERO’s demo of Jurg’s engine design and the original baby engine for scale. Photo via Betatype (http://www.betaty.pe)
- #13: Background master image is a still image of purchased via NASA HHPC Getty Image Account: Haystack-GettyImages-108202375.jpg
- #14: ½ of the fortune 500 from the last 15 years are no longer around. In 1958, the lifespan for a company on the list was roughly 61 years. Today, the average is just 18 years. At this rate, 75 per cent of the S&P 500 will be replaced by 2027. In his book “Creative Destruction”, Richard N. Foster Background image is a still image of video purchased via NASA HHPC Getty Image Account: High Speed - GettyImages-470503649
- #15: ½ of the fortune 500 from the last 15 years are no longer around. In 1958, the lifespan for a company on the list was roughly 61 years. Today, the average is just 18 years. At this rate, 75 per cent of the S&P 500 will be replaced by 2027. In his book “Creative Destruction”, Richard N. Foster Background video purchased via NASA HHPC Getty Image Account: High Speed - GettyImages-470503649
- #16: ½ of the fortune 500 from the last 15 years are no longer around. In 1958, the lifespan for a company on the list was roughly 61 years. Today, the average is just 18 years. At this rate, 75 per cent of the S&P 500 will be replaced by 2027. In his book “Creative Destruction”, Richard N. Foster Background video purchased via NASA HHPC Getty Image Account: High Speed - GettyImages-470503649
- #22: Background image is a still image of video purchased via NASA HHPC Getty Image Account: GettyImages-473009501-Crowd Slomo background.MOV
- #23: Background image from video purchased via NASA HHPC Getty Image Account: GettyImages-473009501-Crowd Slomo background.MOV Logos artwork from vendor sites – self attributed
- #24: Background video purchased via NASA HHPC Getty Image Account: GettyImages-473009501-Crowd Slomo background.MOV Image: Pixabay.com (free to use, no attribution required): Network of Networks (https://pixabay.com/vectors/social-media-connections-networking-3846597/)
- #25: Background image from video purchased via NASA HHPC Getty Image Account: GettyImages-473009501-Crowd Slomo background.MOV Graph generated in PPT.
- #26: Prize Competition GettyImages-889312658.jpg Most of the time, the best way to use these curated communities to help solve a problem or produce a product is through challenges. Challenges are simply a competition where community members develop competing solutions and the solution(s) that are judged the best or successfully solving the problem are awarded a prize. The process is pretty straightforward (reference cart). Note that for some large communities, it is possible to get 100s or 1000s of solutions. Crowdsourcing is a lot like mining! Sometimes you have to process a lot of ore to get the valuable gem! Many of the companies provide solution filtering services that can work to provide you only with the top submissions that meet all of your minimum requirements. An example of that was our Mars Pioneering challenge where we received almost 800 submissions… each a white paper (up to 30 pages) describing an approach to achieving Earth independence on a Mars settlement. As you can imagine, there was quite a bit of sci-fi and/or “crazy” in the submissions given the public appeal of the Mars Mission. However, InnoCentive (the community we used) had one of their staff read through all of the submissions and only forwarded to us those that met our requirements… which was more like 75 solutions… which saved our team a lot of time and effort. I contend that Cathryn Covington at InnoCentive is now the world’s expert on crazy Mars settlement ideas!
- #29: As I mentioned, our toolkit is know to the public as the “NASA Tournament Lab” or “NTL.” For us, this is actually a set of contracts that we make available to all NASA projects and, in fact, to all Federal Agencies to run challenges and access some pretty amazing communities. On the right, you see our main contract set… this is the NASA Open Innovation Services or NOIS contract that we started using in the summer of 2015. This is a multi-vendor IDIQ contract that enables us to get 10 of (what we consider) the most capable curated communities on the planet to compete to run our challenges. We tell them what we want to solve and they compete to give us the best worldwide contest possible for the least amount of money. TopCoder, a community of almost 1 million software & algorithm developers and InnoCentive, a diverse community of 365K have both been on contract with NAS for the last 5 years and are now under the NOIS contract. Most of the examples you will see today are from these 2 companies. However, we are just wrapping up 2 successful challenges with NineSigma, a 2M member network of individuals, universities, and businesses. HeroX is Diamantis’ company associated with the X Prize foundation… Kaggle is a community that specializes in Big Data challenges and machine learning.. .in fact there is a great TED talk by their former head data scientist that really gets into the amazing work that is being done in that area (by Kaggle). These companies are all established and are active solving challenges for industry… that is why there are so many… there is a demand by industry AND they are largely successful in solving hard problems. We also have NASA@work which is our internal Crowdsourcing platform that taps into the amazing talent we have at NASA. Yet2.com is a bit different, but they provide a technology search service that goes far beyond what most of our teams are able to do. We are partnered with Harvard and have been from the beginning. Dr. Karim Lekahni at HBS is one of the world leaders in crowdsourcing research and if you look in almost all of the literature on the subject, you’ll see his name. He works directly with us to understand how this all works and how to best use it for maximum benefit. These last 2 in the corner are somewhat new to our toolkit. These are challenges that we can do on a government purchase card….. So for $3500 or less. We’ve been a bit floored by these… GrabCAD is 2.7M mechanical engineers and Freelancer is a community of over 18M….now think about that for a second… that community is the equivalent of about 1/8th of the US workforce.
- #30: NASA generated or owned imagery and logos/graphics
- #31: May move this to barriers… but I’d like to make the point here and then illustrate with the following 2 examples.
- #32: Photo credit: Potato Chip – Steve Rader
- #33: Background video purchased via NASA HHPC Getty Image Account: GettyImages-664522526-Solar Flare.mp4 Dan’s headshot used by permission.
- #35: Slide/info/graphics used by permission by Subsea7 Pipeline Underwater GettyImages-1124236442
- #36: Background video purchased via NASA HHPC Getty Image Account: GettyImages-664522526-Solar Flare.mp4 Dan’s headshot used by permission.
- #37: NASA contract HHPC Purchased Image: iStockphoto iStock-538916337 Robert’s picture used by permission.
- #38: Pixabay Image: football-1274661_1920
- #39: Photo free From "https://lovepik.com/image-500654812/medical-laboratory.html
- #40: NASA Imagery Imagery of submissions from prize challenge submissions (NASA property) Kristyn’s headshot used by permission.
- #42: <This chart is designed to “reveal” the performance increases of the grad student work and then the Marathon Match.> Harvard did an experiment to see how effective crowdsourcing algorithm contests really was. They found an algorithm that NIH uses for Antibody Sequence Annotation called “Megablast”. This algorithm is used a lot in that community and it was developed over several years for a little over $2M. It takes 4.3 hours to run this algorithm and the result is 0.72 pts accurate. <CLICK> Harvard paid 2 graduate students $120K over 1 year to improve this algorithm and they were successful! They improved the performance 1 order of magnitude for run time to just 47 minutes while improving the accuracy to 0.77 pts. <CLICK> They then ran a 2 week Marathon Match algorithm competition on TopCoder with a $6K prize. They got a winning submission that runs in 16 seconds with an accuracy of 0.80 pts. <CLICK> This algorithm performs 120 times faster than Megablast. But that isn’t even astounding thing about this challenge. This cluster of red dots are all submissions that DIDN”T WIN, but were 1-2 orders of magnitude better performance than the grad student’s algorithm (2-3 orders better than Megablast). Out of the 654 submissions, the found 89 different approaches (which in the algorithm world is gold and provides the possibility to improve this algorithm even more). If you think about it, this is a VERY different approach to building and improving algorithms. Normally, we would either partner with a university or hire a PhD to develop our algorithm. When comparing this method to the new “worldwide competition” method where the participants are mostly community members that regularly develop algorithms competitively (and after each competition, learn from each other what the winning algorithms did), you find that there is really no comparison. At a minimum, YOUR algorithm development team would benefit from running a competition like this JUST TO GET IDEAS FOR IMPROVEMENT. This should also get you thinking about algorithms where we are just living with the current performance. Imagine an analysis tool that currently takes 4 hours to run…. So in trying to optimize your design, you can only run it 1-2 times a day and then compare results. Imagine if that same tool ran in 10 minutes (and/or with better accuracy). The chances of you getting a BETTER design go up by a lot! NOTE: Follow On challenges done by HBS.. Bode Institute (Harvard/MIT non-profit genomics) Dr. Heng Le – world expert researcher in genome sequencing algorithms (over 8000 citations)….. Results were algorithm that is 14x faster w/ same accuracy. Chart used by permission from Topcoder.
- #43: <This chart is designed to “reveal” the performance increases of the grad student work and then the Marathon Match.> Harvard did an experiment to see how effective crowdsourcing algorithm contests really was. They found an algorithm that NIH uses for Antibody Sequence Annotation called “Megablast”. This algorithm is used a lot in that community and it was developed over several years for a little over $2M. It takes 4.3 hours to run this algorithm and the result is 0.72 pts accurate. <CLICK> Harvard paid 2 graduate students $120K over 1 year to improve this algorithm and they were successful! They improved the performance 1 order of magnitude for run time to just 47 minutes while improving the accuracy to 0.77 pts. <CLICK> They then ran a 2 week Marathon Match algorithm competition on TopCoder with a $6K prize. They got a winning submission that runs in 16 seconds with an accuracy of 0.80 pts. <CLICK> This algorithm performs 120 times faster than Megablast. But that isn’t even astounding thing about this challenge. This cluster of red dots are all submissions that DIDN”T WIN, but were 1-2 orders of magnitude better performance than the grad student’s algorithm (2-3 orders better than Megablast). Out of the 654 submissions, the found 89 different approaches (which in the algorithm world is gold and provides the possibility to improve this algorithm even more). If you think about it, this is a VERY different approach to building and improving algorithms. Normally, we would either partner with a university or hire a PhD to develop our algorithm. When comparing this method to the new “worldwide competition” method where the participants are mostly community members that regularly develop algorithms competitively (and after each competition, learn from each other what the winning algorithms did), you find that there is really no comparison. At a minimum, YOUR algorithm development team would benefit from running a competition like this JUST TO GET IDEAS FOR IMPROVEMENT. This should also get you thinking about algorithms where we are just living with the current performance. Imagine an analysis tool that currently takes 4 hours to run…. So in trying to optimize your design, you can only run it 1-2 times a day and then compare results. Imagine if that same tool ran in 10 minutes (and/or with better accuracy). The chances of you getting a BETTER design go up by a lot! NOTE: Follow On challenges done by HBS.. Bode Institute (Harvard/MIT non-profit genomics) Dr. Heng Le – world expert researcher in genome sequencing algorithms (over 8000 citations)….. Results were algorithm that is 14x faster w/ same accuracy.
- #44: Main image generated for DHS and NASA use by Kaggle under NOIS contract. Scanned Image: Wikimedia Commons: https://commons.wikimedia.org/wiki/File:Mmw_large.jpg Scanner Graphic: Purchased Shutterstock airport-fullbody-scanner-sign-monitor-view-243376390 Background image from video purchased via NASA HHPC Getty Image Account: Airport Security GettyImages-145665853 Jim’s headshot used by permission.
- #46: Background video from NASA owned/generated video (NASA_REALM.mp4) REALM project logo owned by NASA Pat Fink headshot used by permission.
- #47: NASA Images Julia’s headshot used by permission.
- #49: CC Participant AI video of event.
- #50: Cost savings averages 40% Background globe image: NASA contract HHPC Purchased Image: iStockphoto iStock-504680761
- #52: NASA internal artwork image.