Behind the Slow Growth of AI: Failed Moonshots, Unprofitable Startups, Error-Ridden Data, Little Reproducibility, and Expensive Training
0 likes1,120 views
The document discusses the slow growth of AI, citing overhyped predictions and underperformance of various AI projects, including significant losses among startups and the struggle for widespread AI adoption in healthcare. Despite initial optimism, experts express skepticism regarding AI's imminent impact, highlighting issues such as unprofitable ventures, unreliable data, and lack of reproducibility in results. Furthermore, it posits that the future trajectory of AI may likely involve augmentation of human roles rather than outright replacement.
![Open AI’s GPT-3
• GPT-3 interprets and creates text by observing statistical relationships
between words and phrases, but it doesn’t understand their meaning
• It will give nonsensical answers (“A pencil is heavier than a toaster”) or outright
dangerous replies
• “Some experts call language models “stochastic parrots” because they echo what they
hear, remixed by randomness, or call them “a mouth without a brain.”
• A UCLA computer science professor says:
• there is no scientific advancement per se”
• But CEO of OpenAI continues hype in essay: “Moore’s Law for Everything.”
• Imagine a world where, for decades, housing, education, food, clothing, etc., all halved [in cost] every
two years”](https://image.slidesharecdn.com/aislidesforslideshareresearchgate-210524030018/85/Behind-the-Slow-Growth-of-AI-Failed-Moonshots-Unprofitable-Startups-Error-Ridden-Data-Little-Reproducibility-and-Expensive-Training-10-320.jpg)
Behind the Slow Growth of AI: Failed Moonshots, Unprofitable Startups, Error-Ridden Data, Little Reproducibility, and Expensive Training
- 1. Behind the Slow Growth of AI: Failed Moonshots, Unprofitable Startups, Error-Ridden Data, Little Reproducibility, and Expensive Training Jeffrey Funk Retired Associate Professor and Independent Consultant
- 2. Market Size is Nowhere Near the Forecasts!! • In 2016, PwC predicted GNP would be 14% or $15.7 trillion higher in 2030 from AI products and services,. • McKinsey, Accenture, and Forrester also forecast similar figures by 2030 • Forrester in 2016 predicted $1.2 trillion for 2020. Five years later, in 2021, Forrester reported that the AI market was only $17 billion in 2020 • and they now project it to reach $37 billion by 2025. Oops! • Note: Markets for smart phones and tablet computers had reached $431 Billion (2012) and $95 Billion (2014) five and three years after introduction off iPhone and iPad respectively
- 3. Even Google’s Sundar Pichai Now Sounds Pessimistic • Last year’s World Economic Forum: the impact of AI could be “more profound than fire or electricity.” • This year’s Economic Forum: • Admits AI didn’t play a significant role in devising a vaccine for Covid, instead backtracking: “AI is laying a foundation to tackle future problems and it can play a much bigger role in tackling future pandemics." • His justification: I am a #technology optimist, I see how people come together to use technology for good, technology created opportunities in my personal life, and I see the long arc of technological progress. https://cio.economictimes.indiatimes.com/news/next-gen-technologies/still-early-days-of-ai-real-potential-to-come-in-place-in-10-20-years-sundar-pichai/80596570 “Still early days of AI, real potential to come in place in 10-20 years” Pichai has access to so much information about AI. Can’t he tell a better story?
- 4. Few AI Startups Disclose Revenue or Income Ones that do have large losses Startup Total Funding ($M) Last Funding Funding Last Two Years ($M) Avant 1600 2015 None Nuro 1500 Nov 20, Feb 19 500 & 94 UiPath 1200 July 20, April 19 225 & 568 Open AI 1000 July 2019 None Dataminr 1100 March 2021 475 Zoox 1000 October 2019 200 Tanium 1000 October 2020 150 Tempus 1100 Mar 20, Dec 20 550 Automation Any. 840 Nov 2019 290 DataRobot 750 December 2020 50 Startup Losses Revenues Ratio Year Megvii 438 110 -3.9 2020 DeepMind 626 360 -1.78 2019 Cloud Minds 157 121 -1.3 2017 Nest 621 726 -0.85 2017 C3.ai 62 171 -0.42 2020 UI Path 92 346 -0.27 2020 Crowdstrike 92 874 -0.11 2020 Ones that don’t, require large funding, suggesting big losses For all technologies: 90% of Unicorn startups lost money in 2019 and in 2020
- 5. Failure of Healthcare Moonshot • IBM Watson was hyped for years, until it wasn’t… • Wall Street Journal published cautionary article in 2017 • 2019 article in IEEE Spectrum concluded Watson had “overpromised and overdelivered.” • Soon afterward IBM pulled Watson from drug discovery, now it is trying to sell Watson • And no form of AI has diffused widely in Healthcare • Survey: only 1/3 of hospitals and imaging centers report using any type of AI “to aid tasks associated with patient care imaging or business operations,” • To go from single cases of usage in hospitals to wide-spread usage while diffusing to other hospitals will take years much less decades • 2020 Mayo Clinic and Harvard survey: clinical staff gave AI-based clinical decision support for diabetes a median score of 11 on a scale of 0 to 100, with only 14% saying that they would recommend the system to another clinic • Global market for AI-based imaging software was only $400 million in 2020, a tiny fraction of $22.8 billion global healthcare software market Jeff Funk and Gary Smith, Why ambitious predictions about A.I. are always wrong Slate, May 2021
- 6. Disappointments in Radiology • 2018 Turing Award Winner Geoffrey Turing claimed in 2016 radiologists would be replaced in five years, but their numbers are still rising • Ratio of images to radiologists also shows no recent acceleration • Only 11% of American radiologists reported using AI for image interpretation in 2020 in clinical practice, 33% if research and other applications are included • “Concerns over inconsistent performance………have made the actual use of AI in clinical practice modest." Europe
- 7. Behind Inconsistent Performance, Andrew Ng* • “When we collect data and test from Stanford Hospital, we can show algorithms are comparable to human radiologists in spotting certain conditions.” • But, “when you take same model/AI system to an older hospital/machine down the street, and technician uses slightly different imaging protocol, data drifts to cause performance of AI system to degrade significantly. • In contrast, any human radiologist can walk down the street to the older hospital and do just fine.“ • “All of AI, not just healthcare, has a proof-of-concept-to-production gap.” • “A good rule of thumb is that you should estimate that for every $1 you spend developing an #algorithm, you must spend $100 to deploy and support it.” *Paraphrased for brevity
- 8. Even Google’s work is being questioned • Scientists described breast cancer paper as • “we see another very high-profile journal publishing a very exciting study that has nothing to do with science. It’s more an advertisement for cool technology. We can’t really do anything with it” • Expert in structural biology said, • “Until DeepMind shares their code, nobody in the field cares and it’s just them patting themselves on the back.” He also said idea that protein folding had been solved was “laughable” • Remember Google Flu’s claims 5 years ago? • It over-estimated number of flu cases for 100 of next 108 weeks, by an average of nearly 100 percent, before being quietly retired
- 9. “I really consider AUTONOMOUS DRIVING a solved problem,” Musk said in 2016. “I think we are probably less than two years away.” • By late 2018, it was clear that self-driving cars were much harder than originally thought, with one Wall Street Journal article titled, “Driverless Hype Collides with Merciless Reality.” • In 2020 startups like Zoox, Ike, Kodiak Robotics, Lyft, Uber, and Velodyne began layoffs, bankruptcies, revaluations, and liquidations at deflated prices. • Uber sold its autonomous unit in late 2020 after years of claiming self- driving vehicles were key to future profitability. • An MIT Task Force announced in mid-2020 that fully driverless systems will take at least a decade to deploy over large areas
- 10. Open AI’s GPT-3 • GPT-3 interprets and creates text by observing statistical relationships between words and phrases, but it doesn’t understand their meaning • It will give nonsensical answers (“A pencil is heavier than a toaster”) or outright dangerous replies • “Some experts call language models “stochastic parrots” because they echo what they hear, remixed by randomness, or call them “a mouth without a brain.” • A UCLA computer science professor says: • there is no scientific advancement per se” • But CEO of OpenAI continues hype in essay: “Moore’s Law for Everything.” • Imagine a world where, for decades, housing, education, food, clothing, etc., all halved [in cost] every two years”
- 11. Even if These Moonshots Succeed, What would be the Economic Benefits? • GPT-3 • Do we expect GPT-3 to write books for us or to write papers for students? • Don’t we need a better writing assistant, one that enables humans to focus on ideas? But this requires something different than GPT! • Self-driving vehicles • What would driver do while car drives itself? Look at their phone? • Why would robot-taxis succeed more than existing ride sharing services? • Problem is not cost of driver, the problem is congestion • Goal should not be to develop cool and awe-inspiring products • Goal should be to provide economic benefits • Economic benefits can be achieved without cool and awe-inspiring technology, but this requires a different approach
- 12. Big Data’s Failures Should Also Make Us Suspicious of AI Moonshot Strategy • Many organizations are using algorithms to decide • Which children enter foster care, which patients receive medical care, which families get access to stable housing, and the bail amounts for arrested suspects • But they don’t work • Example: Government did not reveal an algorithm had determined cutoff from Medicare until she and her lawyer were in front of a judge. • The witness, a nurse, couldn’t explain anything about the algorithm because it was bought off the shelf.” She couldn’t answer what factors go into it. How is it weighted? What are the outcomes that you’re looking for?” • There are hundreds if not thousands of these stories documented in Weapons of Math Destruction and other articles and books https://www.technologyreview.com/2020/12/04/1013068/algorithms-create-a-poverty-trap-lawyers-fight-back https://blogs.scientificamerican.com/roots-of-unity/review-weapons-of-math-destruction//
- 13. There are Some Successes • Biggest online companies: Facebook, Google, Amazon…. • Advertising, News: using AI to bring better news, content, and ads to users • E-commerce: Amazon is purportedly using AI to deliver us better product choices and deliver products more efficiently to us • Social networking: Facebook and Instagram • Proponents point to big profits as evidence of AI working • But are profits due to AI or good old-fashioned monopolies? • Also, some successes in • Finance, logistics, manufacturing • Robotic process automation for white-collar workers • But more augmentation than replacement
- 14. Augmentation is Likely Future, not Replacement • Failure of Moonshots confirms low chance of replacement occurring in next 20 years • Road from augmentation to replacement typically takes decades • 90 years for agriculture jobs to fall from 60% to 20% • 55 years for manufacturing jobs to fall from 26% to 10%, and this was mostly due to imports, not automation • Service worker replacement will be even slower with few cases of replacement (bookeepers, data entry) • AI will be the same, first augmentation followed slowly by replacement Agriculture Manufac turing
- 15. How Might Augmentation Look? • Robotic process automation • mimics actions of human workers, recording clicks, determining places where humans make judgements, and rules they follow • Natural language processing • Interprets documents, web site, videos, and messages • For example, Counter terrorism • analysts work through millions of Twitter and Facebook messages, YouTube videos, and websites in multiple languages • AI systems crawl through documents, automatically translating them, extracting names of people and organizations, and doing sentiment analysis of conversations • RPA organizes text into bins, enabling a data processing and analytics pipeline that handle content at speeds never possible in the past • Similar examples in accounting, legal, journalism, finance, and architects AI’s Future: Combining Rpa With AI To Augment Knowledge Workers, Mind Matters, April 19, 2021
- 16. How Might Such a Trajectory Evolve? • Improvements in robotic process automation • Suppliers and users develop broader libraries of solved processes, each with increasing automation of tasks • Improvements in natural language processing • Broader and better interpretation of online information, bringing broader and better information to white-collar workers • Expands breadth of work for • accountants, lawyers, journalists, financial analysts, and architects • Leads to gradual yet consistent improvements in productivity for white-collar workers • and later for engineers and scientists and thus improvements in productivity of R&D?
- 17. Big Challenges Moving Forward • Falling training time, but mostly from using more computers • Supercomputer slowdown • Exponentially rising demands on computers for high accuracies • Datasets riddled with errors • Reproducibility
- 18. Stanford AI Index Report Training time for AI systems on standard images (called Image Net) fell by 8 times while number of accelerators increased by 6.4 times So 80% of improvements came from more computing and only 20% from better algorithms Training costs fell by 30%, partly because cost of computers fell But declines in computer costs are slowing & higher accuracies require much higher costs Training Time IMAGENET: TRAINING TIME and ACCELERATORS (COMPUTING POWER) of BEST SYSTEM
- 19. Improvements in Supercomputers Have Slowed: Annual Increases Since 2013 Are Much Smaller Supercomputers Source: After Moore’s Law: How Will We Know How Much Faster Computers Can Go? (or how fast can AI progress) Fractional Increase Per Year
- 20. Benchmark Error Rate Computation Required (Gflops) Environmental Cost (CO2) Economic Cost ($) Image Net Today: 11.5% 10 14 10 4 10 6 Target 1:5% 10 19 10 10 10 11 Target 2: 1% 10 21 10 20 10 20 MS COCO Today: 47% 10 14 10 4 10 4 Target 1: 30% 10 23 10 14 10 15 Target 2: 10% 10 44 10 36 10 36 Exponential Increases in Computation and Cost to Achieve Higher Accuracies. “Without Major Breakthroughs, Reducing Image Net Error Rate from 11.5% to 1% Would Require Over $100B!” Conclusion from State of AI (Artificial Intelligence) Report
- 21. “Datasets Riddled with Errors” • ImageNet and other key AI data sets contain many errors • Researchers found incorrect labels on 6% of images • Such errors can lead systems to choose wrong AI model • Big reason: data typically collected and labeled by low-paid workers • Big data sets are essential to how AI systems built and tested • Millions of road scene images fed to algorithms help AVs perceive road obstacles • Labeled medical records help algorithms predict person’s likelihood of developing particular disease • Fixing this problem requires • More expensive data collection, showing images to more people, or even discarding notion that labels are useful • These solutions will raise training costs above what is shown in previous slides https://www.wired.com/story/foundations-ai-riddled-errors/
- 22. Reproducibility • 85% of studies using machinelearning to detect Covid in chest scans failed a reproducibility test and none are ready for use in clinics, according to Nature’s Machine Intelligence Journal • According to Nature editorial: "biomedical literature is littered with studies that have failed the test of reproducibility, and many of these can be tied to methodologies and experimental practices that could not be investigated due to failure to fully disclose software and data.“ • A recent review found only 15% of #AI studies shared their code • An international group of scientists is demanding scientific journals require more transparency. They were particularly critical of Google’s excuses for not being transparent • Problems extend to all scientific disciplines, particularly drug research
- 23. Moving Forward • Reduce the Hype • Less Emphasis on Moonshots • Stop Hero Worship • Challenge CEOs, consulting companies and AI scientists with better questions, particularly those who have made big promised in past • Find and Focus on the Success Stories • Where is AI raising productivity now? • How does AI successfully augment workers? • What do the success stories tell us about the future AI trajectory? • Fix Datasets • Labels should be correct close to 100% • Demand reproducibility from researchers • Academic papers should disclose code, data sets, and everything necessary to reproduce results
- 25. Appendix
- 26. Will AI Augment or Replace Human Jobs? AI will Augment Humans Because These are the Most Economical Applications AI is slowly succeeding in radiology, factories, RPA because it augments humans $17 Billion Market for AI in 2020 Suggests Slow Diffusion good enough to replace humans Chatbots, self- driving vehicles less successful because AI isn’t
- 27. Is AI the Most Hyped Technology of Our Time? “AI is one of the most profound things we're working on as humanity. It’s more profound than fire or electricity,” Alphabet CEO Sundar Pichai said in Jan 2020 “No matter what the media say, Google contributed (and is contributing) to AI and computer science more than any private company ever contributed to any scientific domain” Jan 2021 Linkedin Post from Machine Learning Expert, Gartner, 700 likes REALITY: Forrester says AI market was $17 billion in 2020 and projects $37 billion by 2025 (smart phones were $720B in 2019). Robotic process automation is most successful, yet receives little attention. Many companies have contributed to advances in science more than Google (see comments) $15 trillion in economic gains by 2030, said McKinsey, PwC, Accenture (2016). Most doctors, lawyers, accountants, journalists, and architects will be displaced In April 2019, Musk predicted 1 million "robotaxis" on the road by 2020
- 28. Benchmark Error Rate Computation Required (Gflops) Environmental Cost (CO2) Economic Cost ($) Image Net Today: 11.5% 10 14 10 4 10 6 Target 1:5% 10 19 10 10 10 11 Target 2: 1% 10 21 10 20 10 20 MS COCO Today: 47% 10 14 10 4 10 4 Target 1: 30% 10 23 10 14 10 15 Target 2: 10% 10 44 10 36 10 36 “Without Major Breakthroughs, Dropping Image Net Error Rate from 11.5% to 1% Would Require Over $100B! Many Practitioners Feel That Real Progress in Mature Areas of Machine Learning is Stagnant” Conclusion from State of AI (Artificial Intelligence) Report