The Slow Growth of AI: The State of AI and Its Applications
2 likes412 views
The document discusses the current state of artificial intelligence (AI) applications, highlighting failures in moonshot projects and challenges faced in healthcare, autonomous vehicles, and government algorithms. It emphasizes that while there are some successes in the virtual realm and specific industries, overall, companies struggle to see substantial returns on AI investments. Key issues include inconsistent performance, flawed datasets, and the need for improved reproducibility in AI research.
![Open AI’s GPT-3
• GPT-3 interprets and creates text by observing statistical relationships
between words and phrases, but it doesn’t understand their meaning
• It will give nonsensical answers (“A pencil is heavier than a toaster”) or outright
dangerous replies
• “Some experts call language models “stochastic parrots” because they echo what they
hear, remixed by randomness, or call them “a mouth without a brain.”
• Nature Digital Medicine said that AI
• will not replace a conversation with another human being for the foreseeable future
in clinical settings
• But CEO of OpenAI continues hype in essay: “Moore’s Law for Everything.”
• Imagine a world where, for decades, housing, education, food, clothing, etc., all
halved [in cost] every two years”](https://image.slidesharecdn.com/presentationfordataworkoutangelsalazar-210611102942/85/The-Slow-Growth-of-AI-The-State-of-AI-and-Its-Applications-10-320.jpg)
The Slow Growth of AI: The State of AI and Its Applications
- 1. The State of AI in Applications AI Moonshots are failing and most successes are in the virtual world. Big challenges for moving forward. BEYOND THE HYPE Organised by DATAWorkout Thought Leader Forum Dr Jeffrey Funk Live Stream 15, 16 & 17 June 2021 Formerly National University of Singapore https://dataworkout.com/beyond-the-hype-forum/on-demand/
- 2. BEYOND THE HYPE Organised by DATAWorkout Thought Leader Forum Dr Jeffrey Funk Live Stream 15, 16 & 17 June 2021 Formerly National University of Singapore PREDICTED GDP REAL GDP US$1.2 Trillion US$17 Billion PwC’s 2016 predictions for 2020 Revised projections for 2025 are US$37 billion Market size is nowhere near the forecasts!
- 3. Failure of Healthcare Moonshot • IBM Watson was hyped for years, until it wasn’t… • And no form of AI has diffused widely in Healthcare • Survey: only 1/3 of hospitals and imaging centers report using any type of AI “to aid tasks associated with patient care imaging or business operations,” • 2020 Mayo Clinic and Harvard survey: only 14% of staff said they would recommend the system to another clinic • Global market for AI-based imaging software was only $400 million in 2020, a tiny fraction of $22.8 billion global healthcare software market
- 4. Disappointments in Radiology • Turing Award Winner Geoffrey Hinton claimed in 2016 radiologists would be replaced in five years, but their numbers are still rising • In Survey of American radiologists • Only 11% reported using AI for image interpretation in 2020 in clinical practice, 33% if research and other applications are included • Of those not using AI, 72% have no plans to do so while about 20% want to adopt within five years • “Concerns over inconsistent performance………have made the actual use of AI in clinical practice modest."
- 5. Behind Inconsistent Performance, Andrew Ng • “When we collect data and test from Stanford Hospital, we can show algorithms are comparable to human radiologists in spotting certain conditions.” But, “when you take same model/AI system to an older hospital/machine down the street, and technician uses slightly different imaging protocol, data drifts to cause performance of AI system to degrade significantly. In contrast, any human radiologist can walk down the street to the older hospital and do just fine.“ • “All of AI, not just healthcare, has a proof-of-concept-to-production gap.” • “A good rule of thumb is that you should estimate that for every $1 you spend developing an #algorithm, you must spend $100 to deploy and support it.”
- 6. Scientists Question Google’s Healthcare Research • Breast cancer paper • “we see another very high-profile journal publishing a very exciting study that has nothing to do with science. It’s more an advertisement for cool technology. We can’t really do anything with it” • Protein folding paper • “Until DeepMind shares their code, nobody in the field cares and it’s just them patting themselves on the back.” • Remember Google Flu’s claims 5 years ago? • It over-estimated number of flu cases for 100 of next 108 weeks, by an average of nearly 100 percent, before being quietly retired
- 7. BEYOND THE HYPE Organised by DATAWorkout Thought Leader Forum Live Stream 15, 16 & 17 June 2021 Even Google’s Sundar Pichai now sounds pessimistic “Still early days of AI, real potential to come in place in 10-20 years“
- 8. Last year’s World Economic Forum: The impact of AI could be “more profound than fire or electricity.” This year’s WEF admits AI didn’t play a significant role in devising a vaccine for Covid, instead backtracking: “AI is laying a foundation to tackle future problems and it can play a much bigger role in tackling future pandemics." “Still early days of AI, real potential to come in place in 10-20 years“ Even Google’s Sundar Pichai now Sounds Pessimistic
- 9. Failure of AV Moonshot “I really consider AUTONOMOUS DRIVING a solved problem,” Musk said in 2016. “I think we are probably less than two years away.” • By late 2018, it was clear that self-driving cars were much harder than originally thought, WSJ: “Driverless Hype Collides with Merciless Reality.” • By 2020, Zoox, Ike, Kodiak Robotics, Lyft, Uber, and Velodyne began layoffs, bankruptcies, revaluations, and liquidations at deflated prices, along with sale of AV units • An MIT Task Force announced in mid-2020 that fully driverless systems will take at least a decade to deploy over large areas Already 100s of $Billions Have been Spent on R&D
- 10. Open AI’s GPT-3 • GPT-3 interprets and creates text by observing statistical relationships between words and phrases, but it doesn’t understand their meaning • It will give nonsensical answers (“A pencil is heavier than a toaster”) or outright dangerous replies • “Some experts call language models “stochastic parrots” because they echo what they hear, remixed by randomness, or call them “a mouth without a brain.” • Nature Digital Medicine said that AI • will not replace a conversation with another human being for the foreseeable future in clinical settings • But CEO of OpenAI continues hype in essay: “Moore’s Law for Everything.” • Imagine a world where, for decades, housing, education, food, clothing, etc., all halved [in cost] every two years”
- 11. Failure of AI Moonshot in Government • Many organizations are using algorithms to decide which • children enter foster care • patients receive medical care • families get access to stable housing • and the bail amounts for arrested suspects • Example: Government did not reveal an algorithm had determined cutoff from Medicare until victim was in front of judge • The witness, a nurse, couldn’t explain anything about the algorithm because it was bought off the shelf.” • There are hundreds of these stories documented in Weapons of Math Destruction (2016) and other articles and books
- 12. There are Some Successes……….. • In virtual world of advertising, news, finance, retail, e-commerce • as an outgrowth of data analytics and software automation in past • Also, some successes in • Robotic process automation for white-collar workers • But more augmentation than replacement • And overall, “Companies Are Rushing to Use AI—but Few See a Payoff,” • only 11% of firms that have deployed AI are reaping “sizable” return on their investments
- 13. Big Challenges Moving Forward • Falling training time, but mostly from using more computers • Supercomputer slowdown • Exponentially rising demands on computers for high accuracies • Datasets riddled with errors • Reproducibility A lot of focus on getting more data to reduce bias, but these challenges suggest success will take many years
- 14. Stanford AI Index Report Training Time IMAGENET: TRAINING TIME and ACCELERATORS (COMPUTING POWER) of BEST SYSTEM Most reductions in training time came from using more accelerators (i.e., computers) So 80% of improvements came from more computing and only 20% from better other things Training costs fell by 30%, partly because cost of computers fell But declines in computer costs are slowing & higher accuracies require much higher costs
- 15. Improvements in Supercomputers Have Slowed: Annual Increases Since 2013 Are Much Smaller Supercomputers Source: After Moore’s Law: How Will We Know How Much Faster Computers Can Go? (or how fast can AI progress) Fractional Increase Per Year
- 16. Benchmark Error Rate Computation Required (Gflops) Environmental Cost (CO2) Economic Cost ($) Image Net Today: 11.5% 10 14 10 4 10 6 Target 1:5% 10 19 10 10 10 11 Target 2: 1% 10 21 10 20 10 20 MS COCO Today: 47% 10 14 10 4 10 4 Target 1: 30% 10 23 10 14 10 15 Target 2: 10% 10 44 10 36 10 36 Exponential Increases in Computation and Cost to Achieve Higher Accuracies Conclusion from State of AI (Artificial Intelligence) Report
- 17. “Datasets Riddled with Errors” • Researchers found incorrect labels on 6% of images in ImageNet • Big reason: data typically collected and labeled by low-paid workers • Big data sets are essential to how AI systems built and tested • Millions of road scene images help AVs perceive road obstacles • Labeled medical records help algorithms predict chance of diseases • Fixing this problem requires • More expensive data collection, showing images to more people, or even discarding notion that labels are useful
- 18. Reproducibility • 85% of studies using machine learning to detect Covid in chest scans failed reproducibility test (and none are ready for use in clinics) • According to Nature editorial: "biomedical literature is littered with studies that have failed test of reproducibility,” …… “due to failure to fully disclose software and data.“ • A recent review (Nature) found only 15% of AI studies shared their code • An international group of scientists (Nature) is demanding more transparency • Problems extend to all scientific disciplines, particularly drug research
- 19. Moving Forward • Reduce Hype • Less Emphasis on Moonshots • Focus on Success Stories • Where is AI raising productivity now? • Probably in less demanding applications that require lower accuracy • More data is needed, but to be beneficial we must • Fix Datasets: labels should be correct close to 100% • Demand reproducibility: academic papers should disclose code, data sets, etc.