Igniting Next Level Productivity with AI-Infused Data Integration Workflows
1 like1,182 views
The document outlines a webinar discussing AI-infused data integration workflows, focusing on various applications of AI in data processes, including the integration of GeoAI models and generative AI tools. Key insights include the potential for AI to enhance productivity, the importance of seamless integration of AI models, and practical examples of AI applications in workflow automation. It concludes with resources and next steps for participants to leverage these technologies effectively.
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
- 1. Igniting Next Level Productivity with AI-Infused Data Integration Workflows
- 2. Agenda 1 Introduction 2 Safe & FME 3 AI - the Good, the Bad and the FME by Locus 4 Integrating GeoAI Models in FME by con terra 5 Unlocking Generative AI on your own terms! by Tensing 6 Bonus! Segment by Dmitri 7 Conclusion & Resources 8 Next Steps 9 Q&A Agenda
- 3. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
- 4. How to download slides 1. Hover over the slide deck in the webinar room 2. Click this button
- 6. AI will take your job. And give you one that’s better. - Dharmesh Shah
- 7. 2 Safe & FME
- 8. 29+ 27K+ 128 190 20K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 29+ 29K+ 128 140+ 25K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 200K+ users worldwide Safe & FME
- 9. One platform, two technologies FME Form FME Flow Build and run data workflows Automate data workflows FME Flow Hosted Safe Software managed instance fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
- 10. 3 AI - the Good, the Bad and the FME
- 12. Introducing Locus ● Safe Software Platinum Partner ● Offices in New Zealand and Australia ● 15+ years on the FME journey; selling and supporting ● 7 experts with FME Technical Certifications AI - the Good, the Bad and the FME
- 13. Agenda 1 Are We Now Doomed 2 Artificial Intelligence Defined 3 FME and AI 4 FME and IoT 5 FME and ChatGPT collaborate – the results
- 14. Introduction
- 15. We are all going to be replaced by AI or are we? Introduction
- 16. We are all going to be replaced by AI?! 1. Pick a subject? 2. Pick a presentation duration? – 1-5 Minutes 3. Pick the number of slides? – 1 to 6 4. Pick an emotion? – Funny, Scary Sad, Happy, Serious… 5. Pick a picture style 6. Off we go… Introduction
- 17. Artificial Intelligence Defined Instead, let’s ask AI, surely it should know? Darren - “Define AI in 15 words exactly” ChatGPT - "AI is the development of computer systems that simulate human intelligence and perform intelligent tasks.“ Bing AI – “Artificial Intelligence (AI) is the ability of machines to perform tasks that typically require human intelligence.” Google Bard - “Artificial intelligence is the simulation of human intelligence in machines.” Darren - “But that’s only 10 words?” Google Bard - “You are right. Here is a definition of AI in 15 words exactly” “The simulation of human intelligence in machines that are able to learn and act autonomously.” Darren – “Thanks buddy” Introduction
- 19. Let’s meet some of the players ▪ There are many consumer systems out there, some are generic, others are very specific. o ChatGPT – Google Bard – Bing AI o AutoGPT ▪ Image Generation o Dall-E-2 o Midjourney o Leap.AI ▪ Others will become part of our working lives o Microsoft 365 Copilot Remember me? - I’m back! Artificial Intelligence
- 20. FME and AI ▪ OpenAI ChatGPT Connector ▪ OpenAICompletions Connector ▪ OpenAIImageGenerator ▪ Leap.AI Connector ▪ HTTP Caller o Google Bard ▪ Use AI to write a workspace FME Haiku Data is like clay Mold it, shape it, transform it Create something new Form and Flow are one Data is transformed, like clay A new shape is born Artificial Intelligence
- 21. Industry 4.0 (4 DOT ZERO) ▪ Industry 4.0 refers to the fourth industrial revolution, characterized by the integration of advanced technologies and digitalization into various industrial sectors ▪ The key features of Industry 4.0 include the following: o Internet of Things (IoT) o Big Data and Analytics o Artificial Intelligence (AI) o Automation and Robotics o Cloud Computing Artificial Intelligence
- 22. FME & IoT ▪ Azure IoT Connector ▪ Google IoT Core Connector ▪ AWS IoT Connector ▪ IBM IoT Connector ▪ MQTT Connector ▪ Kafka Apache Artificial Intelligence
- 24. The Results | FME & ChatGPT Collaborate ● Here is the result ● Let’s take a look at the Workspace and what’s going on Conclusion and Results
- 25. AI has emerged as one of the most transformative technologies of our time and has the potential to assist rather than replace human endeavours, let’s embrace it! Conclusion and Results
- 26. Takeaways ▪ There are more than 5,000 AI tools in the marketplace - if you haven’t explored some of them, now is the time! ▪ People still matter but leveraging the power of these AI tools will improve productivity, decision making and our innovation potential Conclusion and Results
- 29. Dr. Christopher Britsch Consultant con terra GmbH Dennis Wilhelm Consultant con terra GmbH
- 30. Agenda 1 What's GeoAI? 2 Integrating GeoAI in FME Processes 3 con terra GeoAI Framework 4 Project Example 5 Conclusion
- 31. What's GeoAI?
- 32. GeoAI - Definition GeoAI is a Machine Learning technology, which enables the caption and analysis of complex patterns and structures in (geospatial) data. What’s GeoAI?
- 33. Setup of a “typical” GeoAI Project Data Preprocessing AI-Methods Integration and Operation Visualization and Application What’s GeoAI?
- 34. GeoAI at con terra Data Enhancement Natural Language Processing Forecasting Object and Image Recognition GeoAI What’s GeoAI?
- 35. Analyze existing data Accumulate further data Data Preprocessing Prepare first PoC Model Train and Optimize Model Integration Monitoring and Support Use Case Determination What’s GeoAI?
- 36. Integrating where you need it, how you need it What’s GeoAI?
- 38. “Technology must be like oxygen: ubiquitous, necessary and invisible.” - Chris Lehmann Integrating GeoAI in FME
- 40. From Project to Product ● Next slides show our current work in progress ● Approach that has been used in multiple projects is now being generalized to ease usage in the future con terra GeoAI Framework
- 41. Main obstacles ● AI Model implementation is often performed by non-FME users in other environments ○ Jupyter Notebooks, Python Scripts, … ● Integration of these scripts into business processes if often unclear ● AI Models are usually based on many different libraries depending on the use case ○ Tensorflow, Keras, Pytorch, … con terra GeoAI Framework
- 42. Main obstacles ● FME-users should be able to integrate these models ● Integration of these libraries in FME can be complicated (dependency mismatch, different hardware requirements) con terra GeoAI Framework
- 43. Our approach ● Provide an AI model runtime environment that can be maintained separate from FME ● Provide transformers to easily connect to this runtime from FME con terra GeoAI Framework
- 44. Python environment ● Virtual Python environments for individual library management ○ Tensorflow, Pytorch, … ● Well defined interface to run models via HTTP or CLI con terra GeoAI Framework
- 45. FME Process con terra GeoAI Framework
- 46. New Transformer con terra GeoAI Framework
- 47. Project Example
- 48. Use Case: Anomaly Detection • Various files are regularly uploaded to a system ○ Multiple dataset types with different schemas ○ Individual schemas consist of many columns ○ Defining individual rules for each attribute (or combination of attributes) would be extremely time consuming • Errors have occurred in previous uploads that compromise the system Project Example
- 49. Solution • We use an AI model to detect anomalies by comparing the current dataset to previous datasets ○ Autoencoder using Tensorflow ○ Principal component analysis using sklearn • Conditions ○ The schema for each dataset type is consistent ○ A minimum number of previous datasets is required Project Example
- 50. What's an Autoencoder? �� Input Bottleneck Output 🚀 ✅ Project Example
- 51. 🚗 ❌ Project Example What's an Autoencoder? �� Input Bottleneck Output
- 52. Project Example
- 53. Conclusion
- 54. Current Challenge - Integration ● Constructing GeoAI models becomes easier, however… ● …productively integration models can still be challenging ● Integration of models needs to be seamless… ● …in familiar and accustomed environments Conclusion
- 55. Integrating GeoAI in FME ● The GeoAIConnector in FME solves the problem ○ Integration of pretrained models ○ Integration of custom made models ● Allows flexibility in your workspaces ○ There isn't one model for everything ➡ Flexibility to include custom implementations necessary ● Not every functionality has to be implemented in FME, but everything can be connected! Conclusion
- 57. 6 Unlocking the power of generative AI on your own terms!
- 59. Agenda ● Intro - hype to reality ● It’s not just chatGPT (and why that matters) ● Privacy and security concerns ● Integrating locally running, open-source LLM’s ● Sneak peek - multi-modal models ● Wrap up
- 61. Generative AI ≠ chatGPT Platforms including APIs ● OpenAI, Bard, Midjourney Models ● gpt-3.5-turbo, gpt-4, PaLM2 ● Llama, Mistral, LLaVA
- 64. Cost
- 65. Locally deploy Generative AI tools Ollama.ai - open source, permissive MIT license ● Terminal and API interface, runs on Linux, Mac and Windows* Mistral.ai ● requires 5GB drive space and needs approx. 8GB RAM Security / cost does not need to be a blocker for using Generative AI
- 66. Step 1 - Install Windows Subsystem for Linux (WSL2)
- 67. Step 2 - install Ubuntu/Debian
- 68. Step 3 - install Ollama
- 69. Step 4 - install/run and talk to a LLM
- 70. Generate a code-reviewer Ollama installed, mistral model enabled Time to create a prompt…
- 71. Ollama has an API - bring on HTTPCaller
- 72. FME makes things easier Tensing verified publisher FME Hub Transformer Orchestration tool Plug and play bit.ly/FMELocalGenAI
- 75. Visual AI - LLaVA vs ChatGPT4 Vision
- 76. Wrap up You can use Generative AI, keep your IT dept at bay and you budget in check Tasks such as data cleanup, SQL and Python reviewing are easy to integrate into your traditional data ETL pipelines Ollama and FME can help you get started in this new domain Be patient, it’s only going to get better
- 79. 7 Resources
- 80. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips community.safe.com /s/academy FME Academy
- 81. Check out how-to’s & demos in the knowledge base community.safe.com /s/knowledge-base Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
- 82. 8 Next Steps
- 83. We’d love to help you get started. Get in touch with us at info@safe.com Experience the FME Accelerator Contact Us Unlock the power of your data in only 90 minutes Register for free at fme.safe.com/accelerator
- 84. ClaimYour Community Badge ● Get community badges for watching webinars! ● fme.ly/WebinarBadge ● Today’s code: MSLGF Join the Community today!
- 85. 9 Q&A
- 86. ThankYou Recap of Next Steps 1 Join the FME Community 2 Contact today’s guest speakers: Darren.Fergus@locus.co.nz c.britsch@conterra.de d.wilhelm@conterra.de omorris@tensing.com 3 Experience the FME Accelerator Please fill out our webinar survey