TechInAsia PDC 2019 - Unlocking The Potential of IoT with AI
0 likes442 views
This document discusses the integration of Artificial Intelligence (AI) with the Internet of Things (IoT), focusing on applications, architecture, and case studies in Indonesia, particularly in cattle monitoring through the 'Smarternak' system. It emphasizes the advantages of edge computing with AI, such as reduced latency and improved privacy, and outlines various technologies and methodologies for optimizing AI at the edge using low-power microcontrollers. Additionally, it presents practical applications like AI-powered trash management and tools for recognizing traditional batik patterns.
![Demo Video: h]ps://bit.ly/aitrash-fv](https://image.slidesharecdn.com/tiapdc19-unlockingthepotentialofiotwithai-200331154502/85/TechInAsia-PDC-2019-Unlocking-The-Potential-of-IoT-with-AI-81-320.jpg)
TechInAsia PDC 2019 - Unlocking The Potential of IoT with AI
- 1. Unlocking The Potential of IoT with AI Intro, Use Cases, A bit of code #TIAPDC2019 - Jakarta - July 4, 2019 | 11.15AM Andri Yadi CEO, DycodeX
- 3. Andri Yadi Co-founder & CEO of DycodeX Vice Chairman, Indonesia IoT Association (ASIOTI) a (at) dycodex.com | http://andriyadi.com A Physicist, Developer, Maker, Community Guy, Entrepreneur About Me MicrosoN Most Valuable Professional (MVP) of Azure for 12 years Code for food & passion for 20 years Break & make electronic stuVs for 22 years Trying to change the world through entrepreneurship, 15 years now
- 4. PT. DycodeX Teknologi Nusantara Today, we're pioneering and leading in developing end-to-end home- grown A"i$cial Intelligence (AI) & Internet of Things (IoT)-based products & solutions in Indonesia, and enable maker movement along the way. One of pioneers in AIoT in the country and does the tech in-house! As seen on DycodeX Our vision is to solve big problems with technology.
- 5. P.S: Watch the movie. By Steven Spielberg
- 10. Ingestion / API Intelligence EngineInternet AI-powered, Cloud-backed Application Edge Cloud Simplified Architecture Inference result Raw data
- 11. Face API (Azure Cognitive Service) Internet If you remember: how-old.net Edge Cloud how-old.net (circa 2015) (Web Browser) Image data *original version Age & gender prediction, face bounding boxes
- 13. “Network of physical objects with embedded electronics, software, and connectivity, that exchange data, to enable “smart” and advanced applications and services„ So, what is Internet of Things?
- 14. IoT Common (Simplified) Architecture Gateways / Base Station Rule + Alert, Internet CloudEdge Ingestion & Storage Things Visualization User Apps Devices & Firmware Mgt. Analytics (lots of them)
- 15. SMARTernak - 1,000m ViewSMARTernak BASE STATION 5 km coverage, 1000+ devices. TRACKER Catte-wearable tracker contains a bunch of sensors ENVIRONMENTAL SENSORS Collection of sensors to monitor environmental *optional* SMART CAMERA Monitor cattle’s behaviour, body heat, to estimate body weight through image processing. FARM MANAGER / OWNER / INVESTOR DRONE Provide surveillance and to help herding. *In development* VIRTUAL FENCE Contain and move cattle without physical posts and wires. CLOUD Where the heavy- lifting happens. CARETAKER One farmer/ caretaker can easily cover a vast grazing area and hundreds of cattle.
- 16. SMARTernak: Monitoring Monitor cale’s where-about & well-being parameters hip://dycodex.com/smafernak Sensed Parameters: Latitude, Longitude Speed, Direction Body temperature Ambient temperature & humidity Ambient light Ambient sound Movement (Linear, Angular, Direction) Chest Circumference Device removal status Battery voltage & capacity
- 17. Internet SMARTernak: Monitoring Monitor cale’s where-about & well-being parameters hip://dycodex.com/smafernak Sensed Parameters: Latitude, Longitude Speed, Direction Body temperature Ambient temperature & humidity Ambient light Ambient sound Movement (Linear, Angular, Direction) Chest Circumference Device removal status Battery voltage & capacity
- 18. SMARTernak: Monitoring Now imagine… …10,000 cows …10,000 views
- 19. SMARTernak: Monitoring Now imagine… …10,000 cows …10,000 views Farmers may not need or understand raw data!
- 20. What if we put AI into IoT?
- 21. AI + IoT IoT is about automated data collecting, storing, visualisation, and reacting, in massive scale “Data Supplier”
- 22. AI + IoT IoT is about automated data collecting, storing, visualisation, and reacting, in massive scale Combining AI + IoT enables a lot of use cases & business models! “Data Supplier” AI is about making sense of plethora of data, generating insights & recommendation, and predicting the future outcome “Data Miner”
- 23. AI + IoT: Insights This is where AI helps in SMARTernak. Because cale-farmer doesn’t care about (raw) data Mobile App Accelerometer Gyro Body Temperature Chest Circumference
- 24. Hyperlocal Weather Prediction Solar sensor Air temperature, humidity, barometric pressure Rain gauge Wind speedWind direction “Cheap” weather station on each blocks (city, farm, …) Air quality Weather Prediction Use Cases: Farming Online “Ojek” Fare and Distribution Logistics Flooding Damage predictionDeep Learning Model Examples: IBM Deep Thunder Smacphone data
- 25. Ingestion / API Azure Cognitive Service: Face API Internet Example: Gender & Emotion Recognition Edge Cloud Microcontroller + Camera Sensor Inference result (JSON) Image
- 26. Ingestion / API Azure Cognitive Service: Face API Internet Example: Gender & Emotion Recognition Edge Cloud Microcontroller + Camera Sensor Inference result (JSON) Image
- 27. Ingestion / API Azure Cognitive Service: Face API Internet Example: Gender & Emotion Recognition Edge Cloud Male, Neutral Microcontroller + Camera Sensor Inference result (JSON) Image
- 28. Demo: Gender & Emotion Recognition in the Cloud hps://www.youtube.com/watch?v=vvQfYqrUwFg
- 29. That demo is super simple, but it can help visually impaired people to “see”
- 32. …and those are cloud-powered Artificial Intelligence Source: https://www.riskgroupllc.com/wp-content/uploads/Cloud-Powered-AI.jpg
- 33. AI on The Cloud Pros: You can always have more, just scale “Unlimited” computing power: for ML learning & inference “Unlimited” storage: for AI model and dataset Easier to maintain Easier to update (soNware, AI model/algorithm), to secure, etc
- 34. AI on The Cloud Pros: You can always have more, just scale “Unlimited” computing power: for ML learning & inference “Unlimited” storage: for AI model and dataset Easier to maintain Easier to update (soNware, AI model/algorithm), to secure, etc Cons: High latency: as data need to transferred to cloud Must be online: to exchange data with cloud. Connectivity can be challenges for cefain use cases Potential privacy issue: your face goes to the cloud, who know what will happen
- 35. AI on The Cloud - Cons High latency Must be online Potential privacy issue
- 36. AI on The Cloud - Cons High latency Must be online Potential privacy issue What If… We move intelligence closer or at the Edge
- 37. Ingestion / API Intelligence EngineInternet Edge Cloud Simplified Architecture of AI-powered Application Inference result Raw data Back to…
- 39. Ingestion & API Intelligence Engine Internet Edge Cloud Intelligence Current Trend Pac of “intelligence” move to the Edge Hence, AI at The Edge
- 40. But, why AI at the Edge?
- 41. AI at The Edge Pros: High peqormance: less latency as no data need to go to cloud Works orine: no internet connectivity needed to exchange data with cloud. Beier privacy: data can stays on-device if necessary Escient power consumption: no need to send large data to cloud
- 42. AI at The Edge Pros: High peqormance: less latency as no data need to go to cloud Works orine: no internet connectivity needed to exchange data with cloud. Beier privacy: data can stays on-device if necessary Escient power consumption: no need to send large data to cloud Cons: ML model needs to be optimised: optimisation during or post training. Post training optimisation involves model conversion which not always be trivial Challenging deployment & maintenance: new ML model means OTA update is needed for many devices
- 43. Example: Face Recognition at The Edge Ingestion / API Face Detection & Recognition Internet Edge Cloud Camera
- 44. Example: Face Recognition at The Edge Ingestion / API Face Detection & Recognition Internet Edge Cloud Andri Camera
- 45. Example: Face Recognition at The Edge Ingestion / API Face Detection & Recognition Internet Edge Cloud Andri { “detected”: “2019-07-04T090422”, “face_id”, 1, “face_name”, “Andri”, … } Camera
- 46. Example: Face Recognition at The Edge Ingestion / API Face Detection & Recognition Internet Edge Cloud Andri { “detected”: “2019-07-04T090422”, “face_id”, 1, “face_name”, “Andri”, … } Camera 5D 1B 16 56 01 41 6E 64 72 69 or simply…
- 47. Example: Face Recognition at The Edge Ingestion / API Face Detection & Recognition Internet Edge Cloud Andri { “detected”: “2019-07-04T090422”, “face_id”, 1, “face_name”, “Andri”, … } Camera 5D 1B 16 56 01 41 6E 64 72 69 or simply… ~100KB
- 48. Example: Face Recognition at The Edge Ingestion / API Face Detection & Recognition Internet Edge Cloud Andri { “detected”: “2019-07-04T090422”, “face_id”, 1, “face_name”, “Andri”, … } Camera 5D 1B 16 56 01 41 6E 64 72 69 or simply… ~100KB —> 10 Bytes
- 49. Example: Face Recognition + LPWA Ingestion / API Face Detection & Recognition LPWA Edge Cloud Andri { “detected”: “2019-07-04T090422”, “face_id”, 1, “face_name”, “Andri”, … } Camera 5D 1B 16 56 01 41 6E 64 72 69 or simply… LPWA: Low Power Wide Area Network: LoRa/LoRaWAN, SigFox, NB-IoT
- 50. Now… “Jerry Maguire” (1996). P.S: Watch the movie.
- 53. 9Billion+ Microcontroller-powered devices deployed each year AI on “Thing” will open a whole lot oppocunities
- 54. Still not sure?
- 55. So, how can we achieve AI at the Edge?
- 56. ML-accelerated processor a class of microprocessor designed as hardware acceleration for AI applications, e.g. for neural networks, machine vision and machine learning Example: GPU FPGA ASIC Development Board / Accessories a standalone computing system with ML- accelerated processor (main or co-) as a main or additional development unit Example: Coral DevBoard or USB Accelerator Intel Neural Compute Stick Sipeed Maix Sorware Tools Compilers ML Model Convefer SDK OS Example: TensorFlow & TensorFlow Lite OpenVino nncase Suppocs Documentation Model Zoo Examples Datasheets Hardware Sorware What do we need?
- 58. Age, gender, emotion detection at the Edge DEMO
- 60. ML-accelerated processor Optimized deep learning solutions across multiple Intel® hardware plavorms Example: CPU (Xeon) FPGA (Arria) ASIC (Movidius) Development Board / Accessories Example: Intel Neural Compute Stick (Movidius) UP Square Board + AI Vision X Dev Kit IEI TANK AIoT Dev Kit Sorware Tools Model Optimizer Inference Engine Model Downloader Suppocs Documentation Reference Implementation Model Zoo Examples Datasheets Hardware Sorware Intel OpenVino Open Visual Inferencing and Neural Network Optimization hips://docs.openvinotoolkit.org
- 61. UP Square Board + AI Vision X Dev Kit IEI TANK AIoT Dev Kit Dev Kit for Edge Intelligence using Intel Hardware
- 62. Dev Kit: Raspberry Pi + Intel NCS 2
- 63. Image Classification at the Edge What does really happen there? DEMO
- 64. So much easier if we use…
- 65. UP Square Board + AI Vision X Dev Kit Nvidia Jetson Nano Google Coral Dev Board Raspberry Pi 3 + Intel NCS 2 We can use SBC + NNA
- 66. 5-10Wais But…
- 67. For Low Power Use Case Use Microcontroller (MCU) Sipeed Maix K210-powered dual core 64-bit RISC-V 400 MHz With/Without WiFi SparkFun Edge 32-bit ARM Cortex-M4F 48 MHz; 1MB Flash; 384KB SRAM BLE 5 Radio SensorTile STM32L476JG 32-bit ARM Cortex- M4 80 MHz; 1MB Flash; 128KB SRAM Accel, Gyro, Compass, Pressure, Microphone sensor BLE 4.2 Radio
- 68. Let me talk a bit of Sipeed Maix Disclaimer: I have no auliation with Sipeed (yet)
- 69. ML-accelerated processor Optimized deep learning processor based on RISC- V architecture: K210. Kendryte K210: Dual core 64-bit RISC-V 400 MHz KPU: NeuralNet Processor APU: Audio Processor 8 MB SRAM Flexible Programmable IO Array (FPIOA) DVP for camera & LCD FFT, AES, SHA256 Accelerator Development Board / Modules Module: MAIX-I: SoM with K210 inside, power, yash storage, with/without WiFi Dev Boards: Maixduino, Maix Go, Maix Bit, M1 Dock, AI HAT Accessories: LCD, Camera, Binocular Camera, I2S Microphone, Microphone Array Sorware Tools Framework: Maixduino MaixPy K210 FreeRTOS SDK K210 Standalone SDK IDE: VS Code + PlavormIO MaixPy IDE ML tools: nncase: NeuralNet optimization toolkit (from TensorFlow Lite) Suppocs Documentation: hips://maixduino.sipeed.com/en/ hips://maixpy.sipeed.com/en/ Examples & reference implementation Datasheets MaixHub (Models Hub) Forum & Telegram Group Hardware Sorware Sipeed Maix RISC-V-based hardware and sorware plaMorm for AI and IoT
- 70. Sipeed Maix Go - K210-based AI Board hps://www.seeedstudio.com/Sipeed-MAix-GO-Suit-for-RISC-V-AI-IoT-p-2874.html
- 71. Yes, DycodeX has it. One of the zrst in the world receiving the board Grove AI HAT
- 72. 0.3-0.5Wais Better yet… Idle to full power inference 0.03 wa for sleeping mode
- 73. Back to Image Classixcation Demo Choosing Network Architecture for Image Classixcation
- 74. KPU DVP Image Classification using Maix K210 Processor Camera Sensor MobileNet Model CPU Classixcation Result Object MaixPy (MicroPython) But, 17MB is still too big!
- 75. ML Model Pipeline for K210 Compress (toco) Optimize (nncase) Train (on your awesome machine or Cloud) .h5 .pb .Mlite .kmodel Copy to MCU via SD Card, or Burn to its Flash Inference (KPU library) Inference Result
- 76. ML Model Pipeline for AI at The Edge - in General Optimize (compress, remove, replace) Train (on your awesome machine or Cloud) .h5 .pb .cazemodel ONNX Intermediate Representation File Transfer IR zle to Edge device Inference (Edge-optimized Inference library) Inference Result
- 77. OK, show me real product
- 78. AIoT-powered Smart Trash Management Platform Introducing... by DycodeX
- 79. Sorting Plastic and non-plastic trash using AI-Powered machine vision Compacting Optimizing trash storage Reward Calculate reward point based on trash type Notify & Monitoring Let the waste management official knows upon full capacity or other conditions What it can do
- 80. Compactor Sorting Compartment Can “mouth” Touch Display Proximity sensor Camera Object insertion sensor Sensors (level, smoke) Sorter mechanic Replaceable trash bin Raspberry Pi + Intel NCS 2 AITrash Can - Internal
- 82. NeuralNet Accelerator (Intel NCS 2) Compute Camera Sensor CPU (Raspberry Pi) Other Sensors (Level, Proximity, Smoke, …) Connectivity (LongRange:NB-IoT/CellularorLoRa, ShocRange:WiFi) Display Cloud Ingestion Visualisation User Apps Devices & Firmware Mgt. Processing, Trigger & Alec Edge Firmware OpenVino AI Model SDK & OS Sorware Baery Power Management Power Backhaul AI Model Mgt. App Server Mechanical System Architecture
- 83. AITrash Cloud Payment Gateway End-user Experience AITrash Can 1. Throw a trash 2. Camera captures trash photo & AI classizes trash 3. Submit Trash data (type, dimension, …) 4. Calculate reward 5. Reward6. Reward7. Display QR Code (to redeem reward using Mobile App) Plastic Glass Other
- 84. Batik Analyzer Authentic Batik is made by wriien or stamped process. It’s not the paiern. DycodeX works closely with Balai Besar Kerajinan & Batik to collect Dataset and train ML model. We also developed the mobile apps, and specialised hardware It’s for internal tool (for now), not on App Store ML model successfully achieved 87% accuracy. Still needs improvement Video: hps://youtu.be/tlBV1sFLLo0
- 85. AI at the Edge for non-image data?
- 86. NeuralNet Accelerator Compute Low Power MCU IMU Sensor Firmware Custom ML Model Sorware LPWA Body Temperature Sensor AIoT on SMARTernak Connectivity (NB-IoT / LoRa) Predicted activity (standing, lying-down, feeding, and more) App Movement captured by Inefial Measurement Unit (IMU) & other sensor inside caile-wearable device Web Socket 5D 1B 16 56 01 41 …(1080 B) Detect & learn caile behaviours based on sensor data with the help of AI
- 87. NeuralNet Accelerator Compute Low Power MCU IMU Sensor Firmware Custom ML Model Sorware LPWA Body Temperature Sensor AIoT on SMARTernak Connectivity (NB-IoT / LoRa) Predicted activity (standing, lying-down, feeding, and more) App Movement captured by Inefial Measurement Unit (IMU) & other sensor inside caile-wearable device Web Socket 5D 1B 16 56 01 41 …(1080 B) 0A 0B 01 (3 Bytes) Detect & learn caile behaviours based on sensor data with the help of AI
- 88. NeuralNet Accelerator Compute Low Power MCU IMU Sensor Firmware Custom ML Model Sorware LPWA Body Temperature Sensor AIoT on SMARTernak Connectivity (NB-IoT / LoRa) Predicted activity (standing, lying-down, feeding, and more) App Movement captured by Inefial Measurement Unit (IMU) & other sensor inside caile-wearable device Web Socket 5D 1B 16 56 01 41 …(1080 B) 0A 0B 01 (3 Bytes) { “cattle_id”: 94, “act”, “standing”, … } LPWA = Low Power Wide Area Network Payload length: ~200 - 1600 bytes Detect & learn caile behaviours based on sensor data with the help of AI
- 89. Let’s dive into a feature: On-device Activity Prediction Demo Video: http://bit.ly/smrtrnk-ai-3
- 90. Artificial Intelligence of Things
- 91. AIoT
- 93. Intel OpenVino: Docs: hips://docs.openvinotoolkit.org/ Model Zoo: hips://github.com/opencv/open_model_zoo Implementations: hips://soNware.intel.com/en-us/smaf-video Sipeed Maix: Boards: hips://www.seeedstudio.com/tag/MAIX.html MaixPy: hip://github.com/sipeed/MaixPy DycodeX: SMARTernak: hips://smafernak.com ESPectro32: hips://shop.makestro.com/product/espectro32-v2/ Other products & solutions: hips://dycodex.com Azure: Azure Machine Learning: hips://azure.microsoN.com/en-in/services/machine-learning- service/ Contact me: andri@dycodex.com hip://github.com/andriyadi hips://www.slideshare.net/andri_yadi/ Call to Action
- 95. makestroid makestroid makestroid makestro.com An Indonesia Platform for Maker: to “democratize” knowledge, hardware kit, and software to help makers to start making in hardware, to drive into Internet of Things Start Making at Learning Hardware Marketplace Software & Cloud Community RIoT is Makestro’s program for nurturing IoT makers, officially supported by Indonesia’s Ministry of ICT
- 97. Want to put “AI” in “BrAIns”? or invest on AIoT?
- 98. hi@dycodex.com | https://dycodex.com AIoT & Maker Movement enabler Bandung, Indonesia Keep in touch