NoSQL for Artificial Intelligence
2 likes1,218 views
The document discusses the significant impact of NoSQL databases on big data, machine learning, and artificial intelligence, illustrating the limitations of traditional relational databases and the advancements brought by NoSQL. It further explores various applications of NoSQL, including key-value stores, document stores, and search engines, emphasizing their benefits in handling increased data volume, velocity, and variety. Additionally, the text explains the fundamentals of artificial intelligence and machine learning, detailing how NoSQL facilitates their development and deployment.
NoSQL for Artificial Intelligence
- 1. NoSQL for Artificial Intelligence How NoSQL Fundamentally Changed Big Data, Machine Learning and Artificial Intelligence
- 3. Who am I? Sebastián Ramírez Chief Data Scientist at Datum Consultants Boutique Data Science company specialized in Artificial Intelligence
- 4. Two different topics, "joined" at the end NoSQL and Big Data
- 5. NoSQL and Big DataTwo different topics, "joined" at the end Artificial Intelligence
- 6. NoSQL and Big DataTwo different topics, "joined" at the end Artificial Intelligence NoSQL for AI
- 7. How relational ("standard") databases work What was missing, enter "Big Data" The different evolutions of Big Data What we'll see: NoSQL and Big Data
- 8. What is Artificial Intelligence (AI) and Machine Learning (ML) What can be done with AI and ML How AI and ML work (visually, no math) What we'll see: Artificial Intelligence
- 9. What we'll see: NoSQL for AI NoSQL and Big Data Artificial Intelligence NoSQL for AI
- 11. Entities
- 13. RDBMSs features ● Uses as little disk as possible (it was expensive in the 70s) ● Simple and very popular query language (SQL) ● Data consistency (enforced with normalization)
- 14. RDBMSs restrictions ● All data on a single machine, doesn't scale ● Fixed schema from start, difficult to change (add columns) ● Reads and writes are expensive ● Difficulties with fast I/O ● Difficulties with big datasets, even more analytics ● Any failure is catastrophic
- 15. RDBMS: Examples
- 16. Big Data None of these work well with traditional RDBMS
- 18. Big Data: Batch Processing MapReduce ● Distributed ● Parallel ● Fault tolerant ● Batch, data processing
- 19. Big Data: Batch Processing Released in 2005 Open Source MapReduce implementation HDFS (Hadoop Distributed File System)
- 20. Big Data: Distributed Batch Processing ● Distributed ● Batch analytics (processing all the data together) ● Aggregations (avg, max, min) ● Takes long, finishes at some point ● Fault tolerant
- 22. Big Data: Distributed, in-memory, batch processing In-memory first Distributed batch processing MapReduce (as with Hadoop) ...and other algorithms and tools
- 23. Big Data: Distributed, in-memory, batch processing Same features as Hadoop: ● Distributed ● Parallel ● Fault tolerant ● batch, data processing ● MapReduce Plus: ● In-memory first, faster than Hadoop ● Other algorithms and tools apart from MapReduce
- 25. Big Data for applications: NoSQL, key-value stores
- 26. Big Data for applications: NoSQL key-value stores ● Distributed, parallel, fault tolerant, etc ● Minimal latency ● Read / Write individual records ● Know the IDs of each record ● Not complex queries ● Denormalization, duplicate for speed ● Data consistency is harder ● Precomputed aggregations (avg, max, min) ● Reads and Writes are cheap ● High volume, velocity, variety
- 27. NoSQL, key-value stores: examples
- 28. Big Data for applications: NoSQL, key-value memory stores
- 29. Same characteristics as key-value stores: ● Distributed, parallel, fault tolerant, etc ● Minimal latency ● Read / Write individual records ● Know the IDs of each record ● Not complex queries ● Denormalization, duplicate for speed ● Data consistency is harder ● Precomputed aggregations (avg, max, min) ● Reads and Writes are cheap ● High volume, velocity, variety Plus: ● Memory first storage, faster Big Data for applications: NoSQL key-value memory stores
- 30. NoSQL, key-value memory stores: examples
- 31. Big Data for applications: NoSQL, Document Stores
- 32. Big Data for applications: NoSQL Document Stores Plus: ● Complex structures (JSON "documents") ● Arbitrary indexes on non-key fields ● Complex queries, by non-key fields ● Some denormalization, much less duplication ● Data consistency is easier than key-value Key-value stores' characteristics: ● Distributed, parallel, fault tolerant, etc ● Minimal latency ● Read / Write individual records ● Precomputed aggregations (avg, max, min) ● Reads and Writes are cheap ● High volume, velocity, variety
- 33. NoSQL, Document Stores: examples
- 34. Big Data for applications: NoSQL, Search Engines
- 35. Big Data for applications: Search Engines Search Engines' characteristics: ● Distributed, parallel fault-tolerant... ● Minimal latency ● Store complex text documents ● Specialized text processing indexes for search ● Copy data from main data store to search engine
- 36. NoSQL, Search Engines: examples
- 37. Big Data for applications: NoSQL, data synchronization
- 38. Big Data for applications: data synchronization Plus: ● Server to server data synchronization ● Edge data synchronization (mobile, IoT) ● Offline-first complex applications NoSQL document stores characteristics: ● Distributed, parallel, fault tolerant, etc ● Minimal latency ● Read / Write individual records ● Precomputed aggregations (avg, max, min) ● Reads and Writes are cheap ● High volume, velocity, variety ● Complex structures (JSON "documents") ● Arbitrary indexes on non-key fields ● Complex queries, by non-key fields ● Some denormalization, much less duplication ● Data consistency is easier than key-value
- 39. NoSQL, data synchronization: examples
- 40. Big Data for applications: NoSQL, with all the toppings
- 41. NoSQL, data synchronization: examples
- 42. NoSQL restrictions ● Extra work when: ○ Duplication is needed ○ Data updates are required ● New way of thinking and designing systems ○ Some extra learning for RDBMS people ● Strict multi-record transactions require more work ○ But in many cases a single record (document) can store the transaction data ○ Like bank transactions
- 43. What is Artificial Intelligence (AI)? Artificial Intelligence Machine Learning Deep Learning
- 44. What is Artificial Intelligence (AI)? Artificial Intelligence Machine Learning Deep Learning
- 45. What can be done with Machine Learning (ML)?
- 46. What can be done with ML?
- 47. What can be done with ML?
- 48. What can be done with ML?
- 49. How Machine Learning works ...explained without math
- 52. ML: K-Means
- 53. ML: K-Means
- 54. ML: K-Means
- 55. ML: K-Means
- 56. ML: datasets for training
- 57. ML: datasets for prediction
- 58. How NoSQL helps AI / ML
- 59. How NoSQL helps AI / ML Data Volume
- 60. How NoSQL helps AI / ML Data Velocity
- 61. How NoSQL helps AI / ML Data Variety
- 62. How NoSQL helps AI / ML ● ML iteration ● Update schemas, queries
- 63. Thank you! Questions? Sebastián Ramírez @tiangolo sebastian@datumcon.com