An Exploration of 3 Very Different ML Solutions Running on Accumulo
Download as PPTX, PDF1 like391 views
The document presents an exploration of three distinct machine learning solutions utilizing Accumulo, focusing on their application in supply chain risk management, cyber monitoring, and forensic document search. It details the structure and functionality of Accumulo tables, including record and index tables, and highlights the importance of composite indexes and automated processes for improving efficiency and scalability in data handling. The use cases demonstrate diverse interaction patterns and requirements for machine learning implementations while leveraging Accumulo's data storage and retrieval capabilities.
An Exploration of 3 Very Different ML Solutions Running on Accumulo
- 1. An Exploration of 3 Very Different ML Solutions Running on Accumulo By Gadalia O’Bryan and Aaron Cordova, presented by Don Miner
- 2. • Introduction • Koverse Accumulo Table Structures • Supply Chain Risk • Cyber Monitoring • Forensic Document Search • Questions ©Koverse 2 Talk Outline
- 3. • Our customers have an appetite for building very diverse ML solutions on Accumulo • These solutions require varying interaction patterns with Accumulo • We have found we are able to support these use cases using the same set of Accumulo table structures ©Koverse 3 Introduction
- 4. ©Koverse 4 Koverse Accumulo Table Structures
- 5. Record Table: Objectives • Store records under a unique ID • Optimized for reading newly written records in time order • Bucket ID is prepended to distribute the newly written records evenly across tablet servers • Also supports fetching records that match query criteria after consulting index table ©Koverse
- 6. Record Table: Key Components Record ID Bucket ID Dataset ID Timestamp 000 Dataset A 1539458936 000 Dataset A 1539458937 000 Dataset B 1539458931 000 Dataset B 1539458933 001 Dataset A 1539458932 001 Dataset A 1539458935 001 Dataset B 1539458930 001 Dataset B 1539458932 ©Koverse
- 7. ©Koverse Record Table: Organization Bucket 00 Bucket 01 Dataset A Dataset B Dataset A Dataset B Time ordered
- 8. ©Koverse Record Table: Ingest Bucket 00 Bucket 01 Dataset A Dataset B Dataset A Dataset B Writes to the ends of many buckets
- 9. ©Koverse Record Table: Bulk Reading Bucket 00 Bucket 01 Dataset A Dataset B Dataset A Dataset B Sequential reads from many buckets Process newest batches incrementally
- 10. ©Koverse Index Table: Objectives • Store a value-to-record ID pairs • Support point queries and range queries on values in specific fields, or ’any’ field • Set intersection is done by query client, comparing sorted record IDs for each criterion, so matching record IDs do not need to fit in memory
- 11. ©Koverse Index Table Index Entry Dataset ID Field Type Value Record ID Dataset A $any N 45 000_Dataset_A_1539458936 Dataset A $any N 63 003_Dataset_A_1539458929 Dataset A $any S Bob 000_Dataset_A_1539458936 Dataset A Age N 45 000_Dataset_A_1539458936 Dataset A Age N 63 003_Dataset_A_1539458929 Dataset A Name S Bob 000_Dataset_A_1539458936 Dataset B $any S B0001 001_Dataset_B_1539458920 Dataset B Order ID S B0001 001_Dataset_B_1539458920
- 12. Index Table: Querying Dataset A $any N 45 000_Dataset_A_1539458936 Dataset A $any N 63 003_Dataset_A_1539458929 Dataset A $any S Bob 000_Dataset_A_1539458936 Dataset A Age N 45 000_Dataset_A_1539458936 Dataset A Age N 63 003_Dataset_A_1539458929 Dataset A Name S Bob 000_Dataset_A_1539458936 Dataset B $any S B0001 001_Dataset_B_1539458920 Dataset B Order ID S B0001 001_Dataset_B_1539458920 SELECT * FROM TABLE WHERE AGE > 40 AND AGE < 70 AND $any CONTAINS ‘Bob’ ©Koverse
- 13. ©Koverse Index Table: Querying Dataset A $any N 45 000_Dataset_A_1539458936 Dataset A $any N 63 003_Dataset_A_1539458929 Dataset A $any S Bob 000_Dataset_A_1539458936 Dataset A Age N 45 000_Dataset_A_1539458936 Dataset A Age N 63 003_Dataset_A_1539458929 Dataset A Name S Bob 000_Dataset_A_1539458936 Dataset B $any S B0001 001_Dataset_B_1539458920 Dataset B Order ID S B0001 001_Dataset_B_1539458920 SELECT * FROM TABLE WHERE AGE > 40 AND AGE < 70 AND $any CONTAINS ‘Bob’
- 14. ©Koverse Index Table: Querying SELECT * FROM TABLE WHERE AGE > 40 AND AGE < 70 AND $any CONTAINS ‘Bob’ Batch Scan to fetch records Bucket 00 Bucket 01 Dataset A Dataset B Dataset A Dataset B Matching Record IDs Index Table Record Table
- 15. Composite Indexes • Entries in the index table are for a single value found in a field • Record IDs that have a particular value are stored together in sorted order • Spanning a range of values means the Record IDs are no longer totally ordered, so we can’t do streaming set intersection Value Record ID Bat 002 Bat 004 Bat 005 Bask 001 Bask 009 For Bat Record IDs are sorted For Ba* Record IDs are NOT sorted ©Koverse
- 16. Composite Indexes • Composite Indexes allow us to query for multiple ranges of values • E.g. querying for points that fall in a specific latitude longitude box, or query for a time range and a range of port numbers • Basically, enables apps to query a multi- dimensional range ©Koverse
- 17. ©Koverse Composite Indexes Data workers specify which fields to include in a composite index Record ID Age Height 001 14 67 002 23 72 003 13 60 004 64 68 Composite Value (interleaved bytes) Record ID 1647 001 2732 002 1630 003 6648 004
- 19. ©Koverse Composite Index Height Age Age > 30 AND Age < 50 AND height > 50 AND height < 70
- 20. ©Koverse Composite Index Height Age Age > 30 AND Age < 50 AND height > 50 AND height < 70 True Positives False Positives
- 21. ©Koverse 21 Supply Chain Risk
- 22. • PricewaterhouseCoopers works with clients to evaluate risk in their supply chain • E.g., vendors with unethical business practices, based out of sanctioned countries, history of tainted products, etc. • Until now, analysis was very manual • Could only evaluate each vendor every few years • Not enough bandwidth to evaluate vendors’ vendors, or their vendors’ vendors ©Koverse 22 Supply Chain Risk Use Case
- 23. • Automatically evaluate vendor risk on a daily basis • Chain through arbitrary levels of vendors in the supply chain • Incorporate social media ML text analysis ©Koverse 23 Automated ‘Know Your Vendor’ Solution !
- 24. • Storage of various record schemas from Excel files, databases, webservices, and social media • Incremental batch processing to refresh results on a daily basis ©Koverse 24 Accumulo Table Features Leveraged
- 26. • Fully managed service provided by a multinational cybersecurity company • Threat monitoring, detection and mitigation • The use of Accumulo allowed the company to scale their application, which had previously been built on PostgreSQL • Security features of Accumulo allow the managed service to be multi-tenant ©Koverse 26 Managed Cyber Security Services Use Case
- 27. • Streaming writes of cyber logs using Accumulo batchwriters • Bulk threat detection analytics on time- windowed event data • Aggressive use of indexing, including composite indexes, to enable scalable log search on single terms and multiple ranges ©Koverse 27 Accumulo Table Features Leveraged
- 28. ©Koverse 28 Forensic Document Search
- 29. • Investigation team at a large pharmaceutical company • Analysts need to search for and retrieve all relevant documents related to a case • Many users access the application on their mobile devices • Documents come from personal laptops, databases, email attachments, shared drives, Sharepoint, etc. • OCR allows for search on evidence photos that contain text ©Koverse 29 Forensic Document Search Use Case
- 30. • Storage of various record schemas resulting from differing document formats and metadata • Indexing of all terms from document text to enable term search • Incremental batch NLP analytics on raw document records ©Koverse 30 Accumulo Table Features Leveraged