Presto/Accumulo: Lessons Learned
0 likes247 views
The document discusses lessons learned from the Presto-Accumulo connector used in production for over 18 months, highlighting both successes and pain points. Key improvements include enhanced support for data deletion and updates, optimizations for index retrieval, and features like query archiving and handling of visibility and timestamp information. Additionally, challenges with large ad-hoc queries and index design are addressed, emphasizing the need for better resource management and indexing strategies.
Presto/Accumulo: Lessons Learned
- 3. Outline • Presto-Accumulo Overview • Use Case Review • Lessons Learned and New Features Datacatessen
- 4. Presto-Accumulo Review • Open-source and built by Facebook – MPP OLAP engine with pluggable storage • ANSI SQL for NoSQL • Aim is to accelerate relational OLTP use cases by abstracting away common Accumulo design patterns • Load data using SQL or Java • Supports predicate pushdown via advanced indexes and metrics • Queries ranging from milliseconds to seconds • Available since Presto 0.153 – See https://github.com/bloomberg/presto for the latest features Datacatessen
- 5. Client Coordinator WorkerWorker Worker Worker Accumulo Coordinator leverages indexes and optimizations to gather Ranges to scan Each worker is given a subset of the Ranges to read from Accumulo in parallel via BatchScanners Workers pull data from Accumulo, converting it into Presto’s internal object model Accumulo’s job is done, Presto takes over to shuffle data as needed and complete the query Presto/Accumulo Workflow Datacatessen
- 6. Bloomberg Use Case • Surveillance application for Compliance Officer to review events • Web application uses JDBC to execute SQL queries against Presto • Ingest is done via Storm topology using the PrestoBatchWriter • Use case heavily relies on event time index to retrieve any recent events • Query performance ranges from 10 milliseconds to ~10 seconds for most common queries with table size in TBs and thousands of tablets • Presto deployed on Mesos Datacatessen
- 8. Dropping/Updating Data • No trivial way to age off or update data using the PrestoBatchWriter – Delete mutations were ignored – Updating was manual as indexes and metrics needed to be dropped/decremented – Unable to use AgeOffIterator Datacatessen
- 9. Dropping/Updating Data • Led to API improvements within the PrestoBatchWriter to delete and update – Supports delete mutations – Properly handles deleting index entries, decrementing metrics entries, and creating new index/metric entries – Explicit API calls to change the values of columns • Want to implement DELETE so we can drop data via SQL • LL: Need a clean API to delete stuff Datacatessen
- 10. Arbitrary Row Batching • Connector packs 50,000 row IDs into a split – Split is the unit of parallelism in Presto – Resulted in a variable number of splits, frequently creating too few splits and not leveraging the distributed nature of Presto – Caused problems with concurrent queries over larger data sets Datacatessen
- 11. Arbitrary Row Batching • Created a formula for determining the number of splits per batch (with min/max) – Number of splits is a function of desired number of parallel queries and number of concurrent splits per worker • r: number of rows to be scanned • s: splits per worker • w: number of workers • r / s / w • LL: Need to properly generate splits for maximum parallelization Datacatessen
- 12. Bottleneck in Index Retrieval • Connector would regularly spend several seconds fetching row IDs from the index – Process was single-threaded and non-distributed – Regularly retrieving more rows than necessary due to rows being filtered via predicates Datacatessen Row IDs where user='adam' Row IDs where date='2017-10-16' Row IDS where user='adam' AND date='2017-10-16'
- 13. Bottleneck in Index Retrieval • Three new features/optimizations – ThreadPool to fetch row IDs in parallel – Composite indexes – Distributing the index lookup to Workers • LL: More parallelism and more indexes make faster queries Datacatessen 'alice' and 'wendy' 'alice' and 'erin’ 'erin' and 'olivia' 'oscar' and 'wendy' From Coordinator To Worker
- 14. No Query History • Presto’s Coordinator regularly purges past queries, and they are lost on a restart of the Coordinator – Presto released an EventListener API that will provide metrics and metadata about a query to all implementors Datacatessen
- 15. No Query History • Implemented an AccumuloEventListener that archives all queries in a Presto-Accumulo table – Table is queryable via SQL – Very helpful in generating usage reports – History is persisted between restarts • LL: Need proper storage of query history Datacatessen
- 17. No Visibility or Timestamps • Added support for hidden visibility and timestamp columns – Doesn’t clutter up the forward facing DDL – Get them for ‘free’ for all non-row ID columns (don’t have to explicitly define them) – Available via SELECT • <column_name>_vis • <column_name>_ts • LL: Need to expose visibility and timestamp information Datacatessen
- 18. Large Ad-hoc Queries • “Big” scans (millions of rows) are causing problems – Occupying scan threads on TabletServers – Blocked other queries that needed near real-time responses Datacatessen
- 19. Large Ad-hoc Queries • Still not really solved today – We reject queries that will scan more than 50 million rows – Been brain storming ideas such as reading Rfiles • LL: Don’t use the Accumulo connector for OLAP queries Datacatessen
- 20. Index Hotspotting • Indexing 1.0 was a quick win – Basic reverse index – Caused all of the problems you want to avoid with indexes – Low cardinality columns caused very wide rows – Timestamp columns are monotonic increasing – Key distribution was all over the place due to different data types in the same table – Deleting columns requires configuring RegexFilters and compacting/merging the tables Datacatessen
- 21. Index Hotspotting • Large refactoring effort for Indexing 2.0 – Split index table into one table per indexed column – Merged metrics into index table (separate locality group) – Added configurable IndexStorage methods that would encode/decode to shard or post-fix random data • LL: Support multiple index strategies based on the type of data being stored Datacatessen
- 22. Feature Summary • PrestoBatchWriter improvements (deletions) • Smarter resource allocation • Composite Indexes • Accumulo query archive (ROW types) • Distributed Index Lookup • Hidden visibility and timestamp columns • Error on reading too much data • Index hotspotting Datacatessen