Build DynamoDB-Compatible Apps with Python
0 likes754 views
The document discusses a virtual event focused on building applications compatible with DynamoDB using ScyllaDB, featuring presentations by Guilherme Nogueira and Nadav Har'el. It highlights ScyllaDB's performance benefits, including significant throughput and latency improvements over legacy NoSQL databases, as well as the introduction of the ScyllaDB Alternator, which provides a DynamoDB-compatible API. The event also covers practical applications, cloud deployment options, and tools for monitoring and managing ScyllaDB instances.
Build DynamoDB-Compatible Apps with Python
- 1. Guilherme Nogueira — Solutions Architect - NA and LATAM, ScyllaDB Nadav Har'El — Distinguished Engineer, ScyllaDB Building Run-Anywhere DynamoDB-Compatible Applications with Python
- 2. Brought to you by VIRTUAL EVENT | OCTOBER 19 + 20 All Things Performance The event for developers who care about P99 percentiles and high-performance, low-latency applications. Register at p99conf.io
- 3. Poll Where are you in your NoSQL adoption?
- 4. Guilherme Nogueira — Solutions Architect - NA and LATAM, ScyllaDB Nadav Har'El — Distinguished Engineer, ScyllaDB Building Run-Anywhere DynamoDB-Compatible Applications with Python
- 5. Presenters Guilherme Nogueira Guilherme is a Solutions Architect at ScyllaDB, assists customers in designing data models that delivers ScyllaDB's potential. Geek, passionate about Linux and open source software. Helps family and friends adopt OSS. Nadav Har'El Nadav has had a diverse 25-year career in computer programming and computer science. In the past he worked on scientific computing, networking software, information retrieval, virtualization and operating systems. Today he works on Scylla, and among other things led the Alternator development.
- 6. Agenda Objectives What is ScyllaDB Intro to ScyllaDB Alternator Sample Application That Runs Anywhere Libraries Used Application Demo ScyllaDB Monitoring
- 7. + Introduce ScyllaDB Alternator + Capabilities, Monitoring + Demonstrate how an application using DynamoDB API can run anywhere + We updated an Amazon demo app to the latest libraries + Free to run in GCP, Azure, Docker Objectives 7
- 9. + NoSQL database, drop-in replacement for Cassandra and DynamoDB. The Database Built for Gamechangers 9 One of 12 winners of 2020 Technology of the Year awards, “identifying the best and most innovative products in the IT landscape”. “ScyllaDB stands apart...It’s the rare product that exceeds my expectations.” – Martin Heller, InfoWorld contributing editor and reviewer “For 99.9% of applications, ScyllaDB delivers all the power a customer will ever need, on workloads that other databases can’t touch – and at a fraction of the cost of an in-memory solution.” – Adrian Bridgewater, Forbes senior contributor + Superior performance over legacy NoSQL + >5x higher throughput at same cost + >20x lower latency + >75% TCO savings + Avoids vendor lock-in + ScyllaDB can run anywhere. + Also available as DBaaS on AWS and GCP. + Open-Source version available.
- 10. 10 +400 Gamechangers Leverage ScyllaDB Seamless experiences across content + devices Fast computation of flight pricing Corporate fleet management Real-time analytics 2,000,000 SKU -commerce management Real-time location tracking for friends/family Video recommendation management IoT for industrial machines Synchronize browser properties for millions Threat intelligence service using JanusGraph Real time fraud detection across 6M transactions/day Uber scale, mission critical chat & messaging app Network security threat detection Power ~50M X1 DVRs with billions of reqs/day Precision healthcare via Edison AI Inventory hub for retail operations Property listings and updates Unified ML feature store across the business Cryptocurrency exchange app Geography-based recommendations Distributed storage for distributed ledger tech Global operations- Avon, Body Shop + more Predictable performance for on sale surges GPS-based exercise tracking
- 11. ScyllaDB Cloud + Database as a Service on AWS and GCP + Fully managed NoSQL DB clusters + 24*7 monitoring, support and response teams + Secured, private and dedicated cluster + Clusters are multi-zone by default + Finding the right fit/solution - more than 20 instance types across 39 regions + Geo redundancy
- 13. ScyllaDB Alternator ScyllaDB started with Cassandra®’s APIs: CQL & Thrift. Alternator: added a DynamoDBTM -compatible API to ScyllaDB. Why a DynamoDB API? + Popularity growing, closing in on Cassandra. + ScyllaDB solves real AWS lock-in problem. + DynamoDB apps can now be developed, or run, anywhere, and at cheaper cost. + DynamoDB API is close in features to CQL. DynamoDB Cassandra 2013 2022 DB-Engines popularity ranking (log)
- 14. ScyllaDB Alternator + Alternator is part of ScyllaDB open-source and Enterprise editions - + ScyllaDB listens for DynamoDB API requests in addition to CQL. + Same installation, setup, etc. just run with extra options: + alternator_port and/or alternator_https_port alternator_write_isolation (choose how read-modify-write operations are isolated) + Available in ScyllaDB Cloud since June 2020.
- 15. Alternator vs. DynamoDB + Alternator goal: full compatibility with DynamoDB API. + Support unmodified or mostly-unmodified applications designed for DynamoDB. + But run them anywhere that ScyllaDB can run - avoid vendor lock-in! + Already mostly compatible except some unimplemented features listed in docs/alternator/compatibility.md. + Authentication supported but not access control (IAM policies). + GSI supported but can’t be added after the table’s creation. + “Projection” parameter of GSI/LSI is not honored - all base table attributes are available in the index. + All of Alternator’s tables are “Global Tables”, i.e., same on all DCs. + DynamoDB’s Backup APIs not supported - use Scylla’s backup tools instead. + DynamoDB’s new multi-item “transactions” feature is not supported. + Kinesis Streams API not supported (but we do support DynamoDB Streams). + New PartiQL syntax not supported. Use CQL if you want an SQL-like syntax. + Export to S3.
- 16. Alternator vs. DynamoDB Continued + Like ScyllaDB, Alternator is a dedicated installation, not multi-tenant service like DynamoDB: + Cluster is dedicated to one customer. + DynamoDB users are not aware of individual nodes, Scylla users are - + We’ll discuss load balancing in a few minutes. + Powerful monitoring framework from ScyllaDB (Prometheus, Grafana) - + Understand your workload’s performance and get insights on how to improve it.
- 17. Trying Alternator To try Alternator quickly, you can: + Create an Alternator cluster on ScyllaDB Cloud + Run one Alternator node on your machine in 5 minutes, using docker: docker pull scylladb/scylla:latest docker run --name scylla -d -p 8000:8000 scylladb/scylla:latest --alternator-port=8000 --alternator-write-isolation=always
- 18. Migrating from DynamoDB to Alternator + Install ScyllaDB or get a managed cluster on ScyllaDB Cloud + Tell your application, written to use DynamoDB, to use Scylla’s nodes (see “load balancing”) + Migrate existing data from DynamoDB to ScyllaDB using the DynamoDB API + E.g. ScyllaDB migrator: https://www.scylladb.com/2021/11/04/migrating-dynamodb-workloads-from-aws-to-google-cloud-si mplified-with-scylladb-alternator/
- 20. Why Python + Multiplatform, available by default on most Linux distributions + Simplicity + Huge list of libraries available + Extensive use in data-science
- 21. Run Anywhere + Run your application anywhere using ScyllaDB Alternator + Any cloud-provider; + On-prem; + On your CI/CD pipeline; + On your dev machine; + Fully featured installation + Get the exact same features in any of the deployment options;
- 22. Tic-Tac-Toe Example + Tic-tac-toe sample using the boto3 framework; + Code is licensed under the Apache-2.0 and initially written by Amazon; + Works out of the box with Alternator; + Online web multiplayer game; Code is available on: https://github.com/hopugop/dynamodb-tictactoe-example-app
- 23. Application Structure / ├─ application.py ├─ config ├─ models/ │ ├─ game.py ├─ dynamodb/ │ ├─ setupDynamoDB.py │ ├─ connectionManager.py │ ├─ gameController.py ├─ static/ │ ├─ ... ├─ templates/ │ ├─ ...
- 25. Python Has Built in Batteries… But that’s not always enough! In this project, we used the following additional libraries: + boto3 + flask + alternator_boto3 (ScyllaDB-provided)
- 26. Library: flask + Micro web framework for Python; + It does not require particular tools or libraries; + No database abstraction layer, form validation, etc; + Extensions to add functionalities, + Chosen for its lightweight footprint and very easy configurability
- 27. Library: boto3 + Interact with several AWS service APIs + Built on top of botocore + core functionality to access to different types of AWS services + Not restricted to DynamoDB. Actually many services are supported: + S3 + Lambda + EC2 + CloudWatch + And many more… + Our example uses the DynamoDB API only
- 28. alternator_boto3 (ScyllaDB) + Not an actual library; + A library in a file; + Sample code to build a resilient connection against an Alternator cluster; + Live node detection and connection balancing; We will circle back to this library in a moment… See https://github.com/scylladb/alternator-load-balancing
- 30. DynamoDB vs Alternator Load-balancing + DynamoDB is a fully managed solution; + Load balancing is provided by AWS; + Alternator is self-hosted or as a managed solution; + Multi-node cluster architecture, no leaders or main contact points; + Load-balancing flexibility: + Deploy a load balancer between client and server; + Client-side balancing;
- 31. Alternator Local Nodes API + Exposes an API for checking cluster nodes + High-availability workload balancer for the cluster nodes ScyllaDB boto3 alternator load balancer + ScyllaDB-provided sample + Provides balanced connectivity to boto3 programs + Background thread ensures node list is updated
- 32. Alternator Client-side Balancing + Previously-mentioned sample alternator_boto3 library-in-a-file; + Implemented on client-side; + Checks for live nodes in a set interval; + Updates connection string for a live node; + Random selection of nodes to balance out requests; + Ensure only live nodes are tried; + Not token/shard aware;
- 33. Let’s Dive In 33
- 35. Observability in Alternator + Monitoring stack + Prometheus collects metrics + Grafana visualization + Alert manager, Advisor + Hundreds of different metrics. + Counters, rates, histograms, etc. + Total, per node, per shard. 35 Just a small part of the many detailed dashboards in Scylla’s Grafana UI
- 36. Throughput and Latency During Stress YCSB write-only stress test A 3-node cluster Each of the 3 nodes reaching 100% load (CPU utilization)
- 37. How many DynamoDB API requests of each type are served per second? Per-instance (node) graph: About 25,000 PutItem requests per second handled by each node Can also see total of entire cluster: About 75,000 PutItems per second.
- 38. In addition to rate of each operation, we can see its latency PutItem throughput (again) - 75,000 per second Average PutItem latency - 7ms 95th percentile latency - 11ms 99th percentile latency - 12ms Because the cluster is 100% loaded in this example, high latency is expected, but we can see it’s still not unacceptably high.
- 40. Poll How much data do you have under management of your database?
- 41. Q&A WANT TO KEEP LEARNING? Join ScyllaDB University for Free: university.scylladb.com SCYLLADB VIRTUAL WORKSHOP Getting Started with ScyllaDB 29 September, 2022, 12PM GMT | 8 AM ET | 5:30 PM IST
- 42. Thank you for joining us today. @scylladb scylladb/ slack.scylladb.com @scylladb company/scylladb/ scylladb/