Azure Cosmos DB - Azure Austin Meetup
- 1. Azure Cosmos DB Globally-distributed, multi-model database service Matías Quaranta - @ealsur
- 2. James Whittaker - @docjamesw “The world is being reduced to data”
- 3. Developing planet-scale apps comes with planet-scale challenges Scaling both throughput and storage based on global demand Delivering highly-responsive experiences Ensuring an always-on system Write accurate, globally distributed apps Managing and versioning complex schemas Balancing the needs for strong and eventual consistency
- 4. Introducing Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Global distribution Automatically replicate all your data around the world – across more regions than Amazon and Google combined
- 5. Introducing Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Global distribution Multi-model + multi API Use key-value, graph, and document with a schema-agnostic service that doesn’t require any schema or secondary indexes KEY-VALUE COLUMN-FAMILY DOCUMENT GRAPH
- 6. Introducing Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Global distribution Multi-model + multi API Elastic scale-out Independently and elastically scale storage and throughput across regions
- 7. Introducing Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Global distribution Multi-model + multi API Elastic scale-out Choice of consistency Choose from five defined consistency levels for low latency and high availability Strong Bounded-stateless Session Consistent prefix Eventual
- 8. Introducing Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Global distribution Multi-model + multi API Elastic scale-out Choice of consistency Serve <10 ms read and <15 ms write requests at the 99th percentile from the nearest region while delivering data globally Guaranteed single-digit latency Read < 1 ms Writes < 6 ms Read < 10 ms Writes < 15 ms 99%50%
- 9. Introducing Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Global distribution Multi-model + multi API Elastic scale-out Choice of consistency Guaranteed single-digit latency Only service with financially-backed SLAs for millisecond latency at the 99th percentile, 99.99% HA and guaranteed throughput and consistency 99.99% HA Throughput Guaranteed Consistency Guaranteed Enterprise-level SLAs <10ms Latency 99th percentile
- 10. Introducing Azure Cosmos DB A globally distributed, massively scalable, multi-model database service Global distribution Multi-model + multi API Elastic scale-out Choice of consistency Guaranteed single-digit latency Enterprise-level SLAs
- 13. DocumentDB API
- 15. Demo time!
- 16. Programming tools API REST Azure Cosmos DB
- 18. MongoDB API
- 19. Keep using the same tools MongoDB wire protocol Azure Cosmos DB API for Mongo
- 21. Demo time!
- 23. Tables API in Azure Cosmos DB Premium experience (low latency, well-defined consistency) Globally Distributed Secondary Indexes for user-defined queries Millisecond latency, Guaranteed throughput Azure Cosmos DB: Table API Azure Storage: Standard Table API Azure Storage SDKs 100% Backwards compatible, Seamless experience Azure Cosmos DB: Table API Azure Storage SDKs
- 24. Demo time!
- 26. Model the real world Relationship as first-class entities Optimized for graph storage & traversal Gremlin standard Gremlin API in Azure Cosmos DB Azure Cosmos DB: Graph API
- 27. Globally distributed, elastically scalable, low latency, auto-indexed service Independently scalable graph engine (using Tinkerpop framework) Gremlin and SQL query languages Native Graph Processing
- 28. Demo time!
- 29. Change Feed Logging and change-tracking patterns
- 30. Change Feed Logging and change-tracking patterns Microsoft.Azure.DocumentDB.ChangeFeedProcessor
- 31. Demo time!
- 32. Security & Compliance Enterprise grade security Encryption at Rest By default Certified Always encrypted at rest and in motion Data, index, backups, and attachments encrypted No impact on performance, throughput or availability Transparent to your application ISO 27001, ISO 27018, EUMC, HIPAA, PCI SOC1, SOC2, FedRAMP, IRS 1075, IL2, HITRUST (coming soon)
- 33. Powering global solutions Field-tested by Microsoft’s planet-scale services and industry-leading enterprises apps Globally-distributed mission-critical apps IoT Accommodate bursts of traffic and deliver low-latency multiplayer experiences Personalization Retail and e-commerce Gaming Guarantee uptime to users worldwide with high-availability and low-latency Scale instantly for uncertain IoT workloads without sacrificing performance Generate personalized service through low-latency and tunable consistency settings Support queries over product catalogs, traffic spikes, and rapidly changing inventory
- 34. Key benefits • Cosmos DB can scale elastically without operational overhead of MongoDB • Perform fast queries over events to deliver recommended services, safety notices to vehicles • Perform staged migration via MongoDB APIs Business need • Need to ingest massive volumes of diagnostic data from vehicles and take real- time actions as part of connected car platform • Management and operations of database infrastructure to handle exponential growth of data 8TB Vehicle Telemetry 250K Lexus Cars Toyota drives connected car push forward with Azure Cosmos DB Azure Cosmos DBAzure HDInsight Storm Azure Storage (archival)
- 35. Business need • Process Ms of retail transactions per second in milliseconds in inventory pipeline during peak (“Black Friday”) • Fast development cycles and loosely coupled micro-services to keep up with a competitive marketplace Key benefits • Cosmos DB provides elastic scalability from 1-10M requests per second • Improved reliability, and faster order processing times than previous OSS solution • Reduced development time and operational overhead Azure Cosmos DB Azure Service Fabric Pricing Service Azure Cosmos DB Azure Service Fabric Inventory Service E-commerce challenger eyes the top spot, runs on Azure Cosmos DB 12TB Provisioned 10M Peak RPS 64 Databases
- 36. Business need • Handle millions of players on Day 1 due to popularity of the TV series • Match-making of players for competitive and lag-free experience • Provide new content weekly, and iterate on social functionality • Key benefits • Cosmos DB provides elastic scalability for millions of users and flexible schema to support social features and gameplay • Global distribution allows for low latency for players spread worldwide • Automatic indexing used to build real- time leaderboards Performance at massive scale allows millions to play mobile game Azure Traffic Manager Azure API Aps (game backend) Azure CDN Azure Cosmos DB Azure Functions Azure Notification Hubs (push notifications) Azure Storage (game files) 1M Peak Active #1 iOS App Store 1B Daily Queries
- 37. Try before you buy Use the Azure Cosmos DB Emulator aka.ms/cosmosdb-emulator
- 38. Try before you buy Try Azure Cosmos DB for free azure.microsoft.com/try/cosmosdb/
Editor's Notes
- #3: Every interaction you make with your phones, computers, watches or sport tracking devices, cars and even some things like smart fridges, generate big volumes of data. Data that represent you as consumers, users and individuals.
- #5: Worldwide presence Automatic multi-region replication Multi-homing APIs Manual and automatic failovers Latency, throughput, consistency, and availability guarantees
- #6: First party and popular third-party OSS APIs Key-value, Document, Columnar, and Graph DocumentDB (SQL and JavaScript), MongoDB, Table, and Gremlin Supported across number of programming languages More APIs to be added
- #7: Pay by the hour, only what you need No partition management, no limits Change throughput at any time, effective in seconds Scale from 100 to 100s of millions of requests/sec across any number of regions Automatically indexed SSD storage Global distribution across Azure regions Automatic expiration via TTL Now supports requests/min to handle spikes cost-effectively
- #8: While most database services force you to choose between strong or eventual consistency, Azure Cosmos DB provides multiple well-defined, intuitive consistency choices – so you can select just the right one for your app. Every globally distributed databases need to navigate the CAP theorem. This is a fact of life, how ever anyone else markets it. We take a unique approach to it by allowing our developers to define the consistency levels their application needs. With well-defined consistency model, you have the choice and control but also have predictability in the trade offs you are making. As a globally distributed database which is designed from the ground up, Cosmos DB has taken a unique approach to navigate the CAP theorem to provide well-defined behavior between the tradeoffs between latency/high availability and consistency.
- #9: Globally distributed with reads and writes served from local region Write optimized, latch-free database engine designed for SSDs Synchronous and automatic indexing at sustained ingestion rates Single-digit millisecond latency at any scale
- #10: Only service with financially-backed SLAs for millisecond latency at the 99th percentile, 99.99% HA and guaranteed throughput and consistency
- #12: Multi-model nature is based on a schema-less storage
- #13: <Explain BW-TREE indexing as lightly and clear as posible>
- #16: Showcasing CosmosDB Data Explorer & Queries
- #20: Migrate your MongoDB data to CosmosDB and keep using the same SDKs and apps; your applications will work just by changing your connection string
- #21: Benefit from Azure CosmosDB’s global distribution and scaling features while using Mongo in your application
- #22: Showcasing Mongo client with CosmosDB connection string
- #24: Today, I have the privilege of announcing that Azure Cosmos DB will support Azure Tables API for a premium experience for Tables customers. You can now use Azure tables and get the same millisecond latency that Azure Cosmos DB offers and you can control the consistency level for your app. Your Azure Tables can automatically be distributed across any Azure region. With automatic indexing, you do not need secondary indexes and all of this is 100% backwards compatible. As a developer your experience is seamless – All it takes is to change a connection string. With support for Tables API, we are brining to you most of the top 10 asks. By GA, We will be bringing an update for existing Azure Tables and seamlessly migrating all of the current Azure Tables workloads to have the same functionality that we are announcing today with the simplicity and low cost of Azure Tables.
- #25: Showcasing Azure Storage Tables SDK with CosmosDB connection string & Azure Portal Data Explorer
- #27: Graph database provides a highly intuitive data model and allows us to to reflect how the world really operates. Graph databases and traversal allows us to find the relational connections between objects and data. In many scenarios that our customers use Azure Cosmos DB, one of the next steps often identified is the need to find these relationships and operate on them. They need to interact with the graph model. With Gremlin API support, we allow you to work with Azure Cosmos DB using the most popular graph language and standards out there.
- #29: Showcasing Azure Portal Data Explorer Graph queries
- #30: Update a cache, search index, or a data warehouse with data stored in Azure Cosmos DB. Implement application-level data tiering and archival, that is, store "hot data" in Azure Cosmos DB, and age out "cold data" to Azure Blob Storage or Azure Data Lake Store. Implement batch analytics on data using Apache Hadoop. Implement lambda pipelines on Azure with Azure Cosmos DB. Azure Cosmos DB provides a scalable database solution that can handle both ingestion and query, and implement lambda architectures with low TCO. Perform zero down-time migrations to another Azure Cosmos DB account with a different partitioning scheme.
- #31: Changes are persistent in Azure Cosmos DB and can be processed asynchronously. Changes to documents within a collection are available immediately in the change feed. Each change to a document appears exactly once in the change feed, and clients manage their checkpointing logic. The change feed processor library provides automatic checkpointing and "at least once" semantics. Only the most recent change for a given document is included in the change log. Intermediate changes may not be available. The change feed is sorted by order of modification within each partition key value. There is no guaranteed order across partition-key values. Changes can be synchronized from any point-in-time, that is, there is no fixed data retention period for which changes are available. Changes are available in chunks of partition key ranges. This capability allows changes from large collections to be processed in parallel by multiple consumers/servers. Applications can request for multiple change feeds simultaneously on the same collection
- #33: As a foundational Azure Service, Security and Compliance is an important aspect of Azure Cosmos DB. As I previously mentioned, several high value Microsoft services that have stringent compliance and security requirements are built on Azure Cosmos DB. All of the data is encrypted at rest and it is enabled by default. We made sure that we continue to provide the performance guarantees even with encryption enabled and this is transparent to your application. The service is HIPAA , PCI, ISO certified and other compliance certifications are in the process with all of them estimated to be completed by next month.