![We’re going to do this with a Postgres extension
MySQL Listener
MySQL
Protocol
Connector/C[++] Connector/J MySQL Connector/NET
SQL/PSM
Compiler/Interpreter
We’re replacing our background worker listener with AWS’ Babelfish Hooks](https://image.slidesharecdn.com/20240419-the-future-of-mysql-is-postgres-240501230925-c72d6944/85/The-future-of-MySQL-is-Postgres-36-320.jpg)
🐬 The future of MySQL is Postgres 🐘
- 1. The Future of MySQL is Postgres Postgres Conference 2024 Jonah Harris & Mason Sharp Breathing New Life into MySQL Apps With Advanced Postgres Capabilities nextgres.com info@nextgres.com
- 2. We’ve been in data management for a while CTO @ MariaDB (NYSE) CTO & Dir. AI/ML @ The Meet Group (NASDAQ) Founding Engineer @ EDB Recognized Oracle Database Expert Longstanding PostgreSQL Contributor Co-Founder, CEO Jonah Harris Co-Founder, CTO Two-Time Distributed Database Entrepreneur VP of Engineering @ MariaDB Sr. Architect & Engineer @ Immuta, FutureWei, TransLattice, EDB PostgreSQL NYC and SV Meetup Co-organizer Mason Sharp
- 3. For the last 10 years, Postgres grew at a steady rate of about 10% per year. Source: DB-Engines PostgreSQL Trend (2013−2023)
- 4. Postgres is winning hearts and minds Developers love PostgreSQL for its stability, advanced features, and strong community support, making it their go-to database for modern applications. It’s Loved By Developers It’s Trusted By Businesses It’s Backed By Investors Businesses trust PostgreSQL for its proven reliability, security, and high-performance data management, making it a go-to solution for enterprise needs. Investors back PostgreSQL as the optimal foundation for high-growth database ventures, prizing it for its robustness, extensibility, and vibrant ecosystem.
- 5. Despite Postgres’ impressive growth, MySQL remains the most popular open source database. Source: DB-Engines Complete Ranking
- 6. MySQL remains a top choice for web dev It’s the #1 Open Source Database It’s Essential To Large Businesses It’s Heavily Used By Open Source Apps
- 7. The market supports MySQL’s large user base Official Non-Official On-Prem Cloud
- 8. While this success was hard-won, work remains to be done.
- 9. Postgres has advanced features, but needs to improve some of the basics Limited Visibility Tuning Difficulty External Dependencies Vacuuming Perf Degradation System Stalls Secondary Lookups Large Objects Bloat Storage Wraparound Replication & High Availability Limited Built-in Options Complex Setup Automated Failover Internal Instrumentation A Land of Commercial Opportunities
- 10. MySQL gets ease of use right, but is hampered by its architecture Large Table Performance Single-Threaded Design Materialized Views Limited Plug-ins Storage Engine Support Sparse Extension Docs Query Optimization Resource Contention Limited Concurrency Extensibility Mixed Workloads Advanced Features Limited Indexing Basic Partitioning Analytical Functions Large Databases
- 11. Each database has unique strengths and can learn from the other.
- 12. Postgres is the primary database enterprises migrate to from commercial systems Each database prioritizes different things Beginner Friendly & Extensive Documentation Read-Mostly Operations & Simple Transactions Web-Based Applications & Simpler Workloads Advanced Features & SQL Compliance Complex Read/Write Operations & Large Transactions Enterprise Applications & Complex Workloads 90% of the world’s top websites are powered by MySQL
- 13. Their ecosystems are quite different Entrenched Applications In Decline No Development Community Recent Advancements Only In Non-OSS & Cloud (HeatWave) Larger Ecosystem On The Rise Large Multi-Source Development Community Large Advancements
- 14. There are big differences on ownership and licensing IP Owned by Oracle Licensed Commercially & Under the GPL IP Owned by Community Licensed Under the BSD
- 15. Oracle's focus on proprietary MySQL offerings pushes users to alternatives.
- 16. The path forward for MySQL users is limited ● Benefits ○ Actively Developed ○ Strong User Community ○ Supported by Linux Distros ○ Backed by a Foundation ○ GPL (if OSS matters to you) ● Disadvantages ○ MySQL 8+ Incompatibilities ○ GPL (if distribution matters) ○ Mostly Single Source ● Benefits ○ Actively Developed ○ Mostly Open Source ● Disadvantages ○ Single Source ○ Small User Community ● Benefits ○ Actively Developed ○ Strong User & Developer Community ○ Supported by Linux Distros ○ Backed by a Foundation ○ BSD License (best for all) ● Disadvantages ○ Migration Difficulties MySQL Clones & Compatibles
- 17. This is Postgres Conference, damn it! Let’s talk about migrating to Postgres.
- 18. ● Numeric ○ INT, SMALLINT, BIGINT ■ Maps directly ■ If AUTONUMBER is used, use SERIAL, SMALLSERIAL, BIGSERIAL ○ DECIMAL, NUMERIC ■ Maps directly ○ TINYINT -> SMALLINT ○ FLOAT -> REAL ○ DOUBLE -> DOUBLE PRECISION Understanding MySQL vs Postgres data types
- 19. Understanding MySQL vs Postgres data types Continued ● String ○ CHAR, VARCHAR ■ Maps directly ○ TEXT, MEDIUMTEXT, LONGTEXT, VARCHAR(max) -> TEXT
- 20. ● Date & Time ○ DATE, TIME ■ Maps directly ○ TIMESTAMP ■ Maps directly ■ MySQL range: '1970−01−01 00:00:01' UTC to '2038−01−19 03:14:07' ■ Postgres: 4713 BC to 294276 AD ■ Beware of 0000−00−0000 ○ DATETIME -> TIMESTAMP ■ MySQL range: '1000−01−01 00:00:00' to '9999−12−31 23:59:59' ○ YEAR -> SMALLINT, INT Understanding MySQL vs Postgres data types Continued
- 21. Understanding MySQL vs Postgres data types Continued ● Binary / BLOB types ○ BLOB (16 k), MEDIUMBLOB (16 MB), LONGBLOB (4 GB) -> BYTEA (1 GB) ● SET ○ Use VARCHAR column with a CHECK constraint that enforces the allowed set of values ● ENUM ○ Maps directly, or use VARCHAR with a constraint
- 22. Choosing a migration approach depends on size & availability ● Offline: have maintenance window, applications inaccessible ● Online: minimize downtime ○ Capture changes to source database ○ Backup database ○ Transform DDL and data ○ Create target database ○ Load data ○ Apply captured changes to source database ■ May need to repeat ○ Pause writes/access source database ○ Cut over Application user pattern: ● Have one read-write user connection pool and one read-only user connection pool. ● Load balance reads. ● REVOKE write permission for read-write user in source database
- 23. How to Migrate - Offline: Manual ● mysqldump ○ –compatible=ansi will get Postgres-friendlier output ● Edit dumped DDL definitions and data. ○ 3rd party tools ■ FromMySqlToPostgreSql ■ mysql-postgresql-converter ○ May encounter backtick ` issues. In PostgreSQL `` escapes them ● Load into Postgres
- 24. Performing offline migration with pgloader ● pgloader ○ Many options, such as doing subset ○ Can use MySQL as a source database in a single, easy step! (thank you, Dimitri Fountaine) pgloader mysql://root:rootpw@localhost/wordpress pgsql://postgres:postgres@localhost:5432/wordpress Produces error report of failures
- 25. Performing online migration with change data capture (CDC) ● Triggers ● Debezium MySQL Kafka Postgres Connector Postgres
- 26. ● Validate Data ○ Validate that the source and destination table data match ● Don’t forget to test performance. Ideally, have production-like workload to test against, including database size ○ Using only a subset of data may not uncover ■ Bad query plans ■ Indexing issues including missing indexes ■ Bloat / issues with vacuum requiring tuning Testing your migration
- 27. Testing your migration ● Tune postgresql.conf ○ Here, too, the test server should be similar to the production to make sure it is tuned properly Continued
- 28. Testing your migration with target architecture ● If using MySQL replication with something like ProxySQL, test a similar configuration with Postgres replication and load balancer like pgCat ● Test HA (e.g., patroni) ● Test backups
- 29. We've already spent too much time discussing data migration without addressing application changes.
- 30. That's why we’re building a solution that tackles MySQL data and apps for you, and gives your apps additional benefits.
- 31. We’ve been managing, developing against, and developing for Postgres for 20+ years each Started with PostgreSQL 6.5 Community Contributions & EDB Compatibility/Perf Features Led the Database Team @ The Meet Group MariaDB PostgreSQL With Xpand (Xgres) Started with PostgreSQL 8.0 ExtenDB/GridSQL & StormDB/Postgres-XL Distributed Consistent Database @ FutureWei Immuta FDW
- 32. Presented a demo at The Ingres Conference of PL/SQL support added to Ingres r3 with a front-end compatible with SQL*Net. We’ve also been doing database compatibility for 20+ years The first version of MySQL compatibility was written as a man-in-the-middle proxy, in front of an Oracle Database or Postgres backend. The first version of NEXTGRES was created, compatible with Oracle Database. 1999 Joined EDB, licensing some NEXTGRES components, adding Oracle Database compatibility features, and working on joint project to add PL/SQL to IBM DB2. 2004 2005 2009 Major clean-ups and rewriting background worker listeners to use AWS Babelfish hooks. The proxy-based MySQL was replaced with a version integrated within Postgres. 2019 2024
- 33. We want to bring three major MySQL benefits to the Postgres world Many users abandon software that takes too long to set up and become operational. Managing databases himself and caring about the user experience, Monty implemented a “15 Minute Rule” to ensure installation, configuration, and key features were usable within 15 minutes. Monty’s 15 Minute Rule High Performance for Read Workloads Development & Admin Tools Postgres struggles with performance under heavy read loads and simple transactions. MySQL is renowned for its fast read operations, making it more suitable for web applications that require quick data retrieval. Postgres lacks comprehensive tools for development and administration, limiting productivity and ease of use. The MySQL ecosystem offers a wide array of development and administrative tools, enhancing productivity and simplifying database management. This necessitates items such as better built-in replication options.
- 34. MySQL primarily has a user, not developer, community, resulting in few external advancements to its core. Postgres has extensions for practically everything, created by users and developers working with it and solving real-world problems that are not addressed by the official project. MySQL lacks advanced indexing, partitioning, and analytical functions, materialized views and parallel query. Postgres supports all of these things out of the box, with additional improvements from extensions in the ecosystem, meaning more can be done than was possible inside OSS MySQL. MySQL was limited to SQL/PSM, which didn’t gain much adoption, and there’s only a simple preview of JavaScript. Postgres allows substantial work to be done inside the database using almost any language you prefer, keeping the data and business logic close together. This could make MySQL apps even faster. We equally want to bring three major Postgres benefits to the MySQL world Support for Multiple Procedural Langs Large Database Support Access to Extensions
- 35. We want to do this with zero application changes A P P L I C A T I O N C H A N G E S
- 36. We’re going to do this with a Postgres extension MySQL Listener MySQL Protocol Connector/C[++] Connector/J MySQL Connector/NET SQL/PSM Compiler/Interpreter We’re replacing our background worker listener with AWS’ Babelfish Hooks
- 37. Developing On GitHub Just Created Public Repo Follow Along or Contribute Star & Watch We’re making heavy use of Docker for development, so everyone can get the exact same environment. This also allows us, and others, to connect the most commonly used open source MySQL apps to it, as they too have mature Docker containers. Because we’re swapping out our old background worker-based listener approach to use the AWS Babelfish hooks, we require the same patched version of Postgres they have. Having proven their value to multiple Postgres users and developers, our goal is to work with AWS and the community to get their hooks accepted into baseline Postgres. What does development of the extension look like? Docker-Based Development We Require A Patched Postgres https://github.com/nextgres/nextgres
- 38. The main benefits Multi-Source Open Source Growing Market Wire Protocol SQL Syntax Procedural Languages Columnar Support Vector Search & AI/ML Extensions MySQL Compatibility Advanced Capabilities Database Consolidation One System to Manage Improved Productivity Simplified Operations Active Community
- 39. Our Timeline April 2024 September 2024 GA Alpha Beta July 2024 Beta EOY 2024 GA
- 40. We need you (or your team’s) help! ● We want to help companies move from MySQL to Postgres quickly, easily, and cost-effectively. ● We have an aggressive timeline, testing with major MySQL open source packages, but we need more real-world use-cases. ● ~75% of companies running Postgres also use MySQL, which means you (more likely than not) have some systems that could move. ● If you’d like to work with us on the Alpha/Beta, reach out to info@nextgres.com Pretty sweet generation, except those AI hands ¯_(ツ)_/¯
- 42. DEMO WordPress mysqldump from MySQL to NEXTGRES ProxySQL Server Switchover