Cloud Storage Engine for MySQL -- ClouSE

Editor's Notes

• #5: Cloudstorage is great – it’s the most useful and the most used cloud computing utility. It’s fast, reliable, inexpensive and the most mature. There are multiple vendors who offer cloud storage.Unfortunately it provides simplistic vendor-specific API . You can put, get and delete objects by key and that’s pretty much what you can do. You don’t get any kind of query capability, atomic updates or any kind of transactionality.The cloud storage has been widely used by two classes of applications: content distribution and backup. That’s because it’s hard to write other kinds of applications on top of this API.
• #6: Some cloud storage services such as the most popular and mature Amazon S3 provide eventual consistency guarantee.With eventual consistency, the simple question of “if I put something to the cloud storage what do I get back” has a complicated answer: eventually you’ll get what you put there (provided no other changes are made to the object). The lag time is generally milliseconds, but has been known to take several minutes in extreme cases. Just enough for an application to fail unless it’s carefully designed to deal with eventual consistency.Eventual consistency is a conscious design choice and it’s really great for building one-of-a-kind massively distributed applications because it boosts application availability. But if you are NOT building one-of-a-kind massively distributed application, eventual consistency creates more problems than it solves.It took me a while to internalize how I can work with the eventual consistency, it’s really pushing the traditional thinking about how storage works.
• #7: ClouSE solves the cloud storageproblems. Together with MySQL it provides mature and standard SQL access to cloud storage. It supports transactions and is fully ACID-compliant.ClouSE makes cloud storage immediately available to millions of SQL developers and MySQL-based applications.
• #9: The idea of storage is simple: ideally you have enough storage to hold your data, not more not less. This idealistic usage pattern is what you get out of box: you install MySQL, create some tables put some data into them, run some queries, and it works.Until it fails.The problem with storage failure is that you lose your data. Data is very expensive to replace, or it may be irreplaceable. Data loss is very likely to drive a company out of business. In that respect, storage is different from any other hardware component: if the CPU or motherboard fails, it’s easy to buy a new one.
• #10: To deal with storage availability and recover from failure, storage infrastructure is really complicated and costly.First you get expensive high-reliability primary storage. It’s usually overprovisioned to account for data growth.Then you protect your data on-site by backup and / or replication.And finally you hire a vendor to store a copy of your data off-site in case your on-site infrastructure fails.Operating this infrastructure is costly too. The overall solution is typically a set of in-house custom scripts that glue various technologies together: the SAN vendor may provide a backup solution, but it doesn’t work out-of-box with your database or offsite backup vendor. Scripts have bugs, the infrastructure changes every couple years, and so on.But the worst part is that the disaster recovery is often a disaster! In my experience, the actual recovery scenarios are poorly tested, and recovering from a backup takes longer and leads to bigger data loss than expected for various reasons.
• #11: Replication is often used to provide both data protection and high-availability solution.However, failover is fraught with perils. Replication leads to multiple copies of the same data and not only does it use more storage, it also has the risk of data being out of sync.That’s why automatic failover back to master is a big NO-NO, and DBA professionals recommend manual switchover after a careful inspection of the master data.
• #12: ClouSE doesn’t have the problem of out-of-sync data. There is only one copy of data – in the cloud, it does not use extra storage and makes failover safe.Failover is safe to do in any direction and to any machine. You don’t need to inspect the data on the machine – the local data does not matter.
• #13: With ClouSE, MySQL can use cloud storage as primary storage. Cloud storage consumption scales up and down with data size, you pay for what you use.Cloud storage is highly reliable and doesn’t need backup or any other complex procedures to provide data protection.The disaster recovery is built-in, it doesn’t require constant testing.Failover is robust and painless, it doesn’t requires constant testing.
• #15: This is the MySQL’s pluggable storage engine architecture. I took this picture from the MySQL documentation.MySQL server provides SQL interfaces, handles query processing and optimization, provides common API for applications, handles authentication and authorization and so on.The storage engines provide memory, index and storage management. Transactional storage engines also provide transaction management.The most commonly used storage engines are MyISAM and InnoDB.
• #16: On the inside ClouSE is very similar to other transactional storage engines. The primary difference is that ClouSE is designed and optimized to use cloud storage.ClouSE uses strict consistency enforcement algorithms and protocols to implement ACID guarantees on top of eventually consistent storage.ClouSE uses compression and caching techniques to reduce storage and bandwidth consumption. In our experience, the relational data compresses pretty well: 10-20 times.ClouSE uses strong encryption before it sends data to the cloud, which guarantees full data confidentiality in the cloud.ClouSE is the only cloud-based database technology that can provide full data confidentiality in the cloud.
• #17: ClouSE installation is super easy. You need to add 3 configuration options to the config file and execute the INSTALL PLUGIN statement.The first 2 options specify where the data is stored in the cloud and access credentials. In this example data stored in Amazon S3.The 3rd option specifies the encryption key. The encryption key is managed by the DBA and is used to guarantee that the cloud storage provider doesn’t have access to the data.
• #18: The ALTER TABLE statement can be used to a table to the cloud storage.I bet this is the easiest way to use cloud storage from an application: you don’t need to learn new API, you don’t need to re-design the application, you don’t need to even change the application code: the DBA can make this decision during application deployment and configuration.ClouSE provides risk-free gradual cloud adoption, you don’t need any upfront investment to start using the cloud. The way out is as easy as the way in, you can easily test how the application is going to work with the cloud and fix problems as you go.
• #19: Moving one table to the cloud is great for migration, but the real value of ClouSE technology comes when all tables are moved to the cloud.The cloud storage becomes the primary storage for the data, and the server merely exposes the data in an application-friendly form.The database server doesn’t have any valuable data and can be easily replaced if it fails.It is also easy to move the server to a different machine or to the cloud.
• #21: So far I talked about using cloud storage for storing relational data. But ClouSE goes further and takes advantage of its cloud-based architecture to tap into the power of cloud storage to make serving content highly scalable.ClouSE can return cloud storage URLs so that the web browser can use the URL to get the content directly from cloud storage.We call this feature “weblobs”. Here is an example of creating a table with a weblob picture. A weblob is represented by two fields. The first field is just a regular blob and can be atomically updated in a transaction. The second field returns the direct cloud storage URL that the application passes to the web browser.The application scalability may grow significantly without adding complexity for the application developer.Imagine a picture sharing website. Most of the traffic comes from transferring pictures. With ClouSE, the pictures can be stored in weblobs and served from the cloud storage, taking most load off the server.