Migrando seu banco de dados para a AWS - Deep Dive em Amazon RDS e AWS Database Migration Service

Notas do Editor

• #4: So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine types.
• #12: The first thing you want to do is to tightly control access to your data resources. You should start with database-level access restrictions. Create a user that can only read, write, but can’t delete tables. Don’t allow users to Drop Table or Truncate. Next, secure your network. Security Groups will allow you to define which hosts can access your databases. You should define separate Security Groups for your Database hosts and Application hosts. Use the Application Security Group names in the egress/ingress rules for your Database instances instead of CIDRs. The next layer of protection are the VPCs (Virtual Private Cloud). VPCs separate Amazon into logically separate parts. You can further subdivide VPCs into subnets. You should define separate subnets for your databases, application and web servers. The first one will be a private subnet not accessible to the internet. You should deploy your database and application server there. The second one, where you deploy web servers, will be public and accessible to internet hosts. You can use the Network Access Lists (ACLs for short) to define egress/ingress rules for subnets. Lastly, use the Identity and Access Management (IAM for short) to manage permissions for your users.
• #13: One of the reasons that customers are concerned with security is that they have certain compliance requirements that they have to mee. AWS has lots of different customers in the startup, enterprise, and public sector space running a wide range of applications and workloads across many different industries. Some of these applications and workloads require that they meet certain regulatory requirements. To support the audit and compliance requirements of these customers AWS has worked to achieve certification that you can run your workloads on RDS and be able to achieve a full compliant application. So for RDS we currently offer 9 different compliance attestations including Soc1, 2, & 3, FedRamp, PCI/DSS, and HIPAA. With these compliance certifications for RDS it means that you can build and run your applications and workloads, related to these compliance bodies, on RDS. However, it does not relieve you of the responsibility of making sure that your application meets the appropriate compliance requirements. AWS takes responsibility for the compliance of the RDS service and the infrastructure related to it and you, the customer, take care of the application that you have built on top of AWS. With this model you can take the audit findings from your application and combine them with the appropriate attestation from AWS’s third party auditors and have a complete verification of compliance for your application running on AWS. ---------------- Chart of what is certified and what is not is here: https://w.amazon.com/index.php/AWS_Security_Assurance/Compliance/Scope
• #14: Here is a listing of our compliance certifications and which database engines they currently apply to on RDS. You can see here that we have a wide range of certifications covering MySQL, Oracle, SQL Server, and PostgreSQL. As you can see there is a lot of coverage among these four different engines around key areas of compliance. You may be asking where is Aurora in this. What I can say to that is “Compliance is mostly a certification process that takes time with the agencies that do the certification so we are working with them on Aurora but you have to follow a process for certification which takes time”.
• #15: ** Make this one stronger** With some of the compliance bodies that I just mentioned in the previous slides there is a requirement that all traffic between compute nodes and external users be encrypted. HIPAA is a great example of this. At times, even just for internal reasons some companies will make this a requirement as well. With RDS we have the ability to encrypt the traffic to and from your database using SSL. Each of the six engines supports the ability to configure an SSL connection into your database. The way that you implement the appropriate SSL certificates and configurations on each database engine may be a little different from each other and this is primarily due to how each of these engines has built their support for SSL. The main thing to keep in mind here is that you are able to protect your data in traffic via a standard SSL connection to your database. Which SSL versions supported? --------- http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html MySQL, Aurora, SQL Server, MariaDB, PostgreSQL – RDS creates an SSL Cert in the DB instance when the instance is first provisioned. You then configure the client for your engine to reference the DB Instance Endpoint common name and do the setup that is specific to the client you are using. MySQL, Aurora, MariaDB are very similar due to the fact that they are all MySQL or MySQL Based. Oracle – You define SSL as an option when you launch and RDS uses a second port for SSL connections. Then you configure your Oracle client to use SSL.
• #16: After you secured the access to your database, you have additional mechanisms to further securing your databases. You can encrypt data in transit using NNE – native network encryption. You can audit access to your database using the native Orcale audit logs that are accessible to you through the API and the AWS Management Console. And you can also encrypt your data at rest. I want to dive deep into this topic. Volume Encryption: - Works with all editions of Oracle - Use your own AWS KMS keys for more control Use the default key managed by RDS for ease of use Using TDE with Enterprise Edition TDE_HSM and TDE database options TDE_HSM integrates with AWS CloudHSM Share CloudHSM partition between databases and accounts
• #17: Now let’s move into metrics and monitoring on AWS and what you can get out of that. Being able to monitor your database and know how it is performing is a very critical part of managing a database that is supporting your application or workloads.
• #18: For everyone using RDS you get a collection of 15-18 metrics that are automatically made available to you, depending on the type of instance that you are using. RDS sends the necessary information for these metrics to Amazon CloudWatch and then you are able to view the metrics in the RDS console, in the CloudWatch console, or via the CloudWatch APIs. These metrics are made available to you at one minute intervals. With this monitoring you can keep an eye on the performance of your database around key items like CPU utilization, memory, storage, latency, and any lag between your master and read replica databases. You can view this information in individual graphs, multiple graphs, or even pull them into your own monitoring tool. Additionally, you can also take these metrics and build alarms based on thresholds that are meaningful to you. Then whenever you go above that threshold you can have a notification or other action take place that helps to respond to that metric being outside of its normal boundaries. We will actually take a look at an example of what you can do there a bit later in this presentation. --------------- Basic cloudwatch metrics - http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/rds-metricscollected.html
• #19: When it comes to monitoring a database there are a lot of different measurements and metrics that you can examine. With no access to the host operating system of the RDS instance getting at all the monitoring information you need could be a challenge because there are certain operating system metrics that you want to see that would better help you determine the health of your RDS instance. To help with this problem the RDS service gives you the option to turn on Enhanced Monitoring. When you enable enhanced monitoring you get access to around 37 metrics that you can combine with the metrics that you get from basic monitoring to better help you understand what is going on with your database instance. When you enable enhanced monitoring you can define the granularity at which you can see the metrics from sixty seconds all the way to one second. A few of the metrics that you get with enhanced monitoring are: Free Memory, Active Memory, Load Average, and how much of the file system you have used. You might be asking the differences between basic monitoring and enhanced monitoring and why they are not all delivered the same. The reason is that standard monitoring is based on what the Hypervisor that is running your database instance can see. There are metrics related to your OS that the hypervisor cannot see and we utilize an agent running on your RDS instance to collect the necessary metrics to give you enhanced monitoring. Enhanced monitoring is available for: all six engines ------- Discussed in a section on this page: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring.html Available on all classes except t1.micro and m1.small
• #20: ** Crisper** To give you even more visibility into what is going on with your RDS instance you can also use Event Notifications. These Event Notifications allow you to get notifications, via Amazon SNS, when certain events within your RDS instance occur. You can subscribe to events around the DB Instance, DB Snapshots, DB Security Groups, or DB parameter groups. A few examples of what types of event notifications you can receive: Configuration change – you can get notified any time a configuration change such as a security group modification occurs, or when the storage settings for the instance have changed. Low Storage – You could be notified when the database is getting low on storage so that you can take action before you run out of storage. Failover – you could be notified when your master database is failing over to its standby configuration. Overall Event notifications are something to give you extra insight into what is going on with your RDS instance and giving you the opportunity to take action when something does happen. These event notifications not only give you visibility into what might naturally be going on with your RDS instance but also can give you visibility into changes happening that might impact the security of your database such as making a change to the security group for the database. ------------ http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.html Future deep dive item – We should show action happening when an Event Notification happens. Maybe adding storage when we get a low storage notification. This way we are not paying for storage before we need it. Maybe notifying when a security group change occurs.
• #21: Being able to scale your database to keep up with demand is an important part of operating a database for any application or workload. Let’s dive into the ways that the RDS service allows you to scale your database.
• #22: When your master database becomes overloaded in trying to handle read and write requests you can either try and scale the master itself or you can take the path of adding read replicas to handle the read workload. With RDS read replicas do a couple of things. First they allow you to create read only copies of your master database that are kept in synch with your master database. You can then point your read queries from your applications or end users at these read replicas helping to keep the read workload on your master database low. Depending on the database engine you can also place your read replica in a different region from your master which may give you the ability to have a read closer to a particular locality. Finally, read replicas give you the ability to have another avenue for failover when there is a problem with the master, giving you the ability to have coverage in the event of a disaster. ----------- Benefits Offload reads that would normally happen against your master database, allowing the master database to be able to focus on serving up writes and maintaining the integrity of the transactions in your database. Support read intensive applications for specific business units Promote read replica to your production database in the case of failure ** *More on this one – Have to figure out how to tie this one in with High Availability
• #23: For read replicas there are two deployment options. Intra region which allows you to create additional read replicas within the same AWS region, but in the same or different availability zones from your master database. This functionality is supported by MySQL, MariaDB, PostgreSQL, and Aurora Cross Region which allows you to deploy the read replica into an AWS region that is different from the region that your master is located in. This functionality is supported by MySQL, MariaDB, PostgreSQL, and Aurora The approach you take for these options is going to depend on the database engine that you are using as well as the ultimate approach that you are trying to take in regards to why you want the read replicas and what your disaster recovery plans and high availability plans are.
• #24: This is just the same as with our High Availability discussion for Aurora a few slides back. So with Aurora I want to call out that there are a few differences with how read replicas are handled vs the read replicas that are offered for the other engines. With the other engines there is going to be some lag between the master and the read replica in regards to getting updated data synched from the master to the read replica. When it comes to Aurora there is much lower lag between the master and the read replica because in the background they are sharing the same storage layer so there is no need to replicate data between the storage layers of the master and read replica. With the other engines you can create around five read replicas before you start to put a lot of pressure on the master database in regards to supporting replication to all the read replicas. With Aurora you can actually have up to fifteen read replicas all tied into the master and the shared storage related to your aurora cluster. For Aurora these read replicas are actually your failover option as well for when there are problems with your master database. With this approach you have the ability to failover to your read replica with nearly no data loss from the failure of the master due to the shared storage that both instances would use. Finally, if you have a need to have multiple read replicas and want to balance traffic across those nodes we have partnered with the team at MariaDB and there is a driver that you can actually use in your applications that will allow you to balance traffic across your read replicas in order to ensure that you are not overloading any one read replica.
• #25: Up to this point I have been talking about read replicas for four of the engines that are supported on RDS. Currently RDS does not currently support read replicas for Oracle and SQL Server. However you can still accomplish this on RDS. To start out with there are several AWS partner products such as Attunity and SnapLogic that you can use to replicate data between two RDS instances of Oracle or SQL Server. You can also use the Oracle GoldenGate product to replicate data between your databases. I have this in the Oracle section you can actually use the GoldenGate product to replicate data from a source database that is SQL Server as well. Finally you can use snapshots to take a copy of your database and create a new database with a copy of that point in time copy of the database. We will talk more about snapshots in a bit.
• #26: Here is a quick example of what it would take to scale your RDS database instance up or down. Here we have screen shots of what you do when you want to change the instance size. Choose the Modify option from the Instance Actions menu of the RDS console. Then choose what you want your new DB Instance Class to be. Finally, determine if you want to apply the change immediately or not. If you do not apply the change immediately then the change will be scheduled to occur during the preferred maintenance window that you defined when creating the database. Keep in mind that when you are applying the change immediately you could incur some downtime as the instance size is changed so be aware of what your applications that are accessing the database can tolerate in regards to downtime. http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.DBInstance.Modifying.html
• #27: Here is an overview of what happens if you are running a scaling operation on a database that is running in a single availability zone configuration. Final piece of animation is what the RDS console shows in regards to the events that just occurred.
• #28: Here is an example of how scaling works when you are running your RDS instance in a multi availability zone configuration. Final screen shot. You can see the RDS recording of the sequence of events for the multi AZ failover. You can also see what I do at 6:20 AM on a Saturday.
• #29: So I showed you how easy it is to scale your RDS instance via the management console. It is also just as easy to scale your RDS instance via the AWS APIs. The top example is an example of re-sizing an RDS instance with the command line. You can see that I am setting the instance class to an m4.large and I am setting the apply immediately option so that the scaling option happens immediately. Recall the example I talked about a few slides ago where I talked about a scenario where you might want to scale your database down on the weekend when there is lower usage. What I have on the bottom is an example of a cron job that could be scheduled to perform the scaling actions in the above CLI command but on a schedule. So at 8:00 PM on Friday I call the scale down script to set the RDS instance to a lower size and then at 4:00 AM on Monday I call the scale up script to set the instance size back to what I need it to be during the week. Now with the cron job you have to have some sort of compute up and running to support cron and you need to grant that compute access to be able to modify your RDS instance. So you are either running some very cheap compute that is just for this or you have an instance that you are comfortable with giving this level of access.
• #30: Still on the subject of scaling… Another option to accomplish the same scheduled scale activities is using a scheduled lambda function. For those who are not familiar with Lambda, it is an AWS service that allows you to run code on demand based on events. In this case we are creating a scheduled event. On this screen is an example of some python code that I created as a scheduled lambda function to call the modfiy db instance API to scale my database instance. For the weekend scaling example you could have two scheduled lambda functions to handle the Friday scale down and the Monday scale up. One important thing to call out here is that when you are doing these scaling actions you need to take into consideration how your application is going to react to the changes in the database. It may be necessary to have some additional code that pauses the application while the scale action takes place. ----------- Lambda creation using python: http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
• #31: So with RDS there is not a service, like AutoScaling on EC2, that allows you to automatically re-size your database based on metrics. However, there is still a way to implement something like AutoScaling with RDS using a lot of the same services. So you can utilize an approach where you have RDS sending metric information to CloudWatch. You then have alarms happening in CloudWatch based on alerts that you created for thresholds that matter to you. Those alarms are then sending notifications to the Simple Notification Service and then you have a lambda function subscribed that notification to actually take action. So with this approach let’s say that you care about CPU utilization of your RDS instance. You could have an alarm based on CPU that fires when utilization is above 80%. That alarm would then call a notification that would then call a lambda function to scale up your RDS instance.
• #32: Here is an example of a lambda function that gets fired based on an SNS notification. It reads the message sent by the notification and pulls out he name of the database instance and then calls the modify API to scale up the database. This is a super simple example. A more thorough piece of code might pull out the current instance type and determine what the next appropriate step up level would be. The point of all of this is that you have some options when it comes to scaling your database in an automated fashion.
• #33: Amazon Aurora is a MySQL-compatible, ACID compliant enterprise-grade relational database engine built for the cloud. In many ways,  Aurora is a game changer and helps overcome the limitations of traditional relational database engines. Most traditional databases have been built and optimized for on-premises environments, and the fundamentals of their architecture have not changed much in the last few decades. One of the key objectives of  Aurora is to overcome the performance, scalability, and availability limitations of traditional databases in a cost-effective manner similar to open-source databases. Durable, fault tolerant, self healing, storage auto-scaling, automatic backups, performance, read replicas, instant crash recovery, survivable buffer cache, security.
• #34: Let’s look at data that does not fit in cache Here we are scaling a db size up from 1 GB , which can certainly fit in most any cache, to a TB Aurora is basically running about the same performance as long as we are fitting in cache. Than it drops down as you would expect, once you no longer fit in the cache by about 4x Whats interesting with mysql is it starts declining well ahead of that point while the data still is fully cached. Int his case.we are running 67x faster, which is a crazy number, which made us decide to run something else, the Cloudharmony TPCC like benchmark which is pretty much like the std TPC-C benchmark the tpc council runs, and here it is running 136 x faster when it is not fitting in cache. And these are transaction on the right, and statements on the left because those are the two standards for these benchmarks
• #35: If you want to run HA or high throughput read queries you will probably use read replicas Mysql RRs start to degrade, the lag between the master and slaves goes up, as you start to increase the updates per second through the system In Aurora we also use the RRs as failover targets as well. Going from 1000 to 10,000 updates per second we are relatively stable at about 5 ms of replica lag Whereas In MySQL we go from a few ms to 300 seconds
• #36: For the bulk of this talk we want to talk about is HOW. The core of this comes down to improving your algorithms, sharding your locks, removing contention. And that is good, but anyone can do this – it is not architectural work Once you are done with that you are at the point where you are hitting IO. Well designed dbs are all about IO. Once you do network attached storage it is all about packets per second. You can do somewhere between 1 and 2 mil packets per sec on high end hw like EC2 enhanced networking, but that is still a pretty constrained number The other thing that is important if you want to operate at high throughput you cannot afford system calls or context switches. So a lot of our work that has been architectural has been around reducing Ios, minimizing network packets, and moving toward processing asynchrounously.
• #37: Here we have a single Aurora instance shown with its storage layer. This is called an Aurora cluster. The cluster consists of one or more Aurora instances and the underlying shared storage layer which spans 3 azs. Here we are showing Aurora with one primary instance only, but you can have multiple replicas – up to 15 – in these 3 Azs.. Let’s take a look at the storage layer. Aurora maintains 6 copies of your data across three Availability Zones – by default – no matter if you have chosen to have additional Aurora instances – the storage is on a virtual disk that spans 3 availability zones. Because Aurora maintains multiple copies of your data in three Availability Zones, the chance of losing data as a result of a disk failure is minimized. Aurora automatically detects failures in the disk volumes that make up the cluster volume. When a segment of a disk volume fails, Aurora immediately repairs the segment. When Aurora repairs the disk segment, it uses the data in the other volumes that make up the cluster volume to make sure that the data in the repaired segment is current. As a result, Aurora avoids data loss and reduces the need to perform a point-in-time restore to recover from a disk failure. And by the way, at the storage layer, you are only billed for one of those copies, not six.( You are charged based on the storage your database consumes at the database layer, not the storage consumed in this virtualized storage layer.)
• #38: For comparison, this is what your cluster would look like with additional Aurora replicas provisioned, here one in each AZ
• #39: Lets compare and contrast : So here is io traffic in RDS MySQL with a standby instance. You issue a write, it goes to EBS, EBS issues to a mirror asynchronously in parallel. Once both writes have happened we synch are going to go to another AZ to do another write with DRBD. Then you go to the standby, it has to issue a write, and then it proprages and comes back. The interesting thing to point out here is steps 1,3 and 5 are sequential and synchronous. So if there is any network “weather” going on it is oing to amplify both the latency and the jitter – minimally ou are going to be looking at 3x and possibly much more than that. The other thing you see is there are a lot of diferent types of write operations: log, binlob, data, double write to avoid the torn write, and the frm files for metadata And the thing that is particularly difficult int his is the data blocs are a good deal larger than the log, about 40x, and you are writing them twice so that is 80x. So that is a big deal. But like any solid db mysql is going to be doing a lot of batching under the covers to try to mitigate that overhead. So what we did here is we ran a 30 minute write only SysBench workload, with 100GB data set, SingleAZ, 30K PIOPS –and we ran 780 k transactions. We saw over 7.4 million and that’s pretty oggd –that is the work the primary instance is seeing. that is steps 1. Steps 2,3,4,5 are not reflected. You want to focus on the network throughtput on that single instance. Ets compare it to aurora. Aurora – 27,378,519 transactions, 35x more. 158,323 operations – 46x fewer. Latency aggregates Jitter amplified Write both log and data. Data IO flow Observations Multiple sequential steps Latency and jitter amplify Double-writes because overwriting page Performance
• #40: So let’s compare it to Aurora. So in Aurora what we are doing is we are only shipping log records. So if we see a bunch of log records come to us, we boxcar them together SysBench writeonly workload 100GB SingleAZ 30K PIOPS – 30 minutes 779,664 txns. 7,387,798 operations Aurora – 27,378,519 transactions, 35x more. 158,323 operations – 46x fewer. 949938
• #41: So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine types.
• #51: Mention why no older engines as sources +Sybase v15+
• #56: 30 segundos para apresentar a empresa, rapidamente
• #57: Os 4 (máximo) maiores desafios do projeto, que foram resolvidos pela utilização da nuvem da AWS
• #58: Diagrama de solução, e explicar a solução, vantagens, etc
• #59: Os 4 (máximo) maiores desafios do projeto, que foram resolvidos pela utilização da nuvem da AWS
• #60: Quick overview of what RDS is. This is a Deep Dive so there are some assumptions that some of the basics with RDS and the benefits are already understood. We are going to touch on many of these in more depth throughout the presentation. RDS is a managed database service. This service allows you more time to focus on your application: You focus on Schema Design, query construction, query optimization, and building your application. Infra Mgmt AWS does patching AWS Handles backup and replication AWS manages the Infrastructure and making sure that it is healthy You focus on your application HA and automated failover management High end features that you can do on your own but you get automatically. Instant Provisioning Simple and Fast to deploy When you need to launch a new database or change your existing one you can at any point in time with no need to wait for infrastructure to be ordered or configured. Scale up/Down Simple and fast to scale. You can change your configuration to meet your needs when you want to. Cost-effective No Cost to get started Pay only for what you consume Application Compatibility * Six different engines to choose from * There are many popular applications, or even your own custom code code, that you may be running on your own infrastructure and it and it can still work on RDS. If you are using one of the engines that are currently supported then there is a good chance you can get it working on RDS. When you think about all that it takes to get new database infrastructure and an actual database up and running there are a lot of things that an expert DBA and infrastructure person would have to do. With RDS you are getting this with just a few clicks and are up and running in a manner of minutes.