You most probably dont need rman catalog database white paper
5 likes7,983 views
This document discusses the costs and benefits of using an RMAN catalog database. It notes that while a catalog database adds overhead in implementation and maintenance costs, it also provides some important benefits. Specifically, it argues that a catalog database is generally not needed but may be justified in cases involving disaster recovery configurations or long backup retention policies. Additionally, it outlines some optional benefits like centralized monitoring and reporting of backup operations across multiple databases. Overall, the document aims to provide context for DBAs to evaluate whether a catalog database is truly necessary or beneficial for their specific backup environment and use cases.
![to synchronise backup information for both primary and disaster recovery sites. See the “Catalog & DR”
section bellow for further discussion. I think there is no work around here.
KEEP: There are requirements to use a RMAN KEEP command to keep a backup for longer than current
retention policy and a controlfile (control_file_record_keep_time) allows. This way information about backups that
we would like to keep will be stored in a catalog database until it expires. However long term backups need a
special care and should be treated as archiving rather than as regular backups. We may also want to backup
oracle database software that is associated with the backup and possibly operational system. In case of file
system backups many of us feel more comfortable if one off backup’s files are moved (copied to other
directory or volumes) rather than keeping those on the original location. As alternative we could create a
backup on an alternative location and UNCATALOG those to exclude from the purge process. At the time of
recovery you either use a control file created along with the backup or use CATALOG or CATALOG START
WITH RMAN commands to register it in the current control file 7. In both cases KEEP command is not
necessary or in other words there are alternative solutions and the KEEP command is a nice to have. In case of
tapes (MML) backups, most often a retention policy is managed by MML layer (with no good visibility for
DBA teams) and therefore RMAN KEEP command would not be very useful8. The most important point here
would be to ensure that the backup is sent to the right MML retention pool. RMAN backup’s log file contains
important recovery information (e.g. handle and pool names). Those are to be kept a safe place until possible
recovery. However if you have a need for the KEEP command then you should use a catalog database. If you
decide to use RMAN catalog for this reason please confirm that the MML solution used in your environments
supports RMAN retention policy and it has a priority over MML side retention policies.
CONTROL FILE SIZE: If the size of database control files (control_file_record_keep_time) does not allow storing
as long a history as necessary for current retention policy, then you need a catalog database to ensure obsolete
backups get deleted and you can restore from backups older than what is stored in the control file. In this
scenario ability to purge obsolete backups is more important than a restore operation. If for example we use
MML and it controls the backup data purging process, then it is not critical that a current controlfile “knows”
about all backups available. We can always restore a controlfile we need using the right MML “handle”. The
handle could be obtained from either a backup’s log file or recovering an older controlfile copy that the current
controlfile is aware of9. However, based on my experience, if an organisation uses a file system (e.g. NFS) to
store backups then it is rare when a retention policy is greater than a reasonable sized controlfile could handle
(e.g. 30 days). If an organisation uses a tape solution, then most often the retention policy is managed by MML
layer and therefore catalog does not play any role in a backup’s purging process. The controlfile size limitation
could be a legitimate reason to introduce a catalog database, however as you can see there are still some
methods to address some of the challenges without requiring a catalog database.
DUPLICATE does not need a catalog database. Some of you may say that RMAN DUPLICATE from
backups without target connection 10 requires a catalog database. As a matter of fact there is the following
statement in the oracle 11GR2 documentation - “RMAN can perform backup-based duplication with or without either of
the following connections: Target, Recovery catalog”. DUPLICATE does not require catalog database. You may want to
review the “Creation Of Rman Duplicate Without Target And Recovery Catalog Connection. [ID 1113713.1]” note for
additional details.
7 There is a risk however that the current controlfile may be on the different version. This is why the suggestion is to
include operation system and oracle home backups along with the database backup that you are planning to keep longer
than normal retention policy.
8 With exception of recovery process. In this case we will need to restore “old” controlfile from tapes and then process
with regular recovery. However recovery from “old” backup may treated as exceptional case and therefore may not be
significant enough reason for a catalog database introduction. As an additional argument I would like to mention that
restore from an “old” backup often involves more complex activities (e.g. OS and Oracle Home restore) to be treated as
exception.
9 This method could be time consuming as we may need to recover several controlfiles before getting to the
right one. In this scenario storing backups’ log files may be more reasonable solution.
10 This is a new 11G R2 feature](https://image.slidesharecdn.com/youmostprobablydontneedrmancatalogdatabasewhitepaper-130330223545-phpapp02/85/You-most-probably-dont-need-rman-catalog-database-white-paper-4-320.jpg)
You most probably dont need rman catalog database white paper
- 1. Category: Backup and Recovery Product Line: Oracle Database 1.45v 2013.04.10 YOU MOST PROBABLY DON 'T NEED RMAN CATALOG DATABASE Yury Velikanov, The Pythian Group ABSTRACT This paper discusses in details the cost and value of RMAN catalog database. TARGET AUDIENCE This paper is targeted for experienced Oracle DBAs who supported and implemented RMAN based backup and recovery procedures. EXECUTIVE SUMMARY Learner will be able to: Justify RMAN catalog database to system’s business owner Leverage additional RMAN catalog benefits Improve RMAN based backup procedures BACKGROUND The title of this paper is purposely thought provoking. My intention is not to discourage you from using catalog database, but rather encourage you to think about the cost and value it adds. In this paper I will discuss legitimate reasons why one should use an RMAN catalog database, if a catalog database is in use, then what additional value it can provide to improve operations. Many DBAs continue to use RMAN catalog database simply because of historical reasons. Oracle’s general advice in earlier database versions was to use Catalog database. In fact most of us know that in first versions of a recovery manager (RMAN) “catalog” was the default option. With newer versions, Oracle changed the default behavior to “nocatalog” and I think there have been some legitimate reasons for it. This paper will review: What costs are involved with running catalog database; What problems does it address and what risks does it help mitigate; What additional value (e.g. reporting, metadata backup and long storage etc) you can leverage from it. In addition it reviews special cases such as RMAN using MML (Media Management Layer or simply a tape system, some DBAs know it as SBT_TAPE) integration; RMAN in Disaster Recovery configuration; RMAN backups based on file system; Retention policy management and some others. Finally you will find practical and general hints on how to use a catalog database at the end of this paper. I must warn you that this paper assumes that you have previous RMAN knowledge. In the ideal case you should be an Oracle DBA for some time and have experience in backup and recovery. This paper doesn’t provide ready to use solutions. The main purpose is to make a complete overview of why you may need or may not need an RMAN catalog database. Solutions to be provided as additional work related to this paper.
- 2. Acknowledgments: I would like to say a huge thanks to all my social media friends, workmates at Pythian and other DBAs who contributed to this paper with their ideas and suggestions. Thanks a lot for the following people for investing their time and efforts to review and improve the paper: Steve Karam one of the first reviewers / editors Brian Pardy one of the first reviewers / editors Fuad Arshad John Piwowar Martin Bach Kamran Agayev Andrey Goryunov Sergey Kosourikhin Christine Kivi a significant proofreading contributor Why RMAN catalog is an overhead? / COST Let me start with not so obvious statement for many DBAs: RMAN catalog database is an overhead that needs justification to exist. Unlike any general business related database it does not store application data. Therefore a business owner of an IT environment may have a valid question: How much does this component cost me and what value it adds? We as IT representatives should have a very good answers, right? Most of DBAs that I have spoken to about it admitted that they use a catalog database because it was been created before them or they followed Oracle’s recommendations without analyzing if it is necessary. Let us start with understanding how big an overhead the catalog database is and how much does it cost? So, what costs are involved running an RMAN catalog database? By saying “cost” I do not mean just server hardware resources1. In many cases hardware costs are not the main contributing factor (hardware is cheap today, right?). Time is a cost and “inexpensive” DBA time needs to be invested to maintain, troubleshoot, patch and update catalog database. Let us go through some typical activities associated with RMAN catalog database implementation and maintenance: IMPLEMENTATION COST: Catalog database and associated RMAN catalog schema need to be created o Partially we can mitigate this cost by using OEM database (if exists then we do not need to create additional database). It is common to see DBAs using OEM database for the catalog schema as it has the similar availability requirements and licensing implications. However this introduces dependency between OEM and RMAN catalog. This decision limits our maintenance windows and upgrade options to some extent. o If we do not have a database where to locate RMAN schema then we need to create a new one. Any new database creation involves planning and communication with friendly neighbour departments (system administrators, network team, backup team etc). This consumes time and resources. o In business critical environments we should think and ensure that catalog database is highly available. This adds to the implementation costs. o In some larger enterprise networks, DBAs sometimes decides to introduce a separate RMAN catalogs based on geographical locations to reduce risk of network issues impacting database backups. This increases the implementation and maintenance costs in proportion to the number of RMAN catalog databases. MAINTENANCE COST: A catalog database needs to be maintained. A DBA should apply patches and keep the database on the supported versions level. You may say it does not need too much attention. I do agree. However there is still some time a DBA spends planning, preparing and executing maintenance activities. As 1 It is worth mentioning that that you do not need an Oracle licence to run RMAN database unless you customise it or use for other purpose that requires a licence. You may check with your Oracle sales representatives though if you need a licence for RMAN database in DR configuration.
- 3. RMAN catalog database is just another Oracle database it requires some DBA’s attention regularly. Depending on the databases’ versions used in your network you may require to keep RMAN catalog on relatively fresh level2. If Oracle databases maintenance processes are well designed and automated in your organisation it helps reduce catalog database maintenance costs. Delaying RMAN catalog database upgrade to the latest version can save on the maintenance costs, as quite often there is no necessity to upgrade RMAN database as often as other business critical databases. DEPENDENCIES: RMAN Catalog database is typically implemented in organisations with several or even hundreds of databases. The more databases that use Catalog database, the less downtime that is available to maintain it. It can become very difficult to close RMAN database for maintenance resulting in additional planning time. DOWNTIME: Quite often, depending on the implementation, RMAN backups may fail if a catalog database is not available3. Any unplanned downtime of catalog database may trigger backup failures and therefore puts possible recovery at risk. AVAILABILITY (licensing): As catalog database becomes a critical resource we should ensure that it is highly available. We do not want to lose critical backup data, do we? The process could involve Data Guard implementation and other measures that adds to the total cost of catalog database. As the downtime should be minimal the implementation may need a bit more efforts. By the way, you may call your Oracle Sales representative to discuss if you need any additional licences in this case 4. TUNING: In the case of managing hundreds or thousands of databases the catalog database may require additional tuning efforts. However with the right approach (statistics, additional indexes etc) it can manage a lot of databases’ metadata. At the end of the day RMAN catalog is nothing but a schema with approximately 40 tables. Most of traditional optimisation methods apply to catalog database. Some may say that the RMAN catalog database maintenance costs are relatively low as it is nothing but an Oracle database. I do agree. In fact with each additional database that we manage within the catalog database, we reduce the overall cost per database. Plus RMAN database unlike many business application databases does not generate too much headache5. However we do have a real alternative, right? Instead of having an additional database and associated costs, we may use an individual databases controlfiles. Question is: : when does the benefits of having a catalog database justify the additional costs associated with it. If we have just one business database can we justify a catalog database or not? How many databases should an organisation have in order to justify the cost? How critical should a business database be to mitigate associated recoverability risks? Now that we have some ideas on what the costs consist of, let us have a look at the benefits. I would like to start with scenarios when there are technical dependences from catalog database and we must have it. Cases when catalog database is a MUST There are just a few technical reasons when we must have a catalog database and where a control file based RMAN catalog is not good enough. There are solutions for some and limited options for others. Let us discuss each of those: Disaster Recovery: When an organisation uses a Data Guard (standby database) for a business database backed up by RMAN and there are backups that takes place on both DR sides 6 then we must have a catalog database 2 For example can’t use a 10.2 catalog database with a 11.2.0.3 production environment 3 See “Use resync catalog” section for a discussion on how to reduce this risk 4 If you need a license for a catalog database standby you may workaround it by synchronizing you RMAN backups with a RMAN catalogs on both DR sides. I mention it later in the paper. 5 We do not change it as often as other databases/applications 6 It would be my preferable option to reduce backup load on primary database and offsite backups
- 4. to synchronise backup information for both primary and disaster recovery sites. See the “Catalog & DR” section bellow for further discussion. I think there is no work around here. KEEP: There are requirements to use a RMAN KEEP command to keep a backup for longer than current retention policy and a controlfile (control_file_record_keep_time) allows. This way information about backups that we would like to keep will be stored in a catalog database until it expires. However long term backups need a special care and should be treated as archiving rather than as regular backups. We may also want to backup oracle database software that is associated with the backup and possibly operational system. In case of file system backups many of us feel more comfortable if one off backup’s files are moved (copied to other directory or volumes) rather than keeping those on the original location. As alternative we could create a backup on an alternative location and UNCATALOG those to exclude from the purge process. At the time of recovery you either use a control file created along with the backup or use CATALOG or CATALOG START WITH RMAN commands to register it in the current control file 7. In both cases KEEP command is not necessary or in other words there are alternative solutions and the KEEP command is a nice to have. In case of tapes (MML) backups, most often a retention policy is managed by MML layer (with no good visibility for DBA teams) and therefore RMAN KEEP command would not be very useful8. The most important point here would be to ensure that the backup is sent to the right MML retention pool. RMAN backup’s log file contains important recovery information (e.g. handle and pool names). Those are to be kept a safe place until possible recovery. However if you have a need for the KEEP command then you should use a catalog database. If you decide to use RMAN catalog for this reason please confirm that the MML solution used in your environments supports RMAN retention policy and it has a priority over MML side retention policies. CONTROL FILE SIZE: If the size of database control files (control_file_record_keep_time) does not allow storing as long a history as necessary for current retention policy, then you need a catalog database to ensure obsolete backups get deleted and you can restore from backups older than what is stored in the control file. In this scenario ability to purge obsolete backups is more important than a restore operation. If for example we use MML and it controls the backup data purging process, then it is not critical that a current controlfile “knows” about all backups available. We can always restore a controlfile we need using the right MML “handle”. The handle could be obtained from either a backup’s log file or recovering an older controlfile copy that the current controlfile is aware of9. However, based on my experience, if an organisation uses a file system (e.g. NFS) to store backups then it is rare when a retention policy is greater than a reasonable sized controlfile could handle (e.g. 30 days). If an organisation uses a tape solution, then most often the retention policy is managed by MML layer and therefore catalog does not play any role in a backup’s purging process. The controlfile size limitation could be a legitimate reason to introduce a catalog database, however as you can see there are still some methods to address some of the challenges without requiring a catalog database. DUPLICATE does not need a catalog database. Some of you may say that RMAN DUPLICATE from backups without target connection 10 requires a catalog database. As a matter of fact there is the following statement in the oracle 11GR2 documentation - “RMAN can perform backup-based duplication with or without either of the following connections: Target, Recovery catalog”. DUPLICATE does not require catalog database. You may want to review the “Creation Of Rman Duplicate Without Target And Recovery Catalog Connection. [ID 1113713.1]” note for additional details. 7 There is a risk however that the current controlfile may be on the different version. This is why the suggestion is to include operation system and oracle home backups along with the database backup that you are planning to keep longer than normal retention policy. 8 With exception of recovery process. In this case we will need to restore “old” controlfile from tapes and then process with regular recovery. However recovery from “old” backup may treated as exceptional case and therefore may not be significant enough reason for a catalog database introduction. As an additional argument I would like to mention that restore from an “old” backup often involves more complex activities (e.g. OS and Oracle Home restore) to be treated as exception. 9 This method could be time consuming as we may need to recover several controlfiles before getting to the right one. In this scenario storing backups’ log files may be more reasonable solution. 10 This is a new 11G R2 feature
- 5. This leaves us with three legitimate technical reasons when you may need catalog database 11. If you found that one or a combination of the reasons for RMAN catalog database applies to your configuration and you cannot use possible solutions suggested then you may read the next section to learn about possible additional benefits you can leverage from the catalog database. Others may find that the benefits are more valuable than associated costs and therefore leverage catalog database benefits. Benefits of using RMAN catalog / VALUE We have discussed COSTs involved in creating and maintaining catalog database as well as reviewed cases where catalog database is a MUST. It is time to focus on the VALUE added by catalog database. Let us start with a simple and easy to implement optional feature. ADDITIONAL CONTROL: An additional, centralized control level for day to day backups. I like this idea very much. If we have a catalog database with hundreds if not thousands of databases registered and backed up on regular basis then a report could be introduced that would notify a DBA team’s manager (or backup DBA, oncall, etc) about problems with last night backups or just provide a summary for review. It would not replace proper error handling functionality with oncall DBA notification 12 built within the backup process. However, if a DBA manages hundreds of databases then he/she may miss some of the issues (I was there personally). This option introduces a “safety fuse”. It protects an organisation from possible human and technical errors. If I was a DBA manager, I would strongly consider this option so I may sleep soundly at night. MONITORING & REPORTING: Backups’ volumes and throughput day to day monitoring. One of the main Oracle DBA responsibilities is to ensure that database backups run well. Catalog database provides an easy and centralized day to day monitoring repository. o You may introduce several reports that would help you identify any changes in the backups patterns (e.g. backup network slowdown or backup volumes increase because of redo volumes increase etc) without connecting to each of possibly many database servers. o It also could be used as a troubleshooting tool if backup run time increases (comparing the backup volumes, throughput etc with previous executions). For example if there is an archived logs volume increases you may check TOP objects with most block changes or if the volumes are the same it could indicate a problem with backup network or MML layer. o As catalog database contains information about backups’ timing, volumes and compression rates you can use this information for tuning backups, changing parallel parameters and control the impact. o Some service based organisation may charge their clients (e.g. internal clients, projects etc) per backup GB. The catalog database could be used to measure and report the backup volume per database or set of databases, simplifying cost estimates. CAPACITY PLANNING: Depending on the retention policy used and backup cycles it could be challenging to estimate the total capacity necessary for all databases’ backups. As an example, if NFS is used as storage for backups and there are hourly archive logs, daily incremental and weekly full backups then total storage used by the database backup may change significantly 13 depending on the day of the week. Capacity planning may become even more challenging if backups are executed from multiple RAC nodes and there are several databases that use the same NFS mount for storing backups. A centralized catalog makes the space utilisation reporting and capacity planning easier. Please bear in mind that records about backups are removed from the catalog database at the time that backups are obsoleted and deleted based on the retention policy used. There is no difference from that perspective between a controlfile or database based catalog. Therefore for capacity 11 If you disagree and know other reasons please get in touch with me (google my name). Be sure that I am quite happy to be wrong and will include your reason and mention your name (if you ok with it) 12 I strongly believe that oncall DBA must be paged about any backup related problem. As backup is one of the most critical tasks that a DBA manages and responsible for. Therefore he/she should be notified about any problem immediately. Ok, ok there are some exceptions could be made. But those should be very well controlled. 13 The maximum difference is equal to size of all weekly incremental and all archive log backup volumes.
- 6. planning you may want to introduce an additional (custom) set of tables to store day to day measurements for long term capacity planning. MML: MML backups tend to be stored longer than file system backups, therefore there is a tendency to use an RMAN catalog in such configurations. However, quite often MML layer has its own retention policy and RMAN is rarely used to manage the backup retention policy. Therefore, if we ensure that we can restore a controlfile from MML (e.g., using MML repository or backup log files) we may not need an RMAN catalog database. See the “Catalog & MML integration” section for additional details on this topic. RMAN SCRIPTS: A catalog database allows us to store RMAN scripts so they can be reused to backup databases in a standard way across an enterprise. This allows the use of unified backup scripts across many (if not all) databases. The scripts are pulled out from the catalog database each time a backup is performed. Along with RMAN default configuration settings stored on a catalog database, this makes it easy to control and adjust backup scripts without connecting to each server and adjusting every single backup script. o Cons: Backups become highly dependent on the catalog database. If it is not available then ALL backups fail. Alternatively we can make local backups less dependent on the catalog by using local shell scripts, and even if the catalog database is not available backups still run (synchronizing backup metadata later on at the time the catalog becomes available). EASY CONTROLFILE RECOVERY: Independent of the configuration you use, catalog database makes controlfile recovery very easy and a standard procedure across all Oracle environments. In most cases, you just run a single command to recover a controlfile. However you only will need this in the scenario where there is complete loss of the current controlfile. This concludes my list of additional value added options. Now it is up to you to put things together and present it to your system business owner. If you have used an RMAN catalog database before reading this paper you may consider implementing some of the ideas I listed above in order to streamline your operations and get more value from the configuration. Catalog usage In this section I would like to discuss some of the important RMAN catalog database usage scenarios and options you should be careful with. Let me start with a controlfile recovery from an auto backup. While this is not directly related to the catalog database it may explain an additional use case compared to a configuration without a catalog database. CONTROL FILE AUTO RECOVERY From a complete database recovery point of view (e.g., complete server or storage loss 14) the main task for which we possibly need a catalog database is a controlfile restore. As soon as the controlfile is restored, in most cases we do not need a catalog database as the controlfile contains all information necessary for further recovery. It is not as big a problem for file system based backups (e.g., NFS filers) as for tape based backups. Quite often we need to know a “handle” name15 to restore a file or group of files from tapes (MML). Some of you may say that if we configure a controlfile auto backup (RMAN command “CONFIGURE CONTROLFILE AUTOBACKUP ON”) then the controlfile recovery problem is solved. Do not say that too quickly. The problem with controlfile auto recovery from MML is that a restore of a single control file could take several hours. A controlfile's auto backup uses a standard handle name (%F format). The handle name consists of the following components: “c-“ + “database ID” + “date” + “XX”, where “XX” is a hexadecimal sequence that starts with 00 and has a maximum of FF. The sequence increases each time a new controlfile auto backup is created in a given day. It is important to mention that the index resets daily. At recovery time, RMAN, in order to recover the freshest auto backup copy, tries to use a handle name with FF at the end, and then FE and so on until MML sends back a stream of data instead of returning a file/stream not found error. Sometimes it may take up to 240 roundtrips to get to the right index/handle name. It is not rare to see one roundtrip to MML service 14 Please note that while full recovery scenarios happen, most often DBAs deal with cases where either a current controlfile copy is available or it is known from where to recover a controlfile (e.g., cloning from production). Therefore full recovery happens rarely. Nevertheless a DBA should be ready for a worst case scenario. 15 A name assigned to a backup stream at the time of backup.
- 7. taking 2-5 minutes. After a simple calculation it brings us to 8-20 hours16 just for a controlfile recovery. Because of the way a controlfile auto backup works, controlfile file auto recovery may not be an acceptable option to use for a controlfile recovery. An alternative would be to provide a handle name within the recovery command. The handle name could be found in a backup log file or from the MML repository. CATALOG & MML INTEGRATION The next catalog database usage scenario I would like to cover is catalog and MML (tape) configuration. This is a configuration where I personally find a catalog database most reasonable to use, as it reduces significant risks and streamlines the recovery process. CONTROL FILE RECOVERY One important step in the recovery process is a controlfile restore. Depending on your recovery parameters you may want to restore a controlfile from older backups rather than from latest backup (e.g., auto backup). The good news is any controlfile that contains information about a backup you want to restore from will be sufficient for a recovery. This means you have flexibility to restore a controlfile from up to several days after a target backup has been taken. The length of the interval will be dependent on retention policy and controlfile size (control_file_record_keep_time). Let us list several common ways to restore a controlfile from MML repository WITHOUT an RMAN catalog database to make a case for use of a catalog database: - MML REPOSITORY: First of all it is always possible to find a stream of data using MML repository that most likely will be a controlfile. Most MMLs repositories have information about the client that sent each stream of data and the time when it happened. It is just matter of luck and time (trial and error method). However, often in a restore scenario we do not have much time. The fact that there could be several databases running on the same MML client (DB server) could introduce an additional challenge. - RMAN LOG: Recovery Manager records handle names each time it sends data to MML in log files. This is one additional argument for why it is important to store RMAN log files in an easily accessible place (e.g., most commonly the DBA team’s mailbox). This can reduce the trial and error time significantly - HANDLE: For most RMAN & MML configurations it is possible to assign a custom handle name at the time you execute a backup. The method to do so is dependent on the MML software. Most commonly you either set it via RMAN “ENV=” clause or “format” channel keyword. If you assign a meaningful name to a controlfile backup piece (e.g., DB name, date & time) then it will be an easy task to identify a handle you are interested in from the MML repository as it lists all the handles' names. As you can see there are options that allow you to restore a controlfile from MML without using a catalog database. The third option even allows you to pass metadata information to MML that allows you to identify a controlfile “handle” you need to recover from the MML side without any issues. However it requires very close cooperation between the Oracle DBA and the tape system’s management team. In many organizations those two teams do not talk to each other very often. Sometimes MML management is outsourced to third organization. Communications between Oracle DBA and MML teams may take significant time, if possible at all. However RMAN catalog database allows avoiding sometimes “difficult” communications. For many DBA teams it is easier to introduce a catalog database rather than streamline communications that may be a crucial component in a potential restore process. For tape administration teams it has its own benefit: they do not need to provide access to MML repository data to Oracle DBAs. Finally, a catalog database introduces a standard way for restoring controlfiles and databases. This means that even an external Oracle DBA who is not familiar with an organization's specific setup (e.g., log file locations, MML integration details, etc) would not have significant problems restoring controlfiles or databases. If an organization has hundreds of databases it often much easier (cheaper) to introduce a catalog database than to work on other backup and restore options. In addition, the organization could use the other benefits of an RMAN catalog to justify maintenance costs. 16 You can reduce controlfile recovery time from autobackup dramatically with maxseq and maxdays options. See restoreSpecOperand, FROM AUTOBACKUP from Oracle® Database Backup and Recovery Reference 11g Release 2 (11.2)
- 8. RETENTION POLICY MANAGEMENT Typically an organization uses centralized MML services where Oracle databases are just one client of many (others could be: filesystems, MS Exchange, other databases, etc). In such a configuration there are often common backup data retention pools defined (e.g., 1-2 weeks storage, 1-2 month storage & long term or custom) to ensure effective tape life cycles and recycling. RMAN is not used to ensure the backup retention policy and there is a bit of a challenge to synchronize RMAN and MML layer data. RMAN may still contain data about backups that are not available in MML. One of the common approaches used in such a case is to change the status of old backups that have crossed the tape retention policy to UNAVAILABLE to keep records for capacity planning purposes and to UNCATALOG at the time we no longer need the records. Please note that the DELETE command will delete records from the catalog database and they would no longer be available for further capacity planning activities. As the RMAN catalog and MML data are not synchronized in an automatic way there could be situations when RMAN would try to request a non-existent backup. This configuration should be carefully planned and tested to avoid such situations. CATALOG & DR One of the configurations where an RMAN catalog is a MUST is if an organization takes backups from both sites or from the Disaster Recovery (DR) site only 17. If you are running backups from both primary and standby databases you should be aware about backups associations to different sites (SITE_KEY column in RMAN catalog view). Any disk based backups considered as local to the site/database that they have been taken from. Tape based backups are considered to be accessible from both sites. You may want to review “RMAN Backups in a Data Guard Environment” from “Oracle Database Backup and Recovery Reference 11g Release 2 (11.2)” for more information. Special care should be taken running backups in Disaster Recovery configuration and an RMAN catalog is required to complete the task. CATALOG & FS Some may think that this is a very simple configuration, and in fact it is, unless there is a separate file system backup that backs up your RMAN backup directories. In such a case I would strongly suggest you consider implementing MML integration as otherwise you are at risk of either missing copying some RMAN backup files to tape (putting recoverability at risk) or making too many tape backups for some of the RMAN backup files. You may find the reasons behind this advice in the “10 Problems with your RMAN backup script” paper. Despite the warning, many organizations run such a configuration. Therefore in the context of an RMAN catalog and recoverability I would like to suggest you to keep RMAN metadata records in the catalog repository as long as backup files are available on tapes. Do not use DELETE OBSOLETE commands. This ensures that the RMAN catalog repository is “aware” of backups available on the tapes and you could use the “RESTORE … PREVIEW” command to get the full list of files necessary for an intended recovery. Practical hints for catalog usage I am providing additional hints for catalog database usage for your consideration under this section. Most of the information here, for one reason or another, did not find a good place in the previous section. DO NOT SEPARATE DEVELOPMENT & PRODUCTION CATALOGS Having a centralized RMAN repository across development and production environments will make your environment cloning (DUPLICATION) efforts easier, avoiding unnecessary additional operations. You still may want to have an RMAN development environment for upgrade testing. You may want either to keep a database synchronized with both catalogs or to switch some development databases to the second catalog only for the duration of your upgrade testing. DBID MUST BE DIFFERENT FOR ALL DATABASES This is rather small comment. Before Oracle introduced the DB New ID utility (nid) it was a common practice to have multiple databases with the same ID. Nowadays the DUPLICATE command changes the DB’s ID for you and if you 17 I would strongly suggest this option in DR configuration as it allows you to take backup load away from the production system and to create offsite backups at the same time.
- 9. need to you can run the “nid” utility to change the ID yourself. You cannot register multiple databases with the same ID under the same RMAN catalog repository, with the exception of databases in a Data Guard configuration. TO USE OR NOT TO USE RMAN CATALOG STORED SCRIPTS? The RMAN stored scripts are a very powerful feature that simplifies backup script change management across the whole enterprise network. Before each backup the catalog stored script is retrieved from the repository. It is enough to change a script in one place to adjust all or a subset of databases' backup procedures. It pays off in environments with many databases (e.g., private clouds). Shell backup script management may introduce significant overhead in such environments. However, if we introduce catalog stored RMAN scripts we should make sure that the RMAN catalog database is highly available. Otherwise backups will fail when unable to access the backup scripts. An alternative solution is to sacrifice manageability and make sure that backups may happen even if a catalog is not available. RMAN SETUP FOR CATALOG DATABASE FAILURES It is relatively simple to ensure that backups keep running even if a catalog database for one reason or another is not available. First of all you must use shell backup scripts as opposed to RMAN catalog stored scripts. Ensure that the catalog connection string is executed within the RMAN script’s body instead of connecting to a catalog when calling the rman executable. This way even if the catalog database is not available, the backup will continue, reporting a warning on catalog database unavailability. Do not use the following syntax in your backup scripts: rman target / catalog user/password@catalog. This command will fail if the catalog database is not available because of network or other issues. USE RESYNC CATALOG Instead of keeping a connection to a catalog database during the whole backup, we could connect to the catalog database and issue “resync catalog” command at the end of each backup. This provides several benefits, including the following: - Reduces RMAN catalog dependencies for long duration backups. The backup will need connectivity to a catalog database only for a very short time at the end of each run. - If you use traditional connectivity a backup will fail if there is a connection failure to a catalog database during the backup execution. However this is avoided, reducing failures, if we use “resync catalog” at the end of each backup. - Backups will run with no problems even if an RMAN catalog database is not available (e.g., if there is ongoing RMAN catalog maintenance). The backup metadata will get synchronized during the next backup execution. INTRODUCE TWO CATALOG DATABASES TO ENSURE HIGH AVAILABILITY There is nothing that would prevent you from registering a database in two catalog databases if you need it. One potential use case is a DR solution where there are two catalog databases on both sites. Each database would connect to both catalogs at the end of each backup run and issue the “resync catalog” command to keep both up to date. This effectively ensures the catalog database's high availability in the case of a disaster recovery scenario and there is at least one catalog database to synchronize with in case the other RMAN catalog database is unavailable for a maintenance process. Note that a database can resynchronise with the catalog database any time, even after a few weeks out of synchronisation. This solution could be used as an alternative to Data Guard that potentially may trigger additional licensing 18. 18You must check the licensing question with you Oracle Sales representatives. This paper has a technical nature and cannot be used as a reference in licensing questions. The only purpose here is to provide a reader with different possible technical solutions to ensure RMAN catalog database high availability.
- 10. KEEP RMAN CATALOG ON THE SAME HARDWARE AS MML SOFTWARE If possible, you may consider keeping both the RMAN catalog database and MML software on the same piece of equipment. These two components have the same availability requirements. If MML is not available, the fact that the catalog database is unavailable would not make any difference. However, while this configuration saves some resources, it makes MML server dependent on Oracle catalog requirement and vice versa. A proper capacity planning should be put in place before making such decision as MML side processes quite often as resources intensive. Conclusions Databases backup is an Oracle DBA’s primary responsibility area. It is our responsibility to ensure that databases have a valid backup and could be restored based on business requirements at any time. We should streamline and bullet proof the recovery process. The best time to prepare for a comfortable recovery is weeks, months or even years before you need to execute it. An RMAN catalog database helps simplify potential recovery and provides a suite of other benefits. However any additional database in an organization’s network increases operational costs. Today when organizations have limited budgets and resources any cost increase needs to be justified. Many Oracle DBAs do not have a strong justification for RMAN catalog database. Typically if an RMAN catalog database exists it was created for historical reasons or because of Oracle recommendation without requirements analysis. This paper provided you with list of scenarios where an RMAN catalog database must be used because of technical dependences and additional value added and risk reducing use-cases. In large environments with hundreds or even thousands of databases RMAN database is easier to justify as its management costs spread across many databases and value increases as it helps to manage large number of databases reducing related maintenance costs. In configurations where RMAN is integrated with MML (Tapes based backup solution) catalog database simplifies recovery operations. It does not apply to technical procedures only. It also reduces the number of people and communication steps involved in most recovery scenarios and therefore simplify and reduce the time necessary for a database recovery. On the other hand it is simpler to restore a database from file system based backups. In such configuration Oracle DBAs should assess other benefits that RMAN catalog database provides. If a catalog database exists in the environment you are managing already consider implementing other use-cases you and your team can leverage from, simplifying and bulletproofing backup operations and reducing recoverability risks.