Presentation backup and recovery best practices for very large databases (vld bs)
- 1. 1
- 2. 2
- 3. <Insert Picture Here> Backup and Recovery Best Practices for Very Large Databases (VLDBs) Tim Chien Senior Product Manager Oracle Greg Green Senior Database Administrator Starbucks
- 4. 4 <Insert Picture Here> Agenda • VLDB Trends and Requirements • Backup and Recovery Design • Assess Recovery Requirements • Backup and Recovery Deployment • Architect Backup Environment • Evaluate Backup and Recovery Tools • Plan Data Layout • Develop Backup Procedure • Develop Recovery Strategies • Starbucks Case Study • Summary/Q&A
- 5. 5 • Rapidly growing databases • 1 TB+ is ‘normal’ for initial deployments => 100 TB++ • Growth of intranet and internet resources • Data consolidation • Global view of the business • Integrated, targeted analysis critical to business decisions • Concurrent, heavy access • Global user base – internal/partner/customer VLDB Trends & Requirements Need to be cost-effective while scaling for data growth
- 6. 6 VLDB Backup & Recovery • Key database protection requirements: • Growing data – how to backup/restore in tolerable window? • Consolidated data – how to recover at an application level? How to protect data in 24x7 environment? • Cost efficiency with scale – how to manage software, infrastructure, operational costs for data protection/disaster recovery?
- 7. 7 <Insert Picture Here> Agenda • VLDB Trends and Requirements • Backup and Recovery Design • Assess Recovery Requirements • Backup and Recovery Deployment • Architect Backup Environment • Evaluate Backup and Recovery Tools • Plan Data Layout • Develop Backup Procedure • Develop Recovery Strategies • Starbucks Case Study • Summary/Q&A
- 8. 8 Backup and Recovery Design Assess Recovery Requirements • Analyze and identify the cost of unavailable data • Design recovery requirements around data criticality and logical relationships • Assess tolerance for data loss - Recovery Point Objective (RPO) • Prioritize data • Point-in-time recovery required? • Assess total recovery time - Recovery Time Objective (RTO) • Problem identification & recovery planning time + H/W & S/W recovery time + database recovery time (media + instance recovery) • Determine backup retention policy
- 9. 9 <Insert Picture Here> Agenda • VLDB Trends and Requirements • Backup and Recovery Design • Assess Recovery Requirements • Backup and Recovery Deployment • Architect Backup Environment • Evaluate Backup and Recovery Tools • Plan Data Layout • Develop Backup Procedure • Develop Recovery Strategies • Starbucks Case Study • Summary/Q&A
- 10. 10 Backup and Recovery Design Architect Backup Environment Cost Disaster Recovery Performance Compatibility, Provisioning VTL license + disk cost* Optional file-based replication Fast, random I/O access Standard tape interface Virtual Tape Library (VTL) Well-known interface across heterogeneous systems Tapes easily added, as needed OS, drivers, storage must be compatible Disk should be provisioned for ‘worst’ case scenario Sequential-only accessFast, random I/O access Offsite tapes to DR site, long-term archival Optional I/O-based mirroring Best price/capacity, e.g. LTO-3 ($60 list) holds 800 GB compressed Price/capacity starts at few dollars/GB (ATA) Tape LibraryDisk *File-level deduplication may reduce disk cost
- 11. 11 Backup and Recovery Design Architect Backup Environment • Tape and disk considerations for VLDB • Allocate disk backup for most critical tablespaces • Utilize locally attached tape drives • If tapes must be shared, consider disk and/or tape backup SAN • Assess host resource utilization, production disk I/O, HBA/network & tape drive throughput • Minimum performant component of these will be bottleneck
- 12. 12 <Insert Picture Here> Agenda • VLDB Trends and Requirements • Backup and Recovery Design • Assess Recovery Requirements • Backup and Recovery Deployment • Architect Backup Environment • Evaluate Backup and Recovery Tools • Plan Data Layout • Develop Backup Procedure • Develop Recovery Strategies • Starbucks Case Study • Summary/Q&A
- 13. 13 Backup and Recovery Deployment Leverage Oracle Backup and Recovery Tools • Recovery Manager (RMAN) • Oracle-native tool for creating and managing physical backups, restoring and recovering databases • Multiplexed, parallel, block-validated full and incremental backups scale with VLDB needs • Supports data file, tablespace, and database backup and recovery, in addition to block media recovery • Flash Recovery Area manages all recovery-related files, e.g. backups, archived logs • Data Pump • Logical backups of metadata and data (e.g. table, database)
- 14. 14 Backup and Recovery Deployment Leverage Oracle Backup and Recovery Tools • Oracle Secure Backup (OSB) • Centralized tape backup management for database & file system • Fastest database backup to tape on the market via elimination of unused blocks and committed undo during backup • Low price of $3,000/tape drive • Flashback Technologies • Broad suite of user error analysis and correction tools at the row, transaction, table, and database level • Optimized, continuous database snapshot with Flashback Database (built-in Continuous Data Protection (CDP) capability) • Data Guard • Manages fast switchover and failover from production database to synchronized standby database on separate hardware • Serves as a remote, synchronized database ‘backup’
- 15. 15 Error Correction with Flashback • Correct errors at any level • Recovery time is function of changes, not database size • Flashback Database – restore database to point-in- time (flashback log-based) • Flashback Table – recover contents of tables to point- in-time (undo-based) • Flashback Drop – restore accidentally dropped tables (based on free space in tablespace) • Flashback Transaction – back out transaction and all subsequent conflicting transactions (redo-based) 0 10 20 30 40 50 60 70 80 Time To Recover (minutes) Traditional Recovery Flashback Database Customer Order
- 16. 16 Backup and Recovery Deployment Third Party Disk Backup Solutions • Snapshots • Fast point-in-time copy, but exposed to production array failures • Split Mirror Backup • Tape backup can be offloaded, but additional setup required • Starbucks will present their criteria evaluating RMAN+Flash Recovery Area with split mirror backup • CDP Appliances • Continuous snapshot via copy-on-write or allocate-on-write • Only data file block changes are tracked • Only one before-image is logged per 30 min interval, regardless of number of changes to the block • All file changes must be logged (control file, redo, data file) • Each and every block change must be logged I/O consumption Third Party CDP Flashback Database Additional Cost Additional Cost Additional Cost
- 17. 17 <Insert Picture Here> Agenda • VLDB Trends and Requirements • Backup and Recovery Design • Assess Recovery Requirements • Architect Backup Environment • Backup and Recovery Deployment • Evaluate Backup and Recovery Tools • Plan Data Layout • Develop Backup Procedure • Develop Recovery Strategies • Starbucks Case Study • Summary/Q&A
- 18. 18 Backup and Recovery Deployment Plan Data Layout • Exploit partitioning and read-only tablespaces • Older partitions can be moved to read-only tablespaces • Backup read-only tablespaces once, then periodically, depending on tape retention policy • Table Compression – reduce space by 2-3X, no performance degradation accessing compressed data • OLTP Table Compression – minimize write overhead NOV07JUL07 AUG07 SEP07 OCT07 Read-only (older data) Read-write (recent data)
- 19. 19 <Insert Picture Here> Agenda • VLDB Trends and Requirements • Backup and Recovery Design • Assess Recovery Requirements • Architect Backup Environment • Backup and Recovery Deployment • Evaluate Backup and Recovery Tools • Plan Data Layout • Develop Backup Procedure • Develop Recovery Strategies • Starbucks Case Study • Summary/Q&A
- 20. 20 Backup and Recovery Deployment Develop Backup Procedure • Option 1: Use Level 0 (Full) and Fast Incremental Backups (Oracle Database 10g+) • Weekly level 0 and nightly ‘differential’ incremental backups to tape, with optional compression • Only changed blocks are read and written during incremental • Archived logs are backed up and retained on-disk, as needed • Best for: • VLDBs that can tolerate hours/days for restore & recovery • Tape backup-only environments, where disk is premium • Low-medium change frequency between backups Level 0 (full) Level 1 (incremental) …. Archived Logs Archived Logs
- 21. 21 Backup and Recovery Deployment Develop Backup Procedure • Option 2: Use Level 0, Fast Incremental Backups, Incrementally Updated Backup • Initial level 0 image copy to disk, followed by nightly incremental backups • Roll forward image copy with incremental, to produce new on-disk full backup, on regular basis (e.g. daily/weekly/monthly) • Full backup archived to tape, as needed • Archived logs are backed up and retained on-disk, as needed • Best for: • VLDBs that can tolerate no more than several hours for restore & recovery • Fast recovery from disk or directly use image copies • Environments where disk can be allocated for full database image copy or image copy of most critical tablespaces Level 0 (full) + archive to tape Level 1 …. Roll fwd image copy + Level 1 Archived Logs Archived Logs Archived Logs
- 22. 22 Backup and Recovery Deployment Develop Backup Procedure • Option 3: Data Guard + Full/Incremental Backups • Full and incremental backups can be offloaded to physical standby database and used for restore at primary or standby database • Backups can be taken at each database for optimal local protection • Best for: • VLDBs that require no more than several minutes of recovery time, in event of any failure • Environments that can allocate symmetric hardware and storage for physical standby database • Environments whose tape infrastructure can be shared between primary and standby database sites
- 23. 23 Backup and Recovery Deployment Develop Backup Procedure • Option 4: Maintain ETL loads • Take full backup and maintain n ETL loads, where n is number of days between full backups • Restore last full backup and run needed ETL loads • Manual backup and recovery process versus Options 1-3 • Best for: • VLDBs that do not require point-in-time recovery or do not have space for archived logs • VLDBs whose changes are confined to controlled batch jobs • Environments that cannot keep both incremental backups and load files • Legacy systems that will not be upgraded Level 0 (full) Save ETL load … Level 0 (full) …
- 24. 24 Backup and Recovery Deployment Develop Backup Procedure • NOLOGGING considerations • Data warehouse environments commonly use NOLOGGING to increase performance and save space • Recovery to point-in-time within NOLOGGING results in some corrupt blocks • Best practice is to take incremental backup when NOLOGGING finishes, so ideal for controlled batch load environments • E.g. Option 1: Level 0 + Fast Incremental Backups Level 0 (full) Level 1 (incremental) …. Archived Logs Archived LogsNo logging
- 25. 25 Backup and Recovery Deployment Develop Backup Procedure • Other considerations • Divide full backup workload across multiple days • BACKUP DATABASE NOT BACKED UP SINCE ‘SYSDATE-3’ DURATION 06:00 PARTIAL MINIMIZE TIME; • Manage backup window and user impact • Full restore requires data file backups potentially taken on several tapes • Save time with tablespace level backups • Backup index tablespaces less frequently than data tablespaces • Backup little used/less critical tablespaces less frequently • Reduce restore time for most critical tablespaces, by grouping them together in separate backups
- 26. 26 Backup and Recovery Deployment Develop Backup Procedure • Oracle Database 11g Enhancements • Multi-section backups (intrafile parallel backup) • Best for databases composed of a few data files that are significantly larger than the rest (e.g. bigfile tablespace) • Or, if there are fewer data files than # of tape drives • Scales linearly with addition of channels • Start with section size = average data file size / # of channels • Fast backup compression (ZLIB) -- 40% faster than Oracle Database 10g • Suitable for daily, incremental backups • Use BZIP2 for full backup -- better compression ratio • Data Pump export compression, including metadata & table data
- 27. 27 <Insert Picture Here> Agenda • VLDB Trends and Requirements • Backup and Recovery Design • Assess Recovery Requirements • Architect Backup Environment • Backup and Recovery Deployment • Evaluate Backup and Recovery Tools • Plan Data Layout • Develop Backup Procedure • Develop Recovery Strategies • Starbucks Case Study • Summary/Q&A
- 28. 28 • Data Recovery Advisor – built-in database failure diagnosis, analysis, & repair tool Backup and Recovery Deployment Develop Recovery Strategies RMAN – restore all files to new storage location Media Failure RMAN Block Media Recovery, Trial Recovery, LogMiner Block Corruption Flashback Technologies, RMAN TSPITR, LogMiner User/Logical Error Data Guard; RMAN -- restore all files to new host/storage Disaster Oracle TechnologiesRecovery Scenario
- 29. 29 <Insert Picture Here> Starbucks Case Study
- 30. 30 <Insert Picture Here> Agenda • VLDB Trends and Requirements • Backup and Recovery Design • Assess Recovery Requirements • Architect Backup Environment • Backup and Recovery Deployment • Evaluate Backup and Recovery Tools • Plan Data Layout • Develop Backup Procedure • Develop Recovery Strategies • Starbucks Case Study • Summary/Q&A
- 31. 31 Resources • Maximum Availability Architecture white papers: http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm • Oracle HA Portal on OTN: http://www.oracle.com/technology/deploy/availability/ • Oracle HA Customer Success Stories on OTN: http://www.oracle.com/technology/deploy/availability/htdocs/HA_Case Studies.html
- 32. 32 Summary • VLDBs growing rapidly • More multi-PB systems in few years? • Methodically work through backup and recovery design and deployment • Understand Data Layout and Change Characteristics • Assess Recovery Requirements • Architect Backup Environment • Put design to the test with deployment • Leverage Oracle Backup Tools & Strategy • Develop and Test Recovery Strategies
- 33. 33 Q U E S T I O N S A N S W E R S
- 34. 34 •S291495 - Oracle Streams Replication and Advanced Queuing (AQ): What's New in Oracle Database 11g, 8:30 - 9:30 am, Moscone South 304 •S291499 - Best Practices for Implementing Replication with Oracle Streams in Oracle Database 10g and 11g, 10:00 - 11:00 am, Moscone South 304 •S291525 - Maximum Availability Architecture (MAA) Best Practices: Online Patching, Rolling Upgrades and Planned Maintenance with Minimal Downtime with Oracle Database, 11:30 am - 12:30 pm, Moscone South 104 Database HA Sessions From Oracle Development •S291920 - Oracle Active Data Guard: How to Utilize Your Standby Databases for Production Workload - What They Didn’t Print in the Manuals, 3:00 - 4:00 pm, Moscone South 304 •S291917 - Oracle Data Guard Tips and Tricks: Direct From Oracle Development, 4:30 - 5:30 pm, Moscone South 102 Wednesday, Nov 14 Thursday, Nov 15 •S290542 - Maximum Availability Architecture (MAA) Best Practices for Siebel 8.0, 2:30 pm - 3:30 pm, Marriott Salon 10 & 11 •S291487 - Backup and Recovery Best Practices for Very Large Databases (VLDB), 11:15 am - 12:15 pm, Moscone South 104
- 35. 35