Presentations bu 8

Editor's Notes

• #2: An obvious question is – “Why do we need to address Accounting Databases specifically?” In my experience this is a much neglected area – I have even been answered, to my concern regarding backups – “It’s OK, we have paper printouts.” An accounting system is basically an ‘Off-the-shelf’ product which the users do not know much about and do not have access to the back-end programs or data. Typically accountants leave it to the IT department (often the IT department has no DBA) not even considering whether the IT department has the internal expertise to restore the database. This is especially true regarding a ‘Point-in-time’ restore. Most network administrators have never performed a PIT restoration. Ostrich management? The bottom line is - Accountants should be involved with their backups and not bury there heads in the sand! There needs to be documented procedures in place and there needs to regular routines that perform restores to test those procedures. Backups may be worthless unless they have been tested and you will not know their worth until you need them. To think outside the usual box, there can be some operational benefits to be gained from the use of backups. For example, we can create copies of our database for taking the load off our OLTP system or use them as the foundation for analysis reporting. We can also use them as Decision Support Systems (DSS) from which we can run those huge reports, we can use restored backups as test or development databases and gain other benefits as well. Some of the techniques discussed here are for use with large systems but smaller companies need to consider the advantages to be gained and learn how they can use some very simple routines to gain the benefits usually only available only to larger companies.
• #3: DSS (Decision Support System) = a restored backup of the Production OLTP (Online Transaction Processing) database. An Example: If you backup and restore every night then you have a database that you can index extensively without affecting the OLTP database. An advantage is that all reports run against that database will be consistent as of End of Business yesterday. Test Companies can be used for training new staff, testing bulk uploads of data, ‘what-if’ scenarios, testing if the backup restores OK, etc. Development companies may be created from backups and used by developers who need a true example of the database for developing programs and add-ons. Scoreboards are used to hold aggregate totals and denormalized data for easy reference and performance. Lookups and Read-only databases can be comprised of many adaptations so that the production database does not have to carry the load. Examples: Photos of employees could be carried in a separate database. Price lists can be kept separate (Note: the permutations are endless – they may, or may not, be based on backed up databases).
• #4: DSS (Decision Support System) = a restored backup of the Production OLTP (Online Transaction Processing) database. An Example: If you backup and restore every night then you have a database that you can index extensively without affecting the OLTP database. An advantage is that all reports run against that database will be consistent as of End of Business yesterday. Test Companies can be used for training new staff, testing bulk uploads of data, ‘what-if’ scenarios, testing if the backup restores OK, etc. Development companies may be created from backups and used by developers who need a true example of the database for developing programs and add-ons. Scoreboards are used to hold aggregate totals and denormalized data for easy reference and performance. Lookups and Read-only databases can be comprised of many adaptations so that the production database does not have to carry the load. Examples: Photos of employees could be carried in a separate database. Price lists can be kept separate (Note: the permutations are endless – they may, or may not, be based on backed up databases).
• #5: This is a development from the previous slide – it shows tempdb (which is important for tuning performance) and the log file which needs to be on a separate drive.
• #6: As part of the overall Disaster Recovery Plan , overall responsibility lies with the Board of Directors (as with everything else). The Board is responsible to the Shareholders for the continued operations of the company. Typically: The Officers follow the directions of the Board. The Managers implement the instructions of the Officers. Policies are approved at the Officer and then the Board level. Procedures are prepared by the managers for approval by the Officers.
• #7: 1. Disaster recovery Plans are common sense 2. SOX compliance requires BU & Restore procedures to be in place A question I have for you is – “Is there anyone here who either thinks that backups are unnecessary or that if they are done that they do not have to be done properly?” COBIT standards can be downloaded free from www.isaca.org. COBIT (Control Objectives for Information and Related Technology) was developed by the IT Governance Institute (ITGI) SOX adaptation 2003 – IT Control Objectives for Sarbanes Oxley - detailed 27 IT processes and 136 control objectives that are critical to SOX compliance. Research in 2003 by The Meta Group found that 39% of firms said SOX will eventually make them more competitive. COBIT requires adequate backup and restore procedures and they should be regular, tested and written/recorded. While there is quite a lot of feeling that SOX has gone too far there has not been anyone that says that it was/is completely unnecessary or that it is without any merit.
• #8: Since 9/11/2001 no one has said “It will never happen to us!” Prudence dictates that we do not leave the fate of the Company to chance - that we establish internal procedures that safeguard the continuity of the company should any of the risk areas become manifest. A complete Backup and restore plan should include the situation where a plane flies into the building (or something equally horrifying) and the company has to get up and running within a certain time, for example, 48 hours. Seriously and practically, and without the theatrics, - most errors are user errors and it should be a routine exercise to restore the database from a disk backup within the hour. This should be a procedure which is tested regularly.
• #9: Management decisions are often based on a Cost vs. Benefit analysis. If the costs outweigh the benefits we may feel that we should not perform the function. However, there are some results for which we cannot ‘take the chance.’ An often used formula is EMV = %Probability x $Loss – or stated another way - Expected Monetary Value can be defined as Probability times Economic Result. Example: 1% probability of a $1million loss = $10,000. This is sometimes recommended as the amount of money you should spend to avoid the risk and plays a part in the calculation of the basis an Insurance Company uses when setting premiums.
• #10: ISACA may be thought of as the International flavor of SOX with 50,000 members (worldwide). In Orange County there are approx 500 members. SOX needs no introduction - but I would like to say that a system of Internal Control is just common sense. CISA – Certified Information Systems Auditor. CISM - Certified Information Systems Manager. At present, both CISM and CISA are in high demand. The CITP is a fairly recent extension to AICPA membership for CPA’s specializing in IT and allows the use of CITP in conjunction with the CPA designation as CPA.CITP.
• #11: DSS (Decision Support System) = a restored backup of the Production OLTP (Online Transaction Processing) database. An Example: If you backup and restore every night then you have a database that you can index extensively without affecting the OLTP database. An advantage is that all reports run against that database will be consistent as of End of Business yesterday. Test Companies can be used for training new staff, testing bulk uploads of data, ‘what-if’ scenarios, testing if the backup restores OK, etc. Development companies may be created from backups and used by developers who need a true example of the database for developing programs and add-ons. Scoreboards are used to hold aggregate totals and denormalized data for easy reference and performance. Lookups and Read-only databases can be comprised of many adaptations so that the production database does not have to carry the load. Examples: Photos of employees could be carried in a separate database. Price lists can be kept separate (Note: the permutations are endless – they may, or may not, be based on backed up databases).
• #12: Nothing makes sense like backing up to a Hard Drive. It is faster, more reliable and more readily available in case of the need to restore. There has been a period when the predominant opinion was to backup to tape – maybe before hard drives became larger and cheaper. When you need to restore it makes perfect sense to restore from a hard drive copy of the backup.
• #13: Why don’t companies lose data? One reason is that they do not put the data on the same drive as the operating system. The C: drive is the most vulnerable to corruption, virus attack and is heavily used. I recommend = get 2 drives: Use 1 for the operating system and programs Use the other for data Put your paging file on the second disk (it usually gets used less and will increase performance) Note: Partitions MAY have an effect on the vulnerability of your C: drive but will not fulfill performance issues.
• #14: Many people seem to be under the impression that there is only one database. There are many systems that include multiple databases into their architecture. An Example is Microsoft Dynamics Great Plains which uses the Dynamics database to store GP system tables. This is in addition to their company databases. In additions to this, SQL Server itself uses the ‘master’ database for its server-wide metadata, the msdb database for job metadata and also backup details and the model database is used as a template for creating new databases. The log file is not a part of the database – it is a separate file. Another word to describe this file using an accounting term is ‘journal.’ This is a chronological record of every transaction that is written to the database. SQL Server does not write directly to the database files, it writes the transactions to the log then an internal process (called Checkpoint) writes the log file to the database tables. You can use this file when restoring a database to a ‘Point-in-time’ by a ‘play it again’ technique to bring the database to the required position. There are so many different scenarios – please allow me to catch-all by merely saying ‘other’ databases.
• #15: Many people seem to be under the impression that there is only one database. There are many systems that include multiple databases into their architecture. An Example is Microsoft Dynamics Great Plains which uses the Dynamics database to store GP system tables. This is in addition to their company databases. In additions to this, SQL Server itself uses the ‘master’ database for its server-wide metadata, the msdb database for job metadata and also backup details and the model database is used as a template for creating new databases. The log file is not a part of the database – it is a separate file. Another word to describe this file using an accounting term is ‘journal.’ This is a chronological record of every transaction that is written to the database. SQL Server does not write directly to the database files, it writes the transactions to the log then an internal process (called Checkpoint) writes the log file to the database tables. You can use this file when restoring a database to a ‘Point-in-time’ by a ‘play it again’ technique to bring the database to the required position. There are so many different scenarios – please allow me to catch-all by merely saying ‘other’ databases.
• #16: This is a development from the previous slide – it shows tempdb (which is important for tuning performance) and the log file which needs to be on a separate drive.
• #17: When designing storage there considerations other than space. The greatest bottleneck to performance is the physical (the speed of electrical signals is about .6 times the speed of light). Therefore, the if we increase the number of I/O heads we increase the speed of I/O. Online duplication is called Redundancy – write to 2 disks at once and we have twice the reliability – less risk of data loss from a problem with the hard drives.
• #18: When designing storage there considerations other than space. The greatest bottleneck to performance is the physical (the speed of electrical signals is about .6 times the speed of light). Therefore, the if we increase the number of I/O heads we increase the speed of I/O. Online duplication is called Redundancy – write to 2 disks at once and we have twice the reliability – less risk of data loss from a problem with the hard drives.
• #19: A stripe set is comprised of multiple disks joined in series into one volume. A mirrored set is comprised of multiple disks joined in parallel into one duplicated volume. Thus you will use twice the minimum storage space. A parity set is comprised of a set of hard drives arranged so that every volume has a duplicate of the data, compressed and stored on another disk. You ‘lose’ one disk from the set because of this compressed ‘parity’ copy of the data. The resultant storage space is n-1 disks. Raid 5 arrays enable hot-swop of disks. Raid 10 is a set of stripped disks that are them mirrored. They use twice the minimum storage space. Raid 10 arrays also enable hot-swopping disks.
• #20: The most susceptible disk is the one that holds the operating system. This is where problems are most likely to occur. Programs can also be installed on the ubiquitous ‘C:’ drive. This drive is capable of being clustered so that the user company can switch over to the another drive which holds the duplicate operating system and programs. OLTP databases are most suitable for storage on redundant systems. If the databases are small enough they could also just use a mirror which, at 2 disks, would be cheaper than even a Raid 5 system and just as advantageous as Raid 10). DSS databases do not need backups or redundancy as they can be recreated from a backup. Tempdb just needs speed since it is recreated each time SQL Server starts. Log files need redundancy and they are written heavily (by definition) but they are usually small enough not to require Raid 10 so a mirror is usually the answer. Backup files do not need backups and are never written to (unless written over) but they should be isolated from the other files unless there is a space problem.
• #21: This system is shown merely to provide the basis for a discussion of the different types of RAID systems and disk arrangements. Usually you would not design both a RAID 5 and a RAID 10 system – either have one or the other. However, a company usually end up with a compromise system which is driven by the philosophy – “we do not need to have the ultimate in performance as long as the system is fast enough to meet our needs.” Question: When entering data, how fast is fast enough? Answer: You should have sub-second response when entering data, which is the maximum time that passes before a data entry clerk loses momentum. Note: There is a Oracle standard that defines 8 secs as the maximum time that could pass before a person starts to think of something else.
• #22: DSS (Decision Support System) = a restored backup of the Production OLTP (Online Transaction Processing) database. An Example: If you backup and restore every night then you have a database that you can index extensively without affecting the OLTP database. An advantage is that all reports run against that database will be consistent as of End of Business yesterday. Test Companies can be used for training new staff, testing bulk uploads of data, ‘what-if’ scenarios, testing if the backup restores OK, etc. Development companies may be created from backups and used by developers who need a true example of the database for developing programs and add-ons. Scoreboards are used to hold aggregate totals and denormalized data for easy reference and performance. Lookups and Read-only databases can be comprised of many adaptations so that the production database does not have to carry the load. Examples: Photos of employees could be carried in a separate database. Price lists can be kept separate (Note: the permutations are endless – they may, or may not, be based on backed up databases).
• #23: OK, let’s answer some FAQs. Firstly -
• #24: Secondly -
• #25: Notice that this slide shows that the Cluster has 2 nodes (which hold the Operation system and the programs) but only 1 database!
• #26: Log-shipping is akin to – let’s copy our journal and post it to 2 copies of the General Ledger, (do it twice). We then have 2 copies, including our mistakes.
• #27: This is how log-shipping works, in the background.
• #28: The most common arrangement is to perform a Full backup every night. If this is what you wish this is what you get. Of course, if the database becomes in need of restoration the users will have to re-enter all data from the Full backup all over again. “ Hey, I enjoy doing things twice don’t you?” (sic) The cost of this in a 10 user company when the DB goes down at 4.30PM is 10 average man-days or 2 average man-weeks.
• #29: We will look at each of these in turn and discuss the restore patterns necessary.
• #30: Here are the options – that will enable: Restoration to the previous Full backup only Restoration to a Point-in-time. EG. If the database goes down at 4.30 you can restore to the 4.30 point it went down. This aides the restoration time by bypassing the log replays between the Full and the Last Differential BU This is for large DBs (called VLDBs) that need a permanent DBA and a complex routine.
• #31: ALL restorations must start with a restoration from a Full BU. Only the portion of the log that is needed to complete incomplete transactions that might be pending are backed up. There is no ‘real’ Log replay of transactions that were already ‘Checkpointed.’
• #32: A FULL backup strategy works like this, for example. You could backup once per week but every night is more common. You can go back to a previous backup but then you would have to re-enter data from that time.
• #33: By just backing up the log, which is an independent file, you can restore the database to a point-in-time. This means you will not have re-enter all the data since the last Full backup. If you restore from the previous backup you can replay/restore the logs to the same Point-in-time because they are completely sequential. The lsn (log sequence number) is stored in the msdb database. Logs MUST be restored sequentially.
• #34: Differential backups MAY be faster than FULL backups because they only backup the differences between the last FULL backup and the database at the time of the backup. Differential backups may speed the restoration by short-circuiting the restoration process by allowing the differential to be restored instead of the accumulated Logs.
• #35: Here is an illustration of a differential BU strategy. A restore still has to start with the FULL BU but you can skip all differential BUs to the LAST differential.
• #36: A complex arrangement involving filegroups may need an experienced DBA to implement since the planning requires a full understanding of the database and the implications of restoring parts of the database and not others.
• #37: Finally
• #38: And Finally stress this -
• #40: 1. The Application sends modification statements to the Database Engine 2. Modifications are written to the log, they can only be Insert, Update or Delete statements. 3. Data pages are read into RAM 4. the Checkpoint process reads the old data, performs the arithmetic, writes the new data – all in RAM.
• #43: A Volume made from a striped set consists of a number of disks in series, each disk merely adding to the size of the set.
• #44: Mirrored drives set consists of 2 drives in parallel. The second drive is a duplicate of the first drive and everything done on the first drive is duolicated on the second drive.
• #45: A Parity set consists of at least 3 drives. Each drive has an uncompressed area storing data and a compressed area that holds a compressed copy of the data stored on another drive. Drives in the set are hot-swoppable but if this happens, performance is terrible during the outage while the swop is taking place. A RAID 5 array is fast on Reads but a little slower on writing because the Parity data (compressed copy) has to be calculated when writing to the Parity copy.
• #46: For the last few years most sales have been RAID 10, a mirrored stripe set. That is - a Stripe-set that is then mirrored (Note: Not mirrored then striped). Although called RAID 10 it is actually RAID 0 + 1. There is a 50 % reduction in capacity because of the dupication but the copy is not compressed and there is no slow-down in write-speed. Hot swop is enabled and even when doing this there is no slow down in performance.