How to prepare data before a data migration
Download as PPTX, PDF3 likes6,182 views
It is crucial to thoroughly prepare data and systems before a data migration by performing a landscape analysis of the source and target systems, validating data quality through profiling and cleansing, and analyzing the impact of data changes. The guide outlines steps for landscape analysis, data assurance including profiling, quality definition and cleansing, verification, and impact analysis to maximize the chances of a successful migration project.
How to prepare data before a data migration
- 1. How to prepare your data for a data migration www.etlsolutions.com
- 2. Introduction • Data migration is a complex undertaking, and the processes and software used are continually evolving. • Thorough preparation of data and systems before a migration takes place helps to reduce the risks involved. • The aim of this guide is to show the ways in which data can be systematically prepared before a migration to maximise the project’s chance of success. • Don’t hesitate to get in touch with us at info@etlsolutions.com if you have any questions. www.etlsolutions.com Download our data migration planning eGuide for free at: http://www.etlsolutions.co m/free-eguide-preparinga-data-migration-plan/
- 3. Landscape analysis • It is crucial to thoroughly prepare data and systems before a migration takes place. • Landscape analysis provides an overview of the source and target systems, enabling the project team to understand how each system works and how the data within each system is structured. • These areas should be reviewed systematically to ensure that potential errors are identified in advance of the migration. • Ideally, the team should model the links and interactions between the different systems involved, along with the data structures within each system. www.etlsolutions.com
- 4. Data assurance • Data assurance is another important component of thorough preparation. • This procedure validates the data discovered in the landscape analysis and ensures that all data is fit for purpose. • By validating the data, the migration team are then free to focus solely on structural manipulation and movement. • Data assurance has several phases: • Data profiling • Data quality definition • Data cleansing. www.etlsolutions.com
- 5. Data profiling • The aim of the data profiling phase is to ensure that any historical data due to be migrated is suitable for the changes that are taking place in the organisation. • Profiling will identify areas of the data which may not be of sufficient quality. • It should include comprehensive checks of existing model structure, data format and data conformance. • A retirement plan should be used to define the data no longer required: • Any data to be retired should be recorded, along with a description of what replaces it or why it can be removed. • The data that is no longer needed may have to be archived for tax purposes or to meet the requirements of an industry’s governing bodies. www.etlsolutions.com
- 6. Data quality definition • Data quality definitions state the quality that must be attained by elements, attributes and relationships in the source system. • The definitions or rules should be used during profiling to identify whether or not the data is of the correct quality and format. • All data quality rules should be listed at element level, such as data table or flat file. • All data quality issues and queries should be tracked and stored. www.etlsolutions.com
- 7. Data cleansing • The first stage in data cleansing is to define which cleansing rules will be carried out manually and which will be automated. • Splitting the rules into two enables the organisation’s domain experts to concentrate on the manual process, while the migration experts design and develop the automated cleansing. • Typically, the manual cleansing will be carried out before the migration, while the automated cleansing may be carried out before the migration or as part of the migration’s initial phase. www.etlsolutions.com
- 8. Data verification • Data verification is the part of the data cleansing process that checks that the data is available, accessible, complete and in the correct format. • Our consultants often continue to carry out verification once a migration has begun, ensuring that the information is optimised prior to each stage of the migration. www.etlsolutions.com
- 9. Data impact analysis • We find that data impact analysis is a crucial part of data cleansing. • Because cleansing data adds or alters values, data impact analysis ensures that these changes do not have a knock-on effect on other elements within the source and target systems. • It also checks the impact of data cleansing on other systems which currently use the data, and on systems which may use the data once the migration is complete. www.etlsolutions.com
- 10. Download your free copy of our data migration planning guide • Download the PDF copy of this guide for easy reading and printing. It’s free, and no email address is required! • Visit us at: http://www.etlsolutions.com/free -eguide-preparing-a-datamigration-plan/ to download your copy. About us At ETL Solutions, we design software to help developers tackle difficult data transformations. We deliver ready-to-use products and services based on Transformation Manager, a robust integration toolkit. Images from Freedigitalphotos.net
- 11. Contact information Karl Glenn, Business Development Director kg@etlsolutions.com +44 (0) 1912 894040 www.etlsolutions.com Raising data management standards www.etlsolutions.com www.etlsolutions.com
Editor's Notes
- #4: To keep things simple when I’m talking, we’ll discuss loading data into PPDM, but a lot of this applies to generic data loading – moving data out of PPDM, or not involving PPDM at all.Data transformation is mudane from a business perspective, but very important to get right. The less time and trouble it causes, the more time you can spend doing more interesting things directly benefiting your business.Badly loaded data by definition affects the quality of the data in your MDM store.
- #5: To keep things simple when I’m talking, we’ll discuss loading data into PPDM, but a lot of this applies to generic data loading – moving data out of PPDM, or not involving PPDM at all.Data transformation is mudane from a business perspective, but very important to get right. The less time and trouble it causes, the more time you can spend doing more interesting things directly benefiting your business.Badly loaded data by definition affects the quality of the data in your MDM store.
- #6: To keep things simple when I’m talking, we’ll discuss loading data into PPDM, but a lot of this applies to generic data loading – moving data out of PPDM, or not involving PPDM at all.Data transformation is mudane from a business perspective, but very important to get right. The less time and trouble it causes, the more time you can spend doing more interesting things directly benefiting your business.Badly loaded data by definition affects the quality of the data in your MDM store.
- #7: To keep things simple when I’m talking, we’ll discuss loading data into PPDM, but a lot of this applies to generic data loading – moving data out of PPDM, or not involving PPDM at all.Data transformation is mudane from a business perspective, but very important to get right. The less time and trouble it causes, the more time you can spend doing more interesting things directly benefiting your business.Badly loaded data by definition affects the quality of the data in your MDM store.
- #8: To keep things simple when I’m talking, we’ll discuss loading data into PPDM, but a lot of this applies to generic data loading – moving data out of PPDM, or not involving PPDM at all.Data transformation is mudane from a business perspective, but very important to get right. The less time and trouble it causes, the more time you can spend doing more interesting things directly benefiting your business.Badly loaded data by definition affects the quality of the data in your MDM store.
- #9: To keep things simple when I’m talking, we’ll discuss loading data into PPDM, but a lot of this applies to generic data loading – moving data out of PPDM, or not involving PPDM at all.Data transformation is mudane from a business perspective, but very important to get right. The less time and trouble it causes, the more time you can spend doing more interesting things directly benefiting your business.Badly loaded data by definition affects the quality of the data in your MDM store.
- #10: To keep things simple when I’m talking, we’ll discuss loading data into PPDM, but a lot of this applies to generic data loading – moving data out of PPDM, or not involving PPDM at all.Data transformation is mudane from a business perspective, but very important to get right. The less time and trouble it causes, the more time you can spend doing more interesting things directly benefiting your business.Badly loaded data by definition affects the quality of the data in your MDM store.