![• Specify the query to use for the data
extraction. For example:
• SELECT [people].[personal.age] AS
age, [people].[personal.gender] AS
gender, FROM [people] JOIN
[vehicles] ON [people].[_id] =
[vehicles].[people_id]
• Close the REST Source control and
connect it to the ADO.NET
Destination.](https://image.slidesharecdn.com/howtomovedata-220728072010-c26416ff/85/how-to-move-data-from-on-premise-to-ssis-in-google-cloud-platform-azure-snowflake-22-320.jpg)
how to move data from on premise to ssis in google cloud platform ,azure, snowflake
- 1. HOW TO MOVE DATA PRESENTED BY : KEERTHANA B
- 2. TABLE OF CONTENT • Move data from SQL Server to GoogleCloud • Move data from sql server to ADF(Azure Data factory) • Move data from sql server to AWS • Move data from SQL Server to PostgreSQL • Moving data from PostgreSQL to MySQL • Move data from MYSQL to SQL SERVER • How to move data from REST API – Using SSIS – SQL Server ( On Prmise ) • How to move data from Snowflake DB – Using SSIS – SQL Server ( On Prmise )
- 3. MOVE DATA FROM SQL SERVER TO GOOGLECLOUD • In the Google Cloud console, on the project selector page, select or create a Google Cloud project. • Note: If you don’t plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project. • GO TO THE PROJECT SELECTOR PAGE • Make sure that billing is enabled for your Cloud project. • Enable the Cloud Storage and Cloud SQL APIs.
- 4. MOVE SQL SERVER WORKLOADS TO GOOGLE CLOUD • ENABLE THE APIS • Install and initialize the Google Cloud CLI on the server that’s running SQL Server • In the Google Cloud console, go to Cloud Shell. • GO TO Cloud Shell • At the bottom of the Google Cloud console, a Cloud Shell session opens and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed, and with values already set for your current project. It can take a few seconds for the session to initialize.
- 5. STEPS TO MOVE SQL SERVER DATA TO GOOGLE CLOUD • Creating the Cloud SQL instance and Cloud Storage bucket • Backing up your database In this section, you connect to your Windows server running SQL Server , create a backup of your database, and upload the backup database to Cloud Storage. • Importing the backup file to Cloud SQL • Validating the data import
- 6. IMPORT GOOGLE CLOUD STORAGE DATA INTO SQL SERVER USING SSIS • Add the Components To get started, add a new Google Cloud Storage source and SQL Server ADO.NET destination to a new data flow task.
- 7. • Create a New Connection Manager • Follow the steps below to save Google Cloud Storage connection properties in a connection manager • In the Connection Manager window, right-click and then click New Connection. The Add SSIS Connection Manager dialog is displayed. • In the Connection Manager type menu, select GoogleCloudStorage. The Cdata Cloud Storage Connection Manager is displayed. • Configure connection properties. • Authenticate with a User Account
- 8. • Configure the Google Cloud Storage Source • Follow the steps below to specify the query to be used extract Google Cloud Storage data. • Double-click the Google Cloud Storage source to open the source component editor. • In the Connection Manager menu, select the connection manager previously created.The Connection Manager to be used for the source component. (Salesforce is • Specify the query to use for the data extraction. For example:SELECT Name, OwnerId FROM Buckets WHERE Name = ‘TestBucket’ • Close the Google Cloud Storage Source control and connect it to the ADO.NET Destination
- 9. • Configure the SQL Server Destination • Follow the steps below to specify the SQL server to load the Google Cloud Storage data into • Open the ADO.NET Destination and add a New Connection. Enter your server and database information here. • In the Data access mode menu, select “table or • In the Table Or View menu, select the table or view populate. • Configure any properties you wish to on the Mappings screen.
- 10. • Run the Project • You can now run the project. After the SSIS Task has finished executing, your database will be populated with Google Cloud Storage data.
- 11. MOVE DATA TO AZURE • In SSMS, in Object Explorer, select SQL Server Agent, select Jobs, then right-click and select Migrate SSIS Jobs to ADF. Screenshot shows SQL Server Management Studio Object Explorer, where you can select Jobs, then Migrate SSIS Jobs to ADF • Sign In Azure, select Azure Subscription, Data Factory, and Integration Runtime. Azure Storage is optional, which is used in the package location mapping step if SSIS jobs to be migrated have SSIS File System packages. Menu • Map the paths of SSIS packages and configuration files in SSIS jobs to destination paths where migrated pipelines can access. In this mapping step, you can:
- 12. • Select a source folder, then Add Mapping. • Update source folder path. Valid paths are folder paths or parent folder paths of packages. • Update destination folder path. Default is relative path to the default Storage account, which is selected in step 1. • Delete a selected mapping via Delete Mapping.
- 13. • Select applicable jobs to migrate, and configure settings of corresponding Executed SSIS Package activity • Default Setting, applies to all selected steps by default. • Apply Default Setting: default is selected. Unselect to configure setting for selected step only. • For more information of other properties, see Settings tab for the Execute SSIS Package activity when package location is File System (Package).
- 14. • Generate and deploy ARM template. • Select or input the output path for the ARM templates of the migrated ADF pipelines. Folder will be created automatically if not exists. • Select the option of Deploy ARM templates to your data factory: • Default is unselected. You can deploy generated ARM templates later manually. • Select to deploy generated ARM templates to data factory directly.
- 15. • Migrate, then check results. Next steps Run and monitor pipeline
- 16. MOVE DATA FROM SQL SERVER TO AWS • To migrate to an AWS from MySQL DB instance using AWS DMS: • Create a replication instance • Create target and source endpoints • Refresh the source endpoint schemas • Create a migration task • Monitor your migration task
- 17. MOVING DATA FROM POSTGRESQL TO MYSQL • In “Choose a Data Source” dialog, choose “PostgreSQL”; • In “Choose a Destination” dialog, choose “MySQL”; • In “Select source Tables(s) & View(s)” dialog • In “Execution” Dialog; • Finished
- 18. MOVE DATA FROM SQL SERVER TO POSTGRESQL • use the built-in SSIS Import/export feature. You can follow these steps: • Firstly, you need to install the PostgreSQL ODBC Driver for Windows. • Inside Management Studio, Right click on your database: Tasks -> Export Data • Choose SQL Server Native Client as the data source. • Choose .Net Framework Data Provider for ODBC as the destination driver. • Set the Connection String to your database in the following form: • Driver={PostgreSQL ODBC Driver(UNICODE)};Server=;Port=;Database=;UID=;PWD= • In the next page, you just need to select which tables you want to export. SQL Server will generate a default mapping and you are free to edit it. Probably
- 19. MOVE DATA FROM MYSQL TO SQL SERVER • Open SSMA for MySQL. • On the File menu, select New Project. • Enter the project name and a location to save your project and the migration target • In the Connect to MySQL dialog box, enter connection details, and then connect to your MySQL server. • Select the MySQL databases you want to migrate.
- 20. HOW TO MOVE DATA FROM REST API – USING SSIS – SQL SERVER ( ON PRMISE ) • demonstrates how to use the CData SSIS Tasks for REST inside of a SQL Server SSIS workflow to transfer REST data into a Microsoft SQL Server database. • Add the Components • To get started, add a new REST source and SQL Server ADO.NET destination to a new data flow task.
- 21. • Create a New Connection Follow the steps below to save REST connection properties in a connection manager. • In the Connection Manager window, right- click and then click New Connection. The Add SSIS Connection Manager dialog is displayed. • In the Connection Manager type menu, select REST. The Cdata REST Connection Manager is displayed.Manager • Configure the REST Source Follow the steps below to specify the query to be used to extract REST data. 1.Double-click the REST source to open the source component editor. 2.In the Connection Manager menu, select the connection manager previously created
- 22. • Specify the query to use for the data extraction. For example: • SELECT [people].[personal.age] AS age, [people].[personal.gender] AS gender, FROM [people] JOIN [vehicles] ON [people].[_id] = [vehicles].[people_id] • Close the REST Source control and connect it to the ADO.NET Destination.
- 23. Configure the SQL Server Destination • Follow the steps below to specify the SQL server table to load the REST data into. • Open the ADO.NET Destination and add a New Connection. Enter your server and database information here. • In the Data access mode menu, select “table or view”. • In the Table Or View menu, select the table or view to populate. • Configure any properties you wish to on the Mappings screen Run the Project You can now run the project. After the SSIS Task has finished executing, your database will be populated with REST data.
- 24. HOW TO MOVE DATA FROM SNOWFLAKE DB – USING SSIS – SQL SERVER ( ON PRMISE ) • Add the Components • Create a New Connection • In the Connection Manager window, right- click and then click New Connection. The Add SSIS Connection Manager dialog is displayed. • In the Connection Manager type menu, select Snowflake. The Cdata Snowflake Connection Manager is displayed. • Configure connection properties. • To connect to Snowflake: Manager • Configure the Snowflake Source • Double-click the Snowflake source to open the source component editor. • In the Connection Manager menu, select the connection manager previously created. • Specify the query to use for the data extraction. For example:SELECT Id, ProductName FROM Products
- 25. • Configure the SQL Server Destination • Open the ADO.NET Destination and add a New Connection. Enter your server and database information here. • In the Data access mode menu, select “table or view”. • In the Table Or View menu, select the table or view to populate. • Configure any properties you wish to on the Mappings screen
- 26. • Run the Project • You can now run the project. After the SSIS Task has finished executing, your database will be populated with Snowflake data.