Migrate all the things!
1 like253 views
The document discusses the migration process in Drupal, highlighting its importance for improving content workflows and allowing for data import from various sources. It emphasizes the need for clients to engage in content creation to ensure timely launches and introduces various techniques and tools for effective migration, including the use of CSV files and the migrate module. The speaker encourages migrating not just for upgrades but for new content workflows as well, enabling a more efficient system overall.
![27
id: test
label: Test CSV migration
source:
plugin: csv
path: public://test.csv
header_row_count: 1
keys: [Title]
destination:
plugin: entity:node
default_bundle: page
process:
title: Title
body: Body
test.csv migrate_plus.migration.test.yml
Title Body
About Evolving Web…
Careers If you're interested…
Projects We work with…](https://image.slidesharecdn.com/migrateallthethings-170403164616/85/Migrate-all-the-things-27-320.jpg)
![28
id: test
label: Test CSV migration
source:
plugin: csv
path: public://test.csv
header_row_count: 1
keys: [Title]
destination:
plugin: entity:node
default_bundle: page
process:
title: Title
body: Body
test.csv migrate_plus.migration.test.yml
Title Body
About Evolving Web…
Careers If you're interested…
Projects We work with…](https://image.slidesharecdn.com/migrateallthethings-170403164616/85/Migrate-all-the-things-28-320.jpg)
![29
id: test
label: Test CSV migration
source:
plugin: csv
path: public://test.csv
header_row_count: 1
keys: [Title]
destination:
plugin: entity:node
default_bundle: page
process:
title: Title
body: Body
test.csv migrate_plus.migration.test.yml
Title Body
About Evolving Web…
Careers If you're interested…
Projects We work with…](https://image.slidesharecdn.com/migrateallthethings-170403164616/85/Migrate-all-the-things-29-320.jpg)
![30
id: test
label: Test CSV migration
source:
plugin: csv
path: public://test.csv
header_row_count: 1
keys: [Title]
destination:
plugin: entity:node
default_bundle: page
process:
title: Title
body: Body
test.csv migrate_plus.migration.test.yml
Title Body
About Evolving Web…
Careers If you're interested…
Projects We work with…](https://image.slidesharecdn.com/migrateallthethings-170403164616/85/Migrate-all-the-things-30-320.jpg)
![Workflow: Content first, in Drupal
35
REST module to export content as JSON
[
{
"title": "My node",
"path": "/node/1",
"body": "<p>This is a sample
node.</p>n"
},
{
"title": "Another node",
...
},
...
]](https://image.slidesharecdn.com/migrateallthethings-170403164616/85/Migrate-all-the-things-35-320.jpg)
![Workflow: Mirror
• If users are uploading data files, we need to tell migrate about
them.
51
Dynamic source
$path = $form_state->getValue(['source', 0])->getFileUri();
$source = $migration->getSourceConfiguration();
$source['path'] = $path;
$migration->set('source', $source);](https://image.slidesharecdn.com/migrateallthethings-170403164616/85/Migrate-all-the-things-51-320.jpg)
![/**
* @MigrateSource(
* id = "markdown"
* )
*/
class Markdown extends SourcePluginBase {
public function initializeIterator() {
$found = file_scan_directory($this->configuration['path'], '/.md$/');
foreach ($found as $item) {
yield (array) $item;
}
}
public function __toString() {
return $this->configuration['path'];
}
public function fields() {
return ['uri' => $this->t('URI')];
}
public function getIds() {
return ['uri' => ['type' => 'string']];
}
}](https://image.slidesharecdn.com/migrateallthethings-170403164616/85/Migrate-all-the-things-59-320.jpg)
Migrate all the things!
- 1. Migrate all the things! Better Drupal workflows, using Migrate 1 Dave Vasilevsky vasi@evolvingweb.ca twitter.com/djvasi @vasi on drupal.org, github
- 2. About me • Doing Drupal since 2008 • Love open source and the Drupal community • Contributor to Migrate in core (eg: multilingual nodes in D6 → D8 upgrades 2
- 3. Our specialties Multilingual Solr search Testing Custom Drupal applications Responsive themes and of course… Migrate 3 • Drupal development, consulting and training since 2007 • Based in Montreal, clients all over Canada and USA • Very involved in the Drupal community http://evolvingweb.ca
- 5. Once upon a time… 5
- 6. Once upon a time… 6 “We’re going to save some money. You just build the site—we’ll add all the content ourselves.” - The Client
- 7. Once upon a time… 7 A beautiful site, only missing content
- 8. Once upon a time… 8 And then content happens.
- 9. Once upon a time… 9 • Bodies that are only images or tables • Relationships or tags that weren’t identified • Much shorter or longer text than expected • Many more or fewer nodes than expected So many things can go wrong!
- 10. A solution 10 Gather client data in a spreadsheet, import it into Drupal Title Body About Evolving Web is a Montreal-based team of Drupal experts… Careers If you're interested in working at Evolving Web, please fill out the form below… Projects We work with clients in a wide variety of sectors, including health care, tourism…
- 11. A solution • Force client to actually write content quickly, improves likelihood of launch on-time • Client can use tools they know, even without Drupal training • Can check if IA is correct, and adjust before it’s too late • Each developer can import data on their own computer 11 What we gain
- 12. Migrate? • Yes, it’s about migrate. So what’s migrate? 12
- 13. Migrate? • Yes, it’s about migrate. So what’s migrate? • A system that allows importing structured data into Drupal, usually as entities: nodes, users, terms… • Data can come from many different places: DB, CSV, XML… • Very extensible • Included in Drupal 8 core (experimental) • What is migrate not? 13
- 14. Migrate? • There’s already lots of info about what migrations are, how to write a migration: • Michael Anello at DCNJ 2017: http://tiny.cc/ ultimikeMigrate • Evolving Web blog series: http://tiny.cc/ewMigrate • These aren’t the focus of the talk 14
- 15. Migrate? • This talk isn’t just about how to start. More about why and when. • Common impression is that migrate is only for: 15
- 16. Migrate? • This talk isn’t just about how to start. More about why and when. • Common impression is that migrate is only for: • Upgrades from D6/D7 to Drupal 8 16
- 17. Migrate? • This talk isn’t just about how to start. More about why and when. • Common impression is that migrate is only for: • Upgrades from D6/D7 to Drupal 8 • Moving from legacy sites to Drupal 17
- 18. Migrate? • This talk isn’t just about how to start. More about why and when. • Common impression is that migrate is only for: • Upgrades from D6/D7 to Drupal 8 • Moving from legacy sites to Drupal 18
- 19. Migrate? • But there are also many new content workflows that migrate enables • So don’t just migrate things that feel like migrations— migrate all the things 19
- 20. Migrate? • But there are also many new content workflows that migrate enables • So don’t just migrate things that feel like migrations— migrate all many of the things 20
- 22. The parts of migrate 22 Source Destination Process
- 23. The parts of migrate 23 Source Destination Process CSV File First Last Job Dave Vasilevsky Backend Jorge Diaz Frontend Alex Dergachev Marketing
- 24. The parts of migrate 24 Source Destination Process CSV File Users First Last Job Dave Vasilevsky Backend Jorge Diaz Frontend Alex Dergachev Marketing name field_job Dave Vasilevsky Backend Jorge Diaz Frontend Alex Dergachev Marketing
- 25. The parts of migrate 25 Source Destination Process CSV File Users First Last Job Dave Vasilevsky Backend Jorge Diaz Frontend Alex Dergachev Marketing name field_job Dave Vasilevsky Backend Jorge Diaz Frontend Alex Dergachev Marketing First Last Job field_job name
- 26. The parts of migrate • migrate_plus: Allows a migration to be defined with YAML config file • ‘Migration’: Definition of source + process + destination • migrate_tools: Allows running migrations using drush 26 Contrib modules
- 27. 27 id: test label: Test CSV migration source: plugin: csv path: public://test.csv header_row_count: 1 keys: [Title] destination: plugin: entity:node default_bundle: page process: title: Title body: Body test.csv migrate_plus.migration.test.yml Title Body About Evolving Web… Careers If you're interested… Projects We work with…
- 28. 28 id: test label: Test CSV migration source: plugin: csv path: public://test.csv header_row_count: 1 keys: [Title] destination: plugin: entity:node default_bundle: page process: title: Title body: Body test.csv migrate_plus.migration.test.yml Title Body About Evolving Web… Careers If you're interested… Projects We work with…
- 29. 29 id: test label: Test CSV migration source: plugin: csv path: public://test.csv header_row_count: 1 keys: [Title] destination: plugin: entity:node default_bundle: page process: title: Title body: Body test.csv migrate_plus.migration.test.yml Title Body About Evolving Web… Careers If you're interested… Projects We work with…
- 30. 30 id: test label: Test CSV migration source: plugin: csv path: public://test.csv header_row_count: 1 keys: [Title] destination: plugin: entity:node default_bundle: page process: title: Title body: Body test.csv migrate_plus.migration.test.yml Title Body About Evolving Web… Careers If you're interested… Projects We work with…
- 31. A solution • Force client to actually write content quickly, improves likelihood of launch on-time • Client can use tools they know, even without Drupal training • Can check if IA is correct, and adjust before it’s too late • Each developer can import data on their own computer. No database sharing! 31 What we gain
- 32. Workflow: Content First • Only migrate when it’s worth it • Spreadsheets are too simple • Images are hard • Related content is hard 32 ⚠ DANGER ⚠
- 33. Workflow: Content first, in Drupal • Drupal has great content management UI • Rich text editor • Images • References • Don’t need complete theme and behaviour to create content, just content type definitions 33
- 34. Workflow: Content first, in Drupal • Provide client a content staging site • They can build all the content there • Migrate it into Drupal, just like before • Same advantages of “Content first” workflow 34
- 35. Workflow: Content first, in Drupal 35 REST module to export content as JSON [ { "title": "My node", "path": "/node/1", "body": "<p>This is a sample node.</p>n" }, { "title": "Another node", ... }, ... ]
- 36. Workflow: Content first, in Drupal 36 Relationships Company Evolving Web IBM Apple User Company Dave Evolving Web Jorge Evolving Web Employee: Dave nid: 9 Employee: Jorge nid: 10 Company: EW nid: 3 3 3
- 37. Workflow: Drupal for content gathering • Migrate supports more complex mappings: “process plugins” • There’s a list of ones in core: https://www.drupal.org/docs/8/ api/migrate-api/migrate-process 37 Relationships process: field_name: plugin: concat delimiter: “ “ source: - firstname - lastname
- 38. Workflow: Content first, in Drupal 38 Relationships Company Evolving Web IBM Apple User Company Dave Evolving Web Jorge Evolving Web process: field_employer: plugin: migration source: employer migration: companies
- 39. Workflow: Content first, in Drupal • Users can gather complex content, early • Use polished Drupal UI to do so 39 What we gain
- 40. Workflow: Content first, in Drupal • Authentication: Shield + migrate_plus • Translations: Two migrations • No built-in export: Menus, custom blocks 40 Complications
- 41. Workflow: Content first, in Drupal • Site doesn’t look pretty • Once the IA is nailed down—stop! 41 ⚠ DANGER ⚠
- 42. Workflow: Mirror • External tool may be better than Drupal for managing certain content. Eg: • Membership management system • Image management systems • Tools specific to data type, like course catalog • Manage data with external system • Keep Drupal in sync 42
- 48. Workflow: Mirror • Update what’s changed, and only what’s changed • Delete entities that are no longer in source • Trigger without drush • Make it hard to make mistakes 48 Recurring migrations
- 49. Workflow: Mirror • Only update what’s changed: incremental migration 49 source: track_changes: true
- 50. Workflow: Mirror • Trigger on cron, or via UI 50 Triggering migrations without drush $migration_manager = Drupal::service('plugin.manager.config_entity_migration'); $migration = $migration_manager->createInstance('my_migration'); $executable = new MigrateExecutable($migration, new MigrateMessage()); $executable->import();
- 51. Workflow: Mirror • If users are uploading data files, we need to tell migrate about them. 51 Dynamic source $path = $form_state->getValue(['source', 0])->getFileUri(); $source = $migration->getSourceConfiguration(); $source['path'] = $path; $migration->set('source', $source);
- 52. Workflow: Mirror • Users want to know what’s about to happen, check if it’s expected • We can look at the Migrate source object, and check what’s new/updated/ deleted: http://tiny.cc/migrateDryRun 52 Dry run term$ drush scr dry_run.php UPDATED: 2 UPDATED: 4 NEW: 5 DELETED: 3
- 53. Workflow: Mirror • In core, migrate will never delete items that have been removed from source. • But now we know what needs deletion, so we can add a hook to do that too. • BE CAREFUL! Validate your data 53 Delete
- 54. Workflow: Mirror • migrate_devel module will print every row before migrating it • Look at migrate_map_foo and migrate_message_foo DB tables • Use XDebug and step through! 54 Debugging
- 55. Workflow: Mirror • Use the right tool for the right job: Data can live in external tool if it’s better for the job • Don’t need to write any weird integrations (eg: external_entities) 55 What we gain
- 56. Workflow: Mirror • Validate your data! • Data will be overwritten • Never do bidirectional sync. It will break. 56 ⚠ DANGER ⚠
- 57. Workflow: Content in git • Example of an external system: Markdown files in a git repo 57
- 58. Workflow: Content in git • In Drupal 8, almost everything is just a plugin • That goes for Migrate sources and process plugins too • They’re pretty well-designed, don’t be afraid to write one 58 Getting all the files
- 59. /** * @MigrateSource( * id = "markdown" * ) */ class Markdown extends SourcePluginBase { public function initializeIterator() { $found = file_scan_directory($this->configuration['path'], '/.md$/'); foreach ($found as $item) { yield (array) $item; } } public function __toString() { return $this->configuration['path']; } public function fields() { return ['uri' => $this->t('URI')]; } public function getIds() { return ['uri' => ['type' => 'string']]; } }
- 60. Workflow: Content in git • Our client kept a hierarchical YAML file, to turn into a menu • But migrate wants a list of rows, not a tree! • RecursiveIteratorIterator makes this simple: http:// tiny.cc/RecIterIter 60 Source data can be weird
- 61. Workflow: Content in git • Catch problems early: Check the resulting entities, see if they make sense • Are there broken links? Orphan nodes, without a menu item? • Compare generated test site with Drupal: https:// github.com/evolvingweb/sitediff 61 Validation
- 62. Workflow: Content in git • Best tools for the job • Git comes with: branches, merging, global ID, blame… • Can use whichever Markdown editor they like • Can revert to earlier version of site! 62 What we gain
- 63. Workflow: Partial upgrades • A D6/7 to D8 upgrade normally upgrades complete config/ content • This leaves cruft, eg; obsolete fields • Things have changed in ten years! You probably want new content • Building the new site from scratch means rewriting all your content 63
- 64. Workflow: Partial upgrades • Build upgraded site from scratch, using new best-practices • But keep some content. Only what you need! 64
- 65. Workflow: Partial upgrades 65 There’s a shortcut! term$ drush migrate-upgrade --configure-only —legacy-db-url=mysql://localhost/test Exporting d7_dblog_settings as upgrade_d7_dblog_settings Exporting d7_filter_format as upgrade_d7_filter_format Exporting d7_filter_settings as upgrade_d7_filter_settings Exporting d7_global_theme_settings as upgrade_d7_global_theme_settings Exporting d7_image_settings as upgrade_d7_image_settings Exporting d7_image_styles as upgrade_d7_image_styles Exporting d7_language_types as upgrade_d7_language_types Exporting d7_node_settings as upgrade_d7_node_settings ...
- 66. Workflow: Partial upgrades • Export config, delete almost all the migrations. • Only keep what you want to save, eg: • Some content types: upgrade_d7_node_page • Tags: upgrade_d7_taxonomy_term_tags • These contain everything you need to migrate those types, even the hard parts of the migration • Modify as needed. • Eg: If you don’t migrate users, don’t keep the uid in the migrations 66
- 67. Workflow: Partial upgrades • The D6/7 to D8 migrations include every file on the site. Could be many GBs! • It can be worth only keeping the ones referred to by content you’re keeping: • http://tiny.cc/migrateFilterFiles 67 Filtering files
- 68. Workflow: Partial upgrades • URLs that break hurt SEO a lot • Keep the URL alias migration. Migrate redirects as well. • Since you’re throwing out some content, some links will die • Have a plan for adding redirects • Check your site afterwards for broken links • Proactive: Sitediff https://github.com/evolvingweb/sitediff • Reactive: Google Webmaster Tools 68 Broken links
- 69. Workflow: Partial upgrades • Keep the content that’s worth keeping • Keep URLs working • Throw out the junk, build a beautiful new site • Can merge data, as long as there are no shared references 69 What we gain
- 70. What you just saw • Intro to migrate • Workflows • Content first • Content first, in Drupal • Mirror • Content in git • Partial upgrades 70
- 71. What you just saw Migrate can improve your content workflow! 71
- 72. What to do now • Find me on Twitter @djvasi • Learn more about migrate • Try out these content workflows 72
- 73. What to do now • Public: Chicago, Montreal, Toronto, NJ, NYC • Private: Health Canada, McGill University, remote • For new content editors, new developers, teams starting on D8… 73 Evolving Web training
- 74. What to do now • Anything I got wrong? • Any new ideas I should look into? • How should migrate evolve? 74 Ask me questions!