Add the migrate runner commands #3402

claudiu-cristea · 2018-02-20T15:49:57Z

Resolves #2926

Should be fixed before merging:

~~https://www.drupal.org/project/migrate_run/issues/2914429~~ ✔️
~~Migrate import runs dependent migrations multiple times.~~ ✔️
~~Add documentation on how to write and run migrations.~~ ✔️
Add a doc section specifically telling folks how to get started with Drupal => Drupal upgrades. How to make use of the migrations that are in core.
~~Move~~ Copy example modules from migrate_plus into https://www.drupal.org/project/examples. Examples provide in Migrate Plus are coupled to the module so, probably we cannot move them, we should create some adapted versions in https://www.drupal.org/project/examples.
This plan was drafted with Lucas Hedding (@heddn) and Moshe Weitzman (@weitzman). Prepare migrate_tools, migrate_plus to adapt the new Drush core feature. We need to coordinate new releases of Migrate Plus & Migrate Tools and also add the composer constraints in Drush:
- Drush 9 commands will be removed from migrate_tools because they are provided now in core. When used with Drush 8, the commands are still provided in the *.drush.inc file.
- Module migrate_tools will alter the Drush core commands migrate:status, migrate:import and migrate:rollback in order to support migration grouping, which is a feature of Migrate Plus. A new branch (8.x-5.x) will hold these changes. Drupal.org issue #2952465.
Update https://docs.google.com/document/d/1ROgc6OIepmcIPFFCb63tDiKz1aK1pXW3seyYN7rgfLs/edit#

Other improvements can be added after merging.

Related issues

Drupal core: Dispatch the PREPARE_ROW event [NEEDS REVIEW]
Migrate Tools: Drush core to add migrate runner commands [NEEDS REVIEW]
Examples for Developers: Add Migrate example [NEEDS WORK]

claudiu-cristea · 2018-02-20T15:58:12Z

Just note here. I needed a namespace for some classes. I have no idea if my pick was correct: Drush\Drupal\Migrate.

Moshe: LGTM

weitzman · 2018-02-24T15:53:21Z

One thing I did not consider earlier is documentation. Folks need to learn how to write and run migrations. Where should we point them to? I'm a little concerned they will use this issue queue for general migrate support.

claudiu-cristea · 2018-03-04T12:11:37Z

I moved the commands from Drush\Commands\core under Drush\Drupal\Commands\core. @weitzman, could you, please, check the @todo from MigrateRunnerCommands::getMigrationPluginManager()?

claudiu-cristea · 2018-03-04T17:49:10Z

This is ready for review.

claudiu-cristea · 2018-03-07T13:13:09Z

This is blocked now in #3447 because we need that for the migrate_plus patch.

weitzman · 2018-03-08T03:33:42Z

I added a todo about docs for Drupal => Drupal upgrade migrations.

claudiu-cristea · 2018-03-11T10:54:39Z

Reworked the issue summary and added a first try for Migrate Plus in https://www.drupal.org/project/migrate_plus/issues/2951968

heddn · 2018-03-12T14:54:21Z

Add a doc section specifically telling folks how to get started with Drupal => Drupal upgrades. How to make use of the migrations that are in core.

I'm really happy with what I'm seeing going on here. I do want to be cautious about the D2D migration docs. Something like the following might be a good start.

Running drush migrate:status and other core drush migrate commands will not work out of the box for core migrations. Unless site owners have created a 'migrate' db entry in their settings(.local).php. This is because all Core the source plugins are all SQL based and require a defined connection. There is a fallback to a db key called 'migrate'. But that's kinda an evil hack and only supports a single DB at a time. If you want to support more than a single DB source, then install migrate_plus and use migrate groups instead.

^ is probably way too technical, but it is a start.

Eli-TW · 2018-03-12T15:01:59Z

Question with respect to this approach - if migrate_plus alters the core commands rather than migrate_tools, that means people wanting to use other parts of migrate_plus (eg the prepareRow event, the JSON/XML source plugins) will be forced to use its commands rather than the Drush ones, whereas before it would have been possible to use migrate run with those source plugins. Is that correct? And is that a limitation that we have decided to accept?

heddn · 2018-03-12T15:05:00Z

The prepare row event and the JSON/XML and CSV sources will not be effected by Drush core being the runner.

claudiu-cristea · 2018-03-12T15:06:22Z

@Eli-TW this is a good question. The prepare row event is now provided also by this PR, so that will work also without migrate_plus. But the module also provides some plugins. For this reason, I think, it would be correct to consider moving the command alters in migrate_tools (?)

heddn · 2018-03-12T15:14:29Z

@claudiu-cristea do we want to remove the prepare row event from drush core? Or bring it in separately? That would make this focused on the drush commands only then. And we can discuss the event in a follow-up.

claudiu-cristea · 2018-03-12T15:18:11Z

@heddn That event is critical to migrate runner, so we cannot remove it from Drush core. We can, probably, rename it to avoid any confusion (DRUSH_PREPARE_ROW) or, better, move it to Drupal core.

heddn · 2018-03-12T15:22:55Z

@claudiu-cristea idealy, moving to core would ideally happen. Not sure if it will, but I'd like it. That depends on commiter discretion.

I did a quick scan of the code here in this PR, what is using it inside of Drush? Why is MigratePrepareRowEvent critical?

claudiu-cristea · 2018-03-12T15:25:06Z

@heddn to support the runner --idlist option. See https://github.com/drush-ops/drush/pull/3402/files#diff-d3fada9bc3f3bdd833a2400a4e4c57e3R400

claudiu-cristea · 2018-03-12T15:30:03Z

There's a Drupal core issue to add the event to the migrate core module. However, the issue scope seems broader. Probably adding only the event dispatcher could be split into a separate issue https://www.drupal.org/project/drupal/issues/2488836

heddn · 2018-03-12T15:32:52Z

Hmm, I wonder if we could bring the support for idlist in an implementation of the core prepare_row hook for now? With a TODO to a new issue to add it more better. Because ideally we'd have the event in core.

claudiu-cristea · 2018-03-12T15:40:41Z

I split the event thing in https://www.drupal.org/project/drupal/issues/2952291

claudiu-cristea · 2018-03-12T15:43:26Z

~~@heddn using the hook could be also an idea. So we avoid all the event headache.~~ EDIT: this is not gonna work. See below why

heddn · 2018-03-12T15:47:17Z

@claudiu-cristea I've re-booted the core issue. But it more than likely not going to land in 8.5, so let's proceed with a hook for the mean time?

claudiu-cristea · 2018-03-12T17:42:54Z

As we've discussed in the chat and because implementing a hook doesn't work (because the migrate executable is not accessible from the hook implementation), we've decided to keep the Drush core event, even if, eventually, will be redundant with the migrate_plus prepare row event. Both will become obsolete after the event will land in Drupal core, in https://www.drupal.org/project/drupal/issues/2952291. In order to prevent any confusion, we renamed the event from PREPARE_ROW to DRUSH_MIGRATE_PREPARE_ROW.

weitzman · 2018-05-16T12:23:49Z

I've decided not to add to confusion by merging this. migrate_run is a great project and will remain separate from Drush core for now.

weitzman · 2020-05-26T16:55:59Z

Just want top say that I did a large migration with Migrate Run recently and worked really well. I am again happy to merge this. The source repo was deleted but the code here is recoverable via https://patch-diff.githubusercontent.com/raw/drush-ops/drush/pull/3402.patch

heddn · 2020-05-26T17:26:51Z

There's been several new features added to migrate_tool's drush runner recently that might be considered useful. I'd be interested if using it as the basis might be another option.

heddn · 2020-05-26T17:44:02Z

There's also pretty complete testing of the commands at this point too. I'd be more then happy to move those things over into drush core.

weitzman · 2020-05-26T18:01:38Z

I'm interested in new runner features, for sure. And tests. What I'm not so interested in is migrations as config entities. Is that a built-in assumption for drush_tools?

heddn · 2020-05-26T18:04:00Z

@weitzman I think it could be optional. If we wrote it so that the list of migrations be decorated or something. That way migrate_tools could provide config entities without having to support all the runner capabilities.

claudiu-cristea · 2020-05-26T18:26:21Z

The migrate_run is only using Drupal core. Nothing more. For this reason, I think, it better suits Drush.

heddn · 2020-05-26T18:33:29Z

In the area where we've agreed to differ is the fact that migrate_tools also finds migrate_plus config entities. But tools has added several new features in the past few months, namely support for syncing, progress bar, id-list, continue-on-failure. These are features it would be nice to maintain. If we have to drop config entities, I'd be inclined to do that and then figure out a way to re-add that support via some drush alter hook or something.

claudiu-cristea · 2020-05-26T18:36:02Z

Then migrate_tools should implement command hooks that are adding migrations from configs to the discovery.

heddn · 2020-05-26T18:44:02Z

I'm glad we can all agree :)

* Initial commit with code from PR #3402. * Apply https://www.drupal.org/project/migrate_run/issues/3184771. * Fix deprecated assertions. * Remove faulty debug statement. * Try a fix for PHP8 * Try to fix on Win OS. * Avoid passing user data by reference. * Restore composer.lock * New Windows fix. * Resolve the query. * Regression test for https://www.drupal.org/project/migrate_tools/issues/2919108. * Typo * Fix PHPUnit deprecation. * Debug Windows tests. * Only check migration output. * Relax the regexp. * Win debug. * Fix Windows tests. * Regression test for https://www.drupal.org/project/migrate_tools/issues/3015386 * Fix source od random failures. * Coding standards. * Apply https://www.drupal.org/project/migrate_tools/issues/3011996 * Fix strict typing in MigrateRunnerCommands * Coding standards. * Test user entered invalid migrations * Coding standards. * Test failures on rollback * Revisit command arguments & options, camelize variables. * Progress bar. See https://www.drupal.org/project/migrate_tools/issues/3075131 * Make the progress bar output predictable * Fix Win test. * Coding standards. * Partially apply https://www.drupal.org/project/migrate_tools/issues/2798363: - Not applied for migrate:message as it don't work in Migrate Tooles either. No tests. - Not implemented --idlist-delimiter: (1) the naming is miselading, it refers to columns of the same ID rather than intra-ID delimiter and (2) I see no usecase (why ':' is not enough?) * Improve documentation. * Conflicts with drupal/migrate_tools and drupal/migrate_run * Implement https://www.drupal.org/project/migrate_tools/issues/2809433 * What is the origin of this strange @validate-module-enabled tag? * Don't add comma after last param. * Ensure a test that works regardless of Symfony version. * Use the <info> tag. * Move migrate_runner.inc. * Use ::success() instead of ::log() * Exception instead of log error * Use StringUtils::csvToArray() * Simple progress bar on Windows. * Apply https://www.drupal.org/project/migrate_tools/issues/2960221 * Another Windows fix tentative. * Test fix follow-up * Minor docs fix. * Re-fix reverted fix ;) * Reverted by mistake * Remove @drupal-dependencies * Use ->success() instead of ->log() * Remove leftover

This was referenced Mar 4, 2018

Add migrate_run instead of migrate_tools #2926

Closed

Move the setup of testing modules in a trait #3443

Merged

claudiu-cristea mentioned this pull request Mar 7, 2018

Allow registering of command info alterer services #3447

Merged

claudiu-cristea added 5 commits March 11, 2018 11:27

Add migrate runner commands. Provide unish tests.

6d01836

Fix coding standards.

465a648

Migrations with failures should return a non-zero code.

eed58f4

Run dependent migrations only once.

87cf4c8

Add documentation.

3792a3e

claudiu-cristea added 2 commits March 12, 2018 19:32

Dispatch the MigrateEvents::DRUSH_MIGRATE_PREPARE_ROW event.

032ad04

Test if MigrateEvents::DRUSH_MIGRATE_PREPARE_ROW is dispatched.

8013677

Cast the message to string.

d2d7fc3

claudiu-cristea added 2 commits March 14, 2018 15:21

Merge branch 'master' into migrate-runner

ed99dc8

Merge branch 'master' into migrate-runner

9305967

weitzman closed this May 16, 2018

claudiu-cristea added a commit to claudiu-cristea/drush that referenced this pull request Dec 15, 2020

Initial commot with code from PR drush-ops#3402.

3211e0d

claudiu-cristea added a commit to claudiu-cristea/drush that referenced this pull request Dec 15, 2020

Initial commit with code from PR drush-ops#3402.

68e1ce1

claudiu-cristea mentioned this pull request Dec 15, 2020

Move Migrate Run commands in Drush core #4606

Merged

claudiu-cristea added a commit to claudiu-cristea/drush that referenced this pull request Dec 18, 2020

Initial commit with code from PR drush-ops#3402.

cafee39

claudiu-cristea added a commit to claudiu-cristea/drush that referenced this pull request Dec 31, 2020

Initial commit with code from PR drush-ops#3402.

471e247

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the migrate runner commands #3402

Add the migrate runner commands #3402

claudiu-cristea commented Feb 20, 2018 •

edited

Loading

claudiu-cristea commented Feb 20, 2018 •

edited by weitzman

Loading

weitzman commented Feb 24, 2018 •

edited

Loading

claudiu-cristea commented Mar 4, 2018 •

edited

Loading

claudiu-cristea commented Mar 4, 2018

claudiu-cristea commented Mar 7, 2018

weitzman commented Mar 8, 2018

claudiu-cristea commented Mar 11, 2018

heddn commented Mar 12, 2018 •

edited

Loading

Eli-TW commented Mar 12, 2018

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018 •

edited

Loading

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018 •

edited

Loading

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018 •

edited

Loading

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018 •

edited

Loading

weitzman commented May 16, 2018

weitzman commented May 26, 2020

heddn commented May 26, 2020

heddn commented May 26, 2020

weitzman commented May 26, 2020

heddn commented May 26, 2020

claudiu-cristea commented May 26, 2020

heddn commented May 26, 2020

claudiu-cristea commented May 26, 2020

heddn commented May 26, 2020

Add the migrate runner commands #3402

Add the migrate runner commands #3402

Conversation

claudiu-cristea commented Feb 20, 2018 • edited Loading

Should be fixed before merging:

Related issues

claudiu-cristea commented Feb 20, 2018 • edited by weitzman Loading

weitzman commented Feb 24, 2018 • edited Loading

claudiu-cristea commented Mar 4, 2018 • edited Loading

claudiu-cristea commented Mar 4, 2018

claudiu-cristea commented Mar 7, 2018

weitzman commented Mar 8, 2018

claudiu-cristea commented Mar 11, 2018

heddn commented Mar 12, 2018 • edited Loading

Eli-TW commented Mar 12, 2018

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018 • edited Loading

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018 • edited Loading

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018 • edited Loading

heddn commented Mar 12, 2018

claudiu-cristea commented Mar 12, 2018 • edited Loading

weitzman commented May 16, 2018

weitzman commented May 26, 2020

heddn commented May 26, 2020

heddn commented May 26, 2020

weitzman commented May 26, 2020

heddn commented May 26, 2020

claudiu-cristea commented May 26, 2020

heddn commented May 26, 2020

claudiu-cristea commented May 26, 2020

heddn commented May 26, 2020

claudiu-cristea commented Feb 20, 2018 •

edited

Loading

claudiu-cristea commented Feb 20, 2018 •

edited by weitzman

Loading

weitzman commented Feb 24, 2018 •

edited

Loading

claudiu-cristea commented Mar 4, 2018 •

edited

Loading

heddn commented Mar 12, 2018 •

edited

Loading

claudiu-cristea commented Mar 12, 2018 •

edited

Loading

claudiu-cristea commented Mar 12, 2018 •

edited

Loading

claudiu-cristea commented Mar 12, 2018 •

edited

Loading

claudiu-cristea commented Mar 12, 2018 •

edited

Loading