Allow transformEntries style migrations between different Content Models

[Shy]

Context

@benjarwar can jump in here with more context, but we're looking for the ability to allow transformEntries from the migration CLI but between different content models.

For example, Content Model A has Author. We create a new Content Model B. Content Model B needs Author but split into first name, last name.

[benjarwar]

@Shy that's pretty much the idea. Even directly porting fields from one entry type to a new entry type. In our current use case, we're looking to change the entry ID/name (and make some adjustments). Being able to port the content simultaneously would be a big time saver.

[TimBeyer]

Is deriveLinkedEntries what you are looking for, or do you want the same functionality but without the linking part?

[benjarwar]

@TimBeyer deriveLinkedEntries seems more specific to creating new entries out of existing field data. My use case is a little different. I actually need to port/convert an associated reference to a new entry type.

Given an entry type of Parent, which contains an associatedChild field which can be a reference to Child1 entries. I've now created a new entry type, Child2, which should be used in place of Child1.

I would like to now:

• Iterate over all Parent entries
• If it contains a Child1 reference in associatedChild, create a new Child2 entry, copying over all data from the Child1 entry (some fields will also be transformed in this step)
• Delete the original Child1 entry
• Later, deprecate Child1, removing the entry type entirely from the system

[benjarwar]

Researching further, I'm starting to think the solution lies in using the Content Management API rather than trying to fold this into my migration scripts...

[aKzenT]

While it's possible to do this all using the content management API, I would also appreciate a version of this directly in the migration CLI.

We often have the same use cases that @benjarwar mentions. We deprecate one content type in favour of another one or splitting up a content type into several ones.

Another use case would be renaming of content types. It's a real pain that it's impossible to rename contentTypes (sys.id not display name). We have a lot of content types and sometimes the name doesn't match the use cases or our naming conventions anymore. With migrations we could at least create a new content type with the same fields and migrate all entries over.

Thanks!

[Khaledgarbaya]

Hey Folks,
@aKzenT @benjarwar @connor-baer

I would like to understand more the Feature so I can help. Can you share a snippet of code on how you imagine the functionality or maybe a drawing/graph?

Best,
Khaled

[aKzenT]

Hey,

thanks for the response. I would image something like this:

  migration.migrateEntries({
    contentType: 'sourceType',
    targetContentType: 'targetType',
    from: ['a', 'b', 'c' ],
    to: ['d','e','f'],
    shouldPublish: false,
    updateReferences: true, // Replace links to source item to the new target item
    removeSourceEntries: true, // Remove the source item after migration
    shouldMigrate: async(inputFields) =>
    {
        // Possibility to only migrate certain entries based on entry-content
        return true;
    }
    migrateEntryForLocale: async (inputFields, locale) => {
      if (locale !== 'de-DE') {
        // Skip locale
        return;
      }
      // Do transformations, return target entry
      return {
        d,
        e,
        f
      };
    }
  });

Maybe this could even be merged into the existing transformEntries by adding additional properties.

[Khaledgarbaya]

Thank you, I will add this to our internal ticketing system.
Unfortunately, I can't give an exact ETA on this, meanwhile, PRs are welcome.

Best,
Khaled

[GordonApplepie]

Hello,
just here to ask for the current state of this extension. We are currently also investigating the implementation of this feature. Since it would make sense to not do it twice it would be nice to know how far this issue is already. @Khaledgarbaya (the mark WIP has been added recently)

We have multiple changes waiting for this extension since we want to clean up our Contentful spaces and seperate content types out of another content type and migrate all references and trying to avoid doing it manually.

Best regards,
Gordon

[Khaledgarbaya]

Hey @GordonApplepie,
Unfortunately, there is not much progress on this right we had to switch context a bit recently

[GordonApplepie]

UPDATE:
never mind. i think i got it. just calling through usage-as-lib

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "launch program",
            "program": "${workspaceFolder}\\examples\\usage-as-lib"
        }
    ]
}

@Khaledgarbaya since we are not that familiar with debugging node apps would it be possible for you to tell us how to be able to debug the contentful-migration library?

we are able to execute the tool, i am also able to get into debug mode via visual studio code but i am not able to fill in the args (like --space-id) to actually make a call to the different scripts.

example vs code launch settings

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "launch program",
            "program": "${workspaceFolder}\\built\\bin\\usage-params",
            "args": [
                "--space-id hello"
            ],

        }
    ]
}

ends up in
C:\Tools\NodeJS\node.exe --inspect-brk=16926 built\bin\usage-params.js '--space-id hello'

if i open it directly from the terminal, the debugger of vs code does not get attached

[Khaledgarbaya]

Hi @GordonApplepie,

I would recommend using a tool like ndb.
You can add debugger at any line you want and prefix your npm script or command by npm for example if you have npm start you can add ndb npm start and an instance of chrome will open up with the dev tool on and you can step through your code.

[aKzenT]

Hi @Khaledgarbaya,

I've worked with @GordonApplepie on a PR for this feature. We will probably do some more work on it next week, but it would be nice to get some feedback from your side already.

Thanks!

[Khaledgarbaya]

hey @aKzenT 😱 I just saw you huge PR, Thank you a lot for that.
I will take a look definitely on Monday

[aKzenT]

@Khaledgarbaya did you have time to take a look already?

[TimBeyer]

@benjarwar does #143 already solve your use case, or should we keep this issue open?

[benjarwar]

@TimBeyer I'm on a holiday break, but I'll check out #143 soon; looks promising!