Allow migrations to handle converting a JSON field into actual content types

[gkulasik-livongo]

The contentful-migration library is powerful and allows many types of flexible migrations.

One recent issue I've come across is that there is no easy way to convert a JSON field into actual content type entries. Specifically a JSON array.

Ex. I have a content type of called 'TagSet' with a JSON field called 'tags'. This JSON structure would look something like:

{ tags: [ { "tag": "github", "alternative":"github.com" }, { "tag": "bitbucket", "alternative":"bitbucket.com" } ] }

I'd like to be able to convert this into a content type 'Tag' with fields 'tag' and 'alternative'. So one TagSet could become 1+ Tags.

I think that the deriveLinkedEntries method could be a good place to start to support this. I know that there was some recent work done in #127 that solved an n to n deriveLinkedEntries migration that seems in line.

This is more of a feature request than a bug (unless there is a way to do this with the existing tools - without getting too deep into custom API work). I filed this after suggetion from Stefan on Slack.

[Khaledgarbaya]

Hey @gkulasik-livongo
I think you can do something similar to this

module.exports = (migration) => {
  const myContentType = migration.editContentType('myContentType')
  const refContentType = migration.createContentType('tag')
  // TODO add fields here tag and alternative
  // TODO create a multiRef field here to hold all tags
  migration.transformEntries({
    contentType: 'myContentType',
    from: ['json'],
    to: ['multiRef'],
    transformEntryForLocale: (fromFields, currentLocale) => {
      json = JSON.parse(fromFields.json) // assuming this will return an array
      return {multiRef:json}
    }
  })
}

[gkulasik-livongo]

Thanks for the help!

Hm... interesting. Will this work in the Contentful CLI or just the contentful-migration library? We are trying to only use the Contentful CLI if possible in our Contentful management.

Would this work if the 'Tag' content type has a uniqueness validation on the 'tag' field? Would the references automatically resolve like in deriveLinkedEntries?

Also, to build on this. Let's say this works and TagSet now has a list of Tags. If TagSet was part of another content type called Article, meaning TagSet is a field on the Article content type, is it possible to then move the new TagSet.tags to Article.tags (assuming it has an equivalent field already created to consume the tags form TagSet).

Thanks again - the support from the Contentful team is appreciated!

[gkulasik-livongo]

@Khaledgarbaya I tried using the snippet you provided. I am running into an issue (I am using the Contentful CLI).
The log snipped looks like this:

...{
        "name": "unknown",
        "path": [
          "fields",
          "tags",
          "en-US",
          0,
          "tag"
        ],
        "value": "NEAT",
        "details": "The property \"tag\" is not allowed here."
      }...

We use 'en' not 'en-US' (this is older inherited Contentful content). So I not sure where en-US is coming from.

The content type is created and the field for storing the array is present (JSON from UI):

...{
      "id": "tags",
      "name": "Tags",
      "type": "Array",
      "localized": true,
      "required": false,
      "validations": [],
      "disabled": false,
      "omitted": false,
      "items": {
        "type": "Link",
        "validations": [
          {
            "linkContentType": [
              "tag"
            ]
          }
        ],
        "linkType": "Entry"
      }
    }...

I have tried turning localization and required on and off as well.

[Khaledgarbaya]

Hey @gkulasik-livongo,
did you try if(currentLocale !== 'en') return ?

[gkulasik-livongo]

Yup, here is what I have:

  migration.transformEntries({
      contentType: 'tagSet',
      from: ['jsonTags'],
      to: ['tags'],
      transformEntryForLocale: (fromFields, locale) => {
        if(fromFields.jsonTags[locale] === undefined || locale !== 'en'){
          return;
        }
        //json = JSON.parse(fromFields.jsonTags[locale]["tags"]) // This is removed because I get a JSON parse error - the text is already JSON, it seems to automatically be parsed.
        return {tags: fromFields.jsonTags[locale]["tags"]};
      }
    })

[gkulasik-livongo]

Quick update here for others that may run into this problem. Depending on your use case, if you have a limited number of entries in your JSON you could take the following steps.

1. Create a text list field (Array of Symbol)
2. Use transformEntries to parse the JSON, extract a JSON key that is unique to each JSON object, and populate the new text list field (Add demo GIF #1).
3. Then if your maximum text list length is under 48, you should be able to loop and create 48 temporary reference fields for the new content type you are implementing.
4. Loop over each of the new fields and do the following: 1) deriveLinkedEntries from your text list and JSON fields to the new content type into one of the temporary fields. 2) Remove the migrated JSON key from the text list. This helps keep track of where in the migration the process is.
5. Now that all the JSON is extracted into actual content objects and referenced in a 1-1 entry (you could have up to 48 fields to store these entries) you can proceed to merge the fields into one many references field.
6. Use transformEntries to convert all the individual fields into a single field (add each content from each field to an array, then assign the array to the new multiple references field).
7. Clean up and remove the temporary fields.

In terms of where this issue stands with Contentful I leave that up to @Khaledgarbaya. There may be an issue with the CLI but it was able to be somewhat worked around with multiple steps.

@Khaledgarbaya any ideas on the question in #142 (comment)?

[Khaledgarbaya]

Hi @gkulasik-livongo,
Thanks again for the explanation. Do you think you can add an example showcasing this in the examples folder, it would be a great addition.

Hm... interesting. Will this work in the Contentful CLI or just the contentful-migration library? We are trying to only use the Contentful CLI if possible in our Contentful management.

Yes, this should work with the cli although sometimes we can add more feature to the migration library itself but eventually will update the dependencies on the cli to reflect that.

Would this work if the 'Tag' content type has a uniqueness validation on the 'tag' field? Would the references automatically resolve like in deriveLinkedEntries?

Ok now I understand the issue more we should've used deriveLinkedEntries here instead because we need to tell the migration tool to create an entry for each tag. My apologies for the confusion 😞
Yes to solve this you need to do it over 2 steps first json to a single ref field using transformEntries and then a singleREf to multiRef field using deriveLinkedEntries.

I think that the deriveLinkedEntries method could be a good place to start to support this. I know that there was some recent work done in #127 that solved an n to n deriveLinkedEntries migration that seems in line.

This is more of a feature request than a bug (unless there is a way to do this with the existing tools - without getting too deep into custom API work). I filed this after suggetion from Stefan on Slack.

Yes that make sense for advanced use cases, It would be nice if you can opne an issue as a feature request.

[gkulasik-livongo]

Thanks @Khaledgarbaya!

Any ideas on the following? This is the last issue I am struggling with:

Also, to build on this. Let's say this works and TagSet now has a list of Tags. If TagSet was part of another content type called Article, meaning TagSet is a field on the Article content type, is it possible to then move the new TagSet.tags to Article.tags (assuming it has an equivalent field already created to consume the tags form TagSet).

Basically, I am trying to move this list of 'tags' up one level to eliminate an unnecessary content type for our new use case.

I'm doing this work for my employer so will need some time to come up with a non-proprietary example for posting.

Also, for the feature request, where do I open that? I thought this issue on Github was the feature request. Is there another location that you prefer to have the feature requests filed?

[gkulasik-livongo]

Solve this through my procedural comments above (#142 (comment)).

For the final questions, I solved those via the python content management sdk rather than using JS.