Migrate content to Rich Text

The objective of this guide is to migrate your existing content to Rich Text. The existing content exists in the following form:

• Markdown; for plain text
• References; for Composable entries

To explain the steps involved in content migration effectively, the Example App is referred. The Example App uses a Content model that has content pertaining to both the forms mentioned above. Thereby, providing a perfect platform to perform the following transformations for the entries of content types:

• “Lesson > Copy” -> “Rich Text” text
• “Lesson > Image” -> “Rich Text” Embedded Asset
• “Lesson > Code Snippet” -> “Rich Text” Embedded Entry

that are linked from “Lesson” entries.

Prerequisites

• Knowledge of JavaScript and familiarity with Contentful’s migration tooling.
• The migration tool installed in your machine.
• A sandbox environment (available from micro spaces and above).

1. Create an Example app

To replicate the Example app, create a sample space with this Content model first:

Here you can navigate to an entry of “Lesson” Content-Type, similar to the one explaining how content modeling works:

2. Create a sandbox environment

To mitigate the risk of making changes that hamper your production application, create a new sandbox environment. Let’s call this “rich-text-migration”:

3. Add a Rich Text field

This can also be done with the migration script. In this case, the UI is used to keep the code sample light and focused on migration only logic.

4. Create the migration file

You can skip all the intermediate steps and go straight to step 10 to access the final file of the migration.

Following is the shell for the migration file:

module.exports = function(migration) {
  migration.transformEntries({
    contentType: 'lesson',
    from: ['modules'],
    to: ['copy'],
    transformEntryForLocale: function(fromFields) {
      // <-- Logic for migration will go here
      return {};
    }
  });
};

The objective of the next steps is to change the “lesson” content type by:

• transforming the content of the “modules” field content to Rich Text format and,
• export it (currently an empty object is exported) to the newly created “Copy” field.

5. Get the linked Modules for the current Lesson

While iterating over the different entries, you might first need a collection of the Modules that are linked by your “Modules” field. The following code gets the linked Modules:

module.exports = function(migration, { makeRequest }) {
  migration.transformEntries({
    contentType: 'lesson',
    from: ['modules'],
    to: ['copy'],
    transformEntryForLocale: async function(fromFields) {
      // Get the "Lesson > *" modules that are linked to the "modules" field
      // the modules field itself isn't localized, but some of the links contained in the array point to localizable entries.
      const moduleIDs = fromFields.modules['en-US'].map(e => e.sys.id);
      const moduleEntries = await makeRequest({
        method: 'GET',
        url: `/entries?sys.id[in]=${moduleIDs.join(',')}`
      });
      // Filter down to just these Lessons linked by the current entry
      const linkedModuleEntries = moduleIDs.map(id =>
        moduleEntries.items.find(entry => entry.sys.id === id)
      );

      return {};
    }
  });
};

6. Convert “Lesson > Image” entries to Rich Text images

As Rich Text supports images, we can take the “media” field of the linked “Lesson > Image” entries and turn that into a Rich Text embedded asset. There is already some conditional logic on the content type id of the linked module but for now, the Rich Text field with images is populated.

const _ = require('lodash');

module.exports = function(migration, { makeRequest }) {
  migration.transformEntries({
    contentType: 'lesson',
    from: ['modules'],
    to: ['copy'],
    transformEntryForLocale: async function(fromFields) {
      // Get the "Lesson > *" modules that are linked to the "modules" field
      // the modules field itself isn't localized, but some of the links contained in the array point to localizable entries.
      const moduleIDs = fromFields.modules['en-US'].map(e => e.sys.id);
      const moduleEntries = await makeRequest({
        method: 'GET',
        url: `/entries?sys.id[in]=${moduleIDs.join(',')}`
      });
      // Filter down to just these Lessons linked by the current entry
      const linkedModuleEntries = moduleIDs.map(id =>
        moduleEntries.items.find(entry => entry.sys.id === id)
      );

      const allNodeArrays = await Promise.all(
        linkedModuleEntries.map(linkedModule => {
          return transformLinkedModule(linkedModule, currentLocale);
        })
      );

      // The content property of the Rich Text document is an array of paragraphs, embedded entries, embedded assets.
      const content = _.flatten(allNodeArrays);

      // The returned Rich Text object to be added to the new "copy" field
      const result = {
        copy: {
          nodeType: 'document',
          content: contentArray,
          data: {}
        }
      };
      return result;

      // This will have logic for the other linkedModule types like Lesson > Code Snippets and Lesson > Copy
      function transformLinkedModule(linkedModule) {
        switch (linkedModule.sys.contentType.sys.id) {
          case 'lessonImage':
            return embedImageBlock(linkedModule);
        }
      }

      // Return a Rich Text embedded asset object
      function embedImageBlock(lessonImage) {
        // This field is not localized.
        const asset = lessonImage.fields.image['en-US'];
        return [
          {
            nodeType: 'embedded-asset-block',
            content: [],
            data: {
              target: {
                sys: {
                  type: 'Link',
                  linkType: 'Asset',
                  id: asset.sys.id
                }
              }
            }
          }
        ];
      }
    }
  });
};

7. Convert “Lesson > Code Snippets” entries to Rich Text embedded entries

The “Lesson > Code Snippets” entries are embedded as entries instead of assets:

const _ = require('lodash');

module.exports = function(migration, { makeRequest }) {
  migration.transformEntries({
    contentType: 'lesson',
    from: ['modules'],
    to: ['copy'],
    transformEntryForLocale: async function(fromFields) {
      // Get the "Lesson > *" modules that are linked to the "modules" field
      // the modules field itself isn't localized, but some of the links contained in the array point to localizable entries.
      const moduleIDs = fromFields.modules['en-US'].map(e => e.sys.id);
      const moduleEntries = await makeRequest({
        method: 'GET',
        url: `/entries?sys.id[in]=${moduleIDs.join(',')}`
      });
      // Filter down to just these Lessons linked by the current entry
      const linkedModuleEntries = moduleIDs.map(id =>
        moduleEntries.items.find(entry => entry.sys.id === id)
      );

      const allNodeArrays = await Promise.all(
        linkedModuleEntries.map(linkedModule => {
          return transformLinkedModule(linkedModule, currentLocale);
        })
      );

      // The content property of the Rich Text document is an array of paragraphs, embedded entries, embedded assets.
      const content = _.flatten(allNodeArrays);

      // The returned Rich Text object to be added to the new "copy" field
      const result = {
        copy: {
          nodeType: 'document',
          content: contentArray,
          data: {}
        }
      };
      return result;

      // This will have logic for the other linkedModule types like Lesson > Copy
      function transformLinkedModule(linkedModule) {
        switch (linkedModule.sys.contentType.sys.id) {
          case 'lessonImage':
            return embedImageBlock(linkedModule);
          case 'lessonCodeSnippets':
            return embedCodeSnippet(linkedModule);
        }
      }

      // Return a Rich Text embedded asset object
      function embedImageBlock(lessonImage) {
        // This field is not localized.
        const asset = lessonImage.fields.image['en-US'];
        return [
          {
            nodeType: 'embedded-asset-block',
            content: [],
            data: {
              target: {
                sys: {
                  type: 'Link',
                  linkType: 'Asset',
                  id: asset.sys.id
                }
              }
            }
          }
        ];
      }
      // Return a Rich Text embedded entry object
      function embedCodeSnippet(lessonCodeSnippet) {
        return [
          {
            nodeType: 'embedded-entry-block',
            content: [],
            data: {
              target: {
                sys: {
                  type: 'Link',
                  linkType: 'Entry',
                  id: lessonCodeSnippet.sys.id
                }
              }
            }
          }
        ];
      }
    }
  });
};

8. Convert the Markdown text in “Lesson > Copy” to Rich Text

In this step, the Markdown text is transformed using the rich-text-from-markdown tool.

In order to install it, run:

npm i @contentful/rich-text-from-markdown

Then, update your migration script:

const _ = require('lodash');

const { richTextFromMarkdown } = require('@contentful/rich-text-from-markdown');

module.exports = function(migration, { makeRequest }) {
  migration.transformEntries({
    contentType: 'lesson',
    from: ['modules'],
    to: ['copy'],
    transformEntryForLocale: async function(fromFields) {
      // the modules field itself isn't localized, but some of the links contained in the array point to localizable entries.
      const moduleIDs = fromFields.modules['en-US'].map(e => e.sys.id);
      const moduleEntries = await makeRequest({
        method: 'GET',
        url: `/entries?sys.id[in]=${moduleIDs.join(',')}`
      });
      // Filter down to just these Lessons linked by the current entry
      const linkedModuleEntries = moduleIDs.map(id =>
        moduleEntries.items.find(entry => entry.sys.id === id)
      );

      const allNodeArrays = await Promise.all(
        linkedModuleEntries.map(linkedModule => {
          return transformLinkedModule(linkedModule, currentLocale);
        })
      );

      // The content property of the Rich Text document is an array of paragraphs, embedded entries, embedded assets.
      const content = _.flatten(allNodeArrays);

      // The returned Rich Text object to be added to the new "copy" field
      const result = {
        copy: {
          nodeType: 'document',
          content: contentArray,
          data: {}
        }
      };
      return result;

      async function transformLinkedModule(linkedModule, locale) {
        switch (linkedModule.sys.contentType.sys.id) {
          case 'lessonCopy':
            const richTextDocument = await transformLessonCopy(
              linkedModule,
              locale
            );
            return richTextDocument.content;
          case 'lessonImage':
            return embedImageBlock(linkedModule);
          case 'lessonCodeSnippets':
            return embedCodeSnippet(linkedModule);
        }
      }

      // Return Rich Text instead of Markdown
      async function transformLessonCopy(lessonCopy, locale) {
        const copy = lessonCopy.fields.copy[locale];
        return await richTextFromMarkdown(copy);
      }
      // Return a Rich Text embedded asset object
      function embedImageBlock(lessonImage) {
        // This field is not localized.
        const asset = lessonImage.fields.image['en-US'];
        return [
          {
            nodeType: 'embedded-asset-block',
            content: [],
            data: {
              target: {
                sys: {
                  type: 'Link',
                  linkType: 'Asset',
                  id: asset.sys.id
                }
              }
            }
          }
        ];
      }
      // Return a Rich Text embedded entry object
      function embedCodeSnippet(lessonCodeSnippet) {
        return [
          {
            nodeType: 'embedded-entry-block',
            content: [],
            data: {
              target: {
                sys: {
                  type: 'Link',
                  linkType: 'Entry',
                  id: lessonCodeSnippet.sys.id
                }
              }
            }
          }
        ];
      }
    }
  });
};

9. Convert the Markdown text with images

This section explains how to transform Markdown text with images to the Rich Text Document with embedded assets.

To get started, update your migration script for Lesson > Copy:

const mimeType = {
  bmp: 'image/bmp',
  djv: 'image/vnd.djvu',
  djvu: 'image/vnd.djvu',
  gif: 'image/gif',
  jpeg: 'image/jpeg',
  jpg: 'image/jpeg',
  pbm: 'image/x-portable-bitmap',
  pgm: 'image/x-portable-graymap',
  png: 'image/png',
  pnm: 'image/x-portable-anymap',
  ppm: 'image/x-portable-pixmap',
  psd: 'image/vnd.adobe.photoshop',
  svg: 'image/svg+xml',
  svgz: 'image/svg+xml',
  tif: 'image/tiff',
  tiff: 'image/tiff',
  xbm: 'image/x-xbitmap',
  xpm: 'image/x-xpixmap',
  '': 'application/octet-stream'
};
const getContentType = url => {
  const index = url.lastIndexOf('.');
  const extension = index === -1 ? '' : url.substr(index + 1);
  return mimeType[extension];
};
const getFileName = url => {
  const index = url.lastIndexOf('/');
  const fileName = index === -1 ? '' : url.substr(index + 1);
  return fileName;
};

// Return Rich Text instead of Markdown
async function transformLessonCopy(lessonCopy, locale) {
  const copy = lessonCopy.fields.copy[locale];
  return await richTextFromMarkdown(copy, async mdNode => {
    if (mdNode.type !== 'image') {
      return null;
    }
    // Create and asset and publish it
    const space = await managementClient.getSpace(spaceId);
    // Unfortunately, we can't pull the environment id from the context
    const environment = await space.getEnvironment('rich-text-migration');

    let asset = await environment.createAsset({
      fields: {
        title: {
          'en-US': mdNode.title ? mdNode.title + locale : mdNode.alt + locale
        },
        file: {
          'en-US': {
            contentType: getContentType(mdNode.url),
            fileName: getFileName(mdNode.url) + locale,
            upload: `https:${mdNode.url}`
          }
        }
      }
    });
    asset = await asset.processForAllLocales({
      processingCheckWait: 4000
    });
    asset = await asset.publish();
    console.log(`published asset's id is ${asset.sys.id}`);
    return {
      nodeType: 'embedded-asset-block',
      content: [],
      data: {
        target: {
          sys: {
            type: 'Link',
            linkType: 'Asset',
            id: asset.sys.id
          }
        }
      }
    };
  });
}

10. Markdown migration script

Let's combine all pieces of the puzzle in one script:

const richTextFromMarkdown = require('@contentful/rich-text-from-markdown')
  .richTextFromMarkdown;
const _ = require('lodash');
const { createClient } = require('contentful-management');

const mimeType = {
  bmp: 'image/bmp',
  djv: 'image/vnd.djvu',
  djvu: 'image/vnd.djvu',
  gif: 'image/gif',
  jpeg: 'image/jpeg',
  jpg: 'image/jpeg',
  pbm: 'image/x-portable-bitmap',
  pgm: 'image/x-portable-graymap',
  png: 'image/png',
  pnm: 'image/x-portable-anymap',
  ppm: 'image/x-portable-pixmap',
  psd: 'image/vnd.adobe.photoshop',
  svg: 'image/svg+xml',
  svgz: 'image/svg+xml',
  tif: 'image/tiff',
  tiff: 'image/tiff',
  xbm: 'image/x-xbitmap',
  xpm: 'image/x-xpixmap',
  '': 'application/octet-stream'
};
const getContentType = url => {
  const index = url.lastIndexOf('.');
  const extension = index === -1 ? '' : url.substr(index + 1);
  return mimeType[extension];
};
const getFileName = url => {
  const index = url.lastIndexOf('/');
  const fileName = index === -1 ? '' : url.substr(index + 1);
  return fileName;
};

const ENV_NAME = 'rich-text-migration';

module.exports = function(migration, { makeRequest, spaceId, accessToken }) {
  const managementClient = createClient({ accessToken: accessToken }, { type: 'plain' });

  migration.transformEntries({
    contentType: 'lesson',
    from: ['modules'],
    to: ['copy'],
    transformEntryForLocale: async function(fromFields, currentLocale) {
      // Get the "Lesson > *" modules that are linked to the "modules" field
      // the modules field itself isn't localized, but some of the links contained in the array point to localizable entries.
      const moduleIDs = fromFields.modules['en-US'].map(e => e.sys.id);
      const moduleEntries = await makeRequest({
        method: 'GET',
        url: `/entries?sys.id[in]=${moduleIDs.join(',')}`
      });
      // Filter down to just these Lessons linked by the current entry
      const linkedModuleEntries = moduleIDs.map(id =>
        moduleEntries.items.find(entry => entry.sys.id === id)
      );

      const allNodeArrays = await Promise.all(
        linkedModuleEntries.map(linkedModule => {
          return transformLinkedModule(linkedModule, currentLocale);
        })
      );

      // The content property of the Rich Text document is an array of paragraphs, embedded entries, embedded assets.
      const content = _.flatten(allNodeArrays);

      // The returned Rich Text object to be added to the new "copy" field
      var result = {
        copy: {
          nodeType: 'document',
          content: content,
          data: {}
        }
      };
      return result;

      async function transformLinkedModule(linkedModule, locale) {
        switch (linkedModule.sys.contentType.sys.id) {
          case 'lessonCopy':
            const richTextDocument = await transformLessonCopy(
              linkedModule,
              locale
            );
            return richTextDocument.content;
          case 'lessonImage':
            return embedImageBlock(linkedModule);
          case 'lessonCodeSnippets':
            return embedCodeSnippet(linkedModule);
        }
      }

      // Return Rich Text instead of Markdown
      async function transformLessonCopy(lessonCopy, locale) {
        const copy = lessonCopy.fields.copy[locale];
        return await richTextFromMarkdown(copy, async mdNode => {
          if (mdNode.type !== 'image') {
            return null;
          }
          // Create and asset and publish it
          const space = await managementClient.space.get({
            spaceId,
          });
          // Unfortunately, we can't pull the environment id from the context
          const environment = await client.environment.get({
            spaceId: space.sys.id,
            environmentId: ENV_NAME,
          });

          let asset = await client.asset.create(
            {
              spaceId: space.sys.id,
              environmentId: environment.sys.id,
            },
            {
              fields: {
                title: {
                  'en-US': mdNode.title
                    ? mdNode.title + locale
                    : mdNode.alt + locale
                },
                file: {
                  'en-US': {
                    contentType: getContentType(mdNode.url),
                    fileName: getFileName(mdNode.url) + locale,
                    upload: `https:${mdNode.url}`
                  }
                }
              }
            }
          );
          asset = await client.asset.processForAllLocales(
            {
              spaceId: space.sys.id,
              environmentId: environment.sys.id,
            },
            {
              ...asset,
            },
            {
              processingCheckWait: 4000
            }
          );
          asset = await client.asset.publish(
            {
              spaceId: space.sys.id,
              environmentId: environment.sys.id,
              assetId: asset.sys.id,
            },
            { ...asset },
          );
          console.log(`published asset's id is ${asset.sys.id}`);
          return {
            nodeType: 'embedded-asset-block',
            content: [],
            data: {
              target: {
                sys: {
                  type: 'Link',
                  linkType: 'Asset',
                  id: asset.sys.id
                }
              }
            }
          };
        });
      }
      // Return a Rich Text embedded asset object
      function embedImageBlock(lessonImage) {
        // This field is not localized.
        const asset = lessonImage.fields.image['en-US'];
        return [
          {
            nodeType: 'embedded-asset-block',
            content: [],
            data: {
              target: {
                sys: {
                  type: 'Link',
                  linkType: 'Asset',
                  id: asset.sys.id
                }
              }
            }
          }
        ];
      }
      // Return a Rich Text embedded entry object
      function embedCodeSnippet(lessonCodeSnippet) {
        return [
          {
            nodeType: 'embedded-entry-block',
            content: [],
            data: {
              target: {
                sys: {
                  type: 'Link',
                  linkType: 'Entry',
                  id: lessonCodeSnippet.sys.id
                }
              }
            }
          }
        ];
      }
    }
  });
};

11. What about unsupported Markdown?

Rich Text does not support the following Markdown functionalities:

• Tables
• Code block
• Strike-through

To migrate the above content to Rich Text, you have the following three options:

1. Migrate the content into a linked entry which only has a Markdown field with the supported content, which is similar to the action performed in step 8.
2. Convert the content in a manner that its core meaning is migrated but not its formatting. For example, convert a table to a list or a link to an external PDF with that table.
3. Ignore the content and do not migrate it.

The aforementioned rich-text-from-markdown tool has a callback function where you can decide the logic for migrating every Markdown element. Thereby, giving you control to perform each of the above options. You can read more in the repo.

12. Run the migration

You are now ready to run this migration script by running the following command in your shell:

contentful space migration -s [YOUR_SPACE_ID] -e rich-text-migration -a [YOUR_CMA_TOKEN] migration.js

Conclusion

After running the migration script successfully, the Rich Text field is available on your “Lesson” entries with content that contains rich text content, embedded assets, and embedded code snippets.

You are now ready to:

1. Do the corresponding front-end changes to your application. You can get started by referring the Rich Text guide.
2. Delete the sandbox environment.
3. Create the Rich Text field in your master environment (you can disable editing until the migration script is run).
4. Run the above migration on Production.
5. Promote your code changes to Production.