Was this page helpful?

Integrating migrations in a continuous delivery pipeline with CircleCI

To Skip the tutorial and navigate to the code, see the example repo on GitHub.

Introduction

Like handling changes to a database, integrating Contentful into a continuous integration (CI) pipeline enables changes to a content model and migrating data as part of a deployment process. Further, it enables testing content changes and canceling a deploy process thereby preventing interruptions or downtime, in case the tests fail.

Using the Contentful CLI's space migration command, it is possible to create, delete, and edit content models and entries programmatically entirely in JavaScript. By using the Migrations DSL tool over the Contentful web app, updating a content model becomes a repeatable process. Changes made to a content model can also be tracked by integrating it to the version control.

Space environments are entities within a space that allow a developer to create and maintain multiple versions of space-specific data, and make changes to them in isolation. This promotes teams to work on modifying a Contentful space, as any changes made to an environment do not affect production data. Every time you create a new environment, you're creating a copy from the current version of your master space. Contentful migration tooling, when combined with space environments, starts integrating Contentful into the CI Pipeline.

A typical continuous integration pipeline creates a build and runs tests against it, before deploying code. Integrating content migrations into that pipeline requires additional steps.

In the build phase, add a step to programmatically add a new environment, while running any needed migrations as part of deployment on that new environment. During the testing phase, utilize the newly-migrated environment rather than the master. Assuming everything passes, delete the environment, run the migrations against master and then deploy the code into production.

Pre-requisites

Setting up Continuous Integration

Step 1: Fork and clone our example repo and configure it to utilize a Contentful space

For this project we’ll be setting up a Continuous Integration pipeline for a barebones Flask App. The tutorial repository includes the Flask site and an export of a Contentful space which has a single content type.

After cloning our tutorial repository, create a new Contentful space and import the tutorial content model.

The Contentful part (required)

  • Create a new space using the Contentful CLI
    $ contentful space create --name "continuous delivery example"
  • Set the newly-created space as the default space for all further CLI operations. This will present a list of all available spaces – choose the one we just created
    $ contentful space use
  • Import the provided content model (./import/export.json) into the newly-created space
    $ contentful space import --content-file ./import/export.json

These following steps to get the Flask site up and running are completely optional as it will only be used to test and build in our CI Pipeline. We won’t be making changes to the Python code outside of our tests in this example.

Local development environment (optional)

  • Create a virtual environment
    $ virtualenv env
  • Activate the virtual environment
    $ source env/bin/activate
  • Install all Python dependencies
    $ pip install -r requirements.txt
  • Start the Flask app
    $ python myapp.py
  • Rename config file .env.example to .env and add missing keys

Step 2: Enable CircleCI to create a new Environment

Using CircleCI, we’ll create a list of steps that are run as part of the build, test and deployment phases. For this project, whenever we trigger a build phase, we want CircleCI to create a new environment, run a migration against that environment and then run our tests against that migrated environment. For each of those steps, we’ll need to write code that CircleCI can run to execute that step. Create a folder named scripts and inside it, a Javascript file called migrate.js.

To begin, we’ll need to import our dependencies and connect to Contentful via the content management SDK.

(async () => {
  const {promisify} = require('util');
  const {readdir} = require('fs');
  const readdirAsync = promisify(readdir);
  const path = require('path');
  const { createClient } = require('contentful-management');
  const {default: runMigration} = require('contentful-migration/built/bin/cli');

  // utility fns
  const getVersionOfFile = (file) => file.replace('.js', '').replace(/_/g, '.');
  const getFileOfVersion = (version) => version.replace(/\./g, '_') + '.js';

  //
  // Configuration variables
  //
  const [,, SPACE_ID, ENVIRONMENT_ID, CMA_ACCESS_TOKEN] = process.argv;
  const MIGRATIONS_DIR = path.join('.', 'migrations');

  const client = createClient({
    accessToken: CMA_ACCESS_TOKEN
  });
  const space = await client.getSpace(SPACE_ID);

  let environment;
  console.log('Running with the following configuration');
  console.log(`SPACE_ID: ${SPACE_ID}`);
  console.log(`ENVIRONMENT_ID: ${ENVIRONMENT_ID}`);

Next we'll check if an environment already exists and if it does, to destroy it. When we call this script, we’ll provide it with the name of the branch that it will use to create the new environment. Then, a new environment that’s a copy of master is created and polled until we received confirmation that it exists.

    // ---------------------------------------------------------------------------
    console.log(`Checking for existing versions of environment: ${ENVIRONMENT_ID}`);

    try {
      environment = await space.getEnvironment(ENVIRONMENT_ID);
      if (ENVIRONMENT_ID != 'master'){
        await environment.delete();
        console.log('Environment deleted');
      }
    } catch(e) {
      console.log('Environment not found');
    }

    // ---------------------------------------------------------------------------
    if (ENVIRONMENT_ID != 'master'){
      console.log(`Creating environment ${ENVIRONMENT_ID}`);
      environment = await space.createEnvironmentWithId(ENVIRONMENT_ID, { name: ENVIRONMENT_ID });
    }
    // ---------------------------------------------------------------------------
    const DELAY = 3000;
    const MAX_NUMBER_OF_TRIES = 10;
    let count = 0;

    console.log('Waiting for environment processing...')

    while (count < MAX_NUMBER_OF_TRIES) {
      const status = (await space.getEnvironment(environment.sys.id)).sys.status.sys.id;

      if (status === 'ready' || status === 'failed') {
        if (status === 'ready') {
          console.log(`Successfully processed new environment (${ENVIRONMENT_ID})`);
        } else {
          console.log('Environment creation failed');
        }
        break;
      }

      await new Promise(resolve => setTimeout(resolve, DELAY));
      count++;
    }

Lastly we'll update our API keys to have access to the newly created environment so that our tests will be able to utilize it with the existing delivery API key we'll configure later in CircleCI.

    // ---------------------------------------------------------------------------
    console.log('Update API keys to allow access to new environment');
    const newEnv = {
      sys: {
        type: 'Link',
        linkType: 'Environment',
        id: ENVIRONMENT_ID
      }
    }

    const {items: keys} = await space.getApiKeys();
    await Promise.all(keys.map(key => {
      console.log(`Updating - ${key.sys.id}`);
      key.environments.push(newEnv);
      return key.update();
    }));

Step 3: Create a new Content Type named versionTracking

For CircleCI to know which migrations it should run, we’ll need to track which migrations have been run by adding a version number into Contentful. We accomplish this in Contentful by creating a new content model with an ID of versionTracking that has a single short-text-field named version.

You’ll also need to create one entry of your new content model with the value 1. We'll be using integers in this demo to track migrations.

Step 4: Create migrations

The Contentful Migration DSL allows us to:

  • Create, delete and edit content models and content entries all in JavaScript.
  • Establish a repeatable process of updating content model (where the Contentful web app does not)
  • Track changes by integrating them into our version control; hence implementing CMS as Code.

The provided space in the example repo contains a list of Marvel superheroes; each having a name, GIF, first appearance and slug. For our first migration, we’ll modify the content model to include an author field.

Start by creating a new folder in the root directory named migration—this is where we’ll place all our migrations. We’ll need to create an empty migration file to represent the initial import that we did in step 1. Create 1.js and include the following code:

module.exports = function runMigration(migration) {
  return;
};

Next, create a JavaScript file named 2.js with the following code to add an author field to our content model:

module.exports = function runMigration(migration) {
  const post = migration.editContentType("post");
  post
    .createField("author")
    .name("author")
    .type("Symbol")
    .required(false);
};

Lastly, we’ll create a second migration named 3.js to set the newly created author field of each hero as Stan Lee.

module.exports = function (migration) {
  migration.transformEntries({
    contentType: 'post',
    from: ['author'],
    to: ['author'],
    transformEntryForLocale: function (fromFields, currentLocale) {
      const author = "Stan Lee";
      return { author: author };
    }
  });
};

Since we’ve changed our content model, we’ll also need to update our tests. Update the test_content_type_post function in the test_app.py file with the following:

def test_content_type_post(self, contentful_client):
        """Test content model of a post"""
        post_content_type = contentful_client.content_type("post")
    # Expect 5 fields in a post now that we’ve added an author field.
        assert len(post_content_type.fields) == 5
        title = next(d for d in post_content_type.fields if d.id == "title")
        assert title.id == "title"
        assert title.type == "Symbol"

And add the following test into our function:

        author = next(d for d in post_content_type.fields if d.id == "author")
        assert author.id == "author"
        assert author.type == "Symbol"

Step 5: Update migrate.js to run our migrations.

Now that we have a directory of migrations to run against our space, we’ll need to update our script so that CircleCI can run those migrations for us.

First we'll append to migrate.js code to set our default locale, check what migrations we have available and the finally detect what migrations need to be run by looking at our versionTracking entry.

    // ---------------------------------------------------------------------------
    console.log('Set default locale to new environment');
    const defaultLocale = (await environment.getLocales()).items
      .find(locale => locale.default).code;

    // ---------------------------------------------------------------------------
    console.log('Read all the available migrations from the file system');
    const availableMigrations = (await readdirAsync(MIGRATIONS_DIR))
      .filter(file => /^\d+?\.js$/.test(file))
      .map(file => getVersionOfFile(file));

    // ---------------------------------------------------------------------------
    console.log('Figure out latest ran migration of the contentful space');
    const {items: versions} = await environment.getEntries({
      content_type: 'versionTracking'
    });

    if (!versions.length || versions.length > 1) {
      throw new Error(
        'There should only be one entry of type \'versionTracking\''
      );
    }

    let storedVersionEntry = versions[0];
    const currentVersionString = storedVersionEntry.fields.version[defaultLocale];

    // ---------------------------------------------------------------------------
    console.log('Evaluate which migrations to run');
    const currentMigrationIndex = availableMigrations.indexOf(currentVersionString);

    if (currentMigrationIndex === -1) {
      throw new Error(
        `Version ${currentVersionString} is not matching with any known migration`
      );
    }
    const migrationsToRun = availableMigrations.slice(currentMigrationIndex + 1);
    const migrationOptions = {
      spaceId: SPACE_ID,
      environmentId: ENVIRONMENT_ID,
      accessToken: CMA_ACCESS_TOKEN,
      yes: true
    };

Lastly we'll run all the migrations that we've evaluated as being necessary and update the version number.

    // ---------------------------------------------------------------------------
    console.log('Run migrations and update version entry');
    while(migrationToRun = migrationsToRun.shift()) {
      const filePath = path.join(__dirname, '..', 'migrations', getFileOfVersion(migrationToRun));
      console.log(`Running ${filePath}`);
      await runMigration(Object.assign(migrationOptions, {
        filePath
      }));
      console.log(`${migrationToRun} succeeded`);

      storedVersionEntry.fields.version[defaultLocale] = migrationToRun;
      storedVersionEntry = await storedVersionEntry.update();
      storedVersionEntry = await storedVersionEntry.publish();

      console.log(`Updated version entry to ${migrationToRun}`);
    }

    console.log('All done!');
  } catch(e) {
    console.error(e);
    process.exit(1);
  }
})();

The finalized migrate.js file should be identical to the version in our completed example.

Step 6: Create CircleCI Config File

The final step before we can push our code to GitHub is to create a CircleCI config file. We’ll start with the default CircleCI config file, which is included in the example directory, and make a few changes. As CircleCI instructs, save, the config.yml file is inside a folder named .circleci on the root directory.

We’ll need to add several modifications to the default config.yml provided by CircleCI.

Start by updating the image to include a version that comes with Node since we’ll need that to use the migration DSL:

- image: circleci/python:3.6.5-node

Update the install dependencies step to additionally install the requirements listed in the package.json provided in the repo:

     - run:
          name: install dependencies
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install -r requirements.txt
            npm install

Add a step to prepare the environment for testing immediately after installing dependencies, but before running the tests. This step will utilize the previously created migrate.js file we created in steps 2.

     - run:
          name: Preparing environment for testing
          command: |
            . venv/bin/activate
            scripts/migrate.js $SPACE_ID "CI_$CIRCLE_BRANCH" $MANAGEMENT_API_KEY

Update the test step to utilize the new environment.

     # run tests!
      - run:
          name: run tests
          command: |
            . venv/bin/activate
            pytest --environment-id="CI_$CIRCLE_BRANCH"

Once completed the config.yml file should look like this.

If we want to optionally include the deployment in our initial CI rollout, add the following deployment step into the config file:

  deploy:
      docker:
        - image: circleci/python:3.6.5-node
      environment:
        # Update HEROKU_APP with the application Name.
        HEROKU_APP: "migration-env-demo"
      steps:
        - checkout

        # Download and cache dependencies
        - restore_cache:
            keys:
            - v1-dependencies-{{ checksum "package.json" }}
            # fallback to using the latest cache if no exact match is found
            - v1-dependencies-

        - run:
            name: install dependencies
            command: |
              npm install

        - save_cache:
            paths:
              - node_modules
            key: v1-dependencies-{{ checksum "package.json" }}

        - run:
            name: Updating Master Environment
            command: |
              scripts/migrate.js $SPACE_ID master $MANAGEMENT_API_KEY

        - run:
            name: Deploy Master to Heroku
            command: |
              git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP.git master

Step 7: Push code to GitHub and configure CircleCI

Now that we’ve made the changes needed for the project, we can push the code to GitHub on master. After logging in into the CircleCI website, click “Set Up Project” on the repository we’re using.

Click “Start building”—this first build will fail because it’ll be missing environment variables. From this screen, click the gear icon to be taken to the project settings. The project settings page allows us to access the environment variables where we can add the following keys: DELIVERY_API_KEY, MANAGEMENT_API_KEY, SPACE_ID and HEROKU_API_KEY.

Rerun the build and everything should pass!

What’s next?

After following these steps, we’ll have a fully-functional CI pipeline integrated with Contentful. Any time someone make a PR, CircleCI will automatically run tests and create a new environment based on the branch name that will be used to verify code quality.

Using this example, to run a migration against Contentful create a new migration file inside the migration folder and bump up the version number. Unfortunately, with this method, once a migration is pushed to production we won’t be able to revert. However, any bad migrations should not affect a production space since any failed tests will cause the build phase of this CI pipeline to not trigger. In fact, since CircleCI runs tests and migrations that are part of the PR review process, a bad migration should theoretically never hit production assuming it isn’t accidentally merged into master on GitHub.