Developer Showcase Spotlight: Deploying flexible data structures with Contentful and TypeScript
Application development today is increasingly agile. The applications being developed are more flexible, and can better adapt to new insights and changing requirements. As a result, the data structures used by these applications need to be more flexible too.
What this means is that many of the systems which application development relies upon have also become more flexible. Contentful and TypeScript are no exception!
Developing large-scale applications with flexible data structures brings challenges, however. To help with this I’ve created Modelberry Factory, which I recently submitted to the Contentful Developer Showcase.
In this post, I’ll explain more about these challenges, the existing solutions, and how Modelberry Factory helps.
The evolution of application development
In the earliest days of application development, the design phase played a vital role. Feasibility studies would be carried out to understand the market fit, and only then would the application be moved to the development phase – where the actual coding happens!
However, this has changed over time. The design phase is still important, but it has been made more modular. Today, a minimum viable product is first designed and developed: A small part of the greater goal. From that tiny part, new features are added as the need arises.
This makes the modern development processes flexible. Adapting to the ever-changing user requirements got a lot easier. A flexible process, however, comes with one or two strings attached. Flexible processes require flexible data structures – which isn't as straightforward as it might sound!
Flexible data structures
Following the flexibility dictated by software development trends like lean and agile, Contentful allows for flexible data structures.
Contentful is a platform that provides flexible data structures in which content types are easily defined and updated. This flexibility requires solid content type maintenance because a simple change of a content type can have a large impact on connected systems. If you change a content type, you might have to make changes in your application to adapt to that change.
Just like Contentful, TypeScript provides flexible data structures where types are easily defined and updated. TypeScript is JavaScript with syntax for types. However, the same maintenance of types is required here. A simple change of a TypeScript type can have a large impact on connected modules and systems.
A note about models
Data structures within Contentful are called ‘content types’. Within TypeScript these structures are referred to as ‘types’.
This post refers to both structures as models. A different name, but exactly the same thing.
In search of the perfect model
Clearly defined models make it possible for various components of an application to communicate. For example, when a TypeScript module is required to communicate with Contentful, the module and Contentful share the same model.
When starting a project from scratch, defining a model is usually straightforward. The model is not used anywhere yet. Every model that fits the application requirements is a good start.
However, updating a model in a legacy context or in the context of a large-scale application is less straightforward. This is because models might already exist and these models are typically used across many modules and systems.
A model update likely requires changes on connected modules and systems. This could cause unforeseen problems because the connected modules and systems were not in scope when initiating the model update.
The perfect model is the model that survives the lifespan of an application without any changes. The more we know about an application when the initial models are defined, the better a model can be prepared for future changes. However, no matter how much is known initially, application development is a process that brings new insights along the way, which results in requiring models to change.
In practice, most applications do not start with a perfect model. The application starts with a simple model for the minimum viable product. This enables for a quick first release. Models for future releases evolve as the application evolves.
The challenges of defining models across modules and systems
Models are typically defined across modules and systems. This is true for Contentful and TypeScript because both define models. But there are a few challenges:
Keeping models in sync
Contentful models and TypeScript models must be in sync for the application to function properly. Keeping these models in sync is one of the challenges when working with flexible models on separate systems. If you update the Contentful model, but don’t update TypeScript, your application might break.
Keeping dev teams informed
Even a small model change could make an application fail. Hence every model change for every release and for every environment like ‘dev’ and ‘prod’ must be documented. A dev team must be able to track every model change that happens over time. However, setting this up is a challenge when working with flexible models on separate systems.
Keeping TypeScript code base up to date
A change on the Contentful side still may result in complex changes on the TypeScript side. Being forced to handle complex changes on the TypeScript side is also a challenge when working with flexible models on separate systems. I've listed it here as a challenge, but in some circumstances it might also be considered an advantage. The need to handle complex changes right away keeps the code base clean and up to date.
Methods for syncing models on separate systems
Of the three challenges we’ve discussed above, syncing the models is the main challenge. If syncing is simple, communication to the dev team is simple. The dev team can then focus on fixing the changes caused by the model update.
Hence, syncing the models in Contentful and TypeScript is the key strategy for working with flexible models on separate systems.
As mentioned previously, working with flexible models on separate systems requires model syncing to be simple. In this section, I’ll cover three approaches to achieve it.
1. Manually syncing
This method keeps Contentful and TypeScript models manually in sync. The TypeScript code base for this method is configured with a testing framework like Jest and a unit test exists for all model definitions.
This method uses unit tests for model definitions because tests are a common source of truth. The steps involved in this approach are as follows:
Steps in Contentful
Apply model changes manually using the Contentful web app.
Steps for the TypeScript code base
Update model definition unit tests.
Run the updated tests and fix the errors by updating the TypeScript models.
Apply the code changes pointed out by the TypeScript compiler because of the updated models.
Commit the TypeScript code changes.
Pros
The commits in the code repository inform the developers which model definition unit tests changed, which helps them know what changed in the Contentful model.
Cons
The main drawback is that comparing Contentful and TypeScript models is a manual process. For larger applications with multiple environments, this method is insufficiently secure.
Another drawback is that the TypeScript code does not contain the full set of Contentful field properties. The example below shows that only model fields and the required attribute are tested for.
Example
The manual method depends on a set of unit tests that are kept in sync with Contentful. This test script tests for the correct model definition.
Testing for model definition is difficult because TypeScript types are not available at runtime. The test defines all required properties and then adds the optional properties individually.
2. Contentful migration scripts
This method keeps Contentful and TypeScript models in sync by running migrations from the TypeScript code base. A migration is a script that uses the Contentful migration library. This library allows for easily modifying content types, entries, fields, and tags.
The steps involved in this method are as follows:
Steps in Contentful
This method requires no steps in Contentful. Model changes are applied by running a migration script from the TypeScript code base.
Steps for the TypeScript code base
Write a migration script for the changed models.
Run the migration script. This changes the models in Contentful.
Update the TypeScript models.
Apply the code changes pointed out by the TypeScript compiler because of the updated models.
Commit the migration script together with the TypeScript code changes.
Pros
The code repository contains all migration scripts which show the changes over time for all Contentful model fields and attributes. This includes changes to, for example, the field help text. By analyzing the migration scripts, the dev team knows what changed in Contentful. This is why the migration scripts add a lot of clarity to the code base.
Another advantage of the migration scripts is that the scripts can be executed in multiple environments. This allows for securely keeping multiple environments in sync.
The scope of this post is models, but a migration script can be written for content as well.
Cons
Compared to the manual method, there is less need for comparing Contentful and TypeScript models. But when needed, this is a manual process. One still needs to write and manage the migration scripts.
The work of writing a migration script could be seen as a downside, but looking at the clarity the script provides, it is better defined as an advantage.
Example
Contentful migrations are well documented. This migration script is taken from the official tutorials and adds a category field to the blogPost model.
3. Modelberry Factory
This method keeps Contentful and TypeScript models in sync by adding Contentful specific TS Doc annotations to the TypeScript models.
The steps for updating a model using this method are as follows:
Steps for Contentful
This method requires no steps in Contentful. Model changes are applied by running Modelberry Factory.
Steps for the TypeScript code base
Update the TypeScript models and their TS Doc annotations.
Run Modelberry Factory. This changes the models at Contentful.
Apply the code changes pointed out by the TypeScript compiler because of the updated models.
Pros
This method does not require writing separate migration scripts. Instead, the full model definitions are in a single TypeScript file.
The commits in the code repository show which model definitions changed. The TS Doc annotations show changes for all Contentful model fields and attributes. This includes changes to, for example, the field help text." This tells the dev team what changed at Contentful.
Just like migration scripts, TS Doc annotations add a lot of clarity to the code base. Because TS Doc annotations do not contain any logic, annotations are easier to read than migration scripts.
The simplicity of the TS Doc tags allows for a digital comparison of Contentful and TypeScript models. Modelberry Factory has a ‘diff’ command that works much like the Unix diff tool. This is useful as a manual command, but also as part of an automated end-to-end test.
Besides pushing model definitions to Contentful, Modelberry Factory also allows for pulling models from Contentful into TypeScript source code.
The scope of this post is models, but just like the Contentful migration scripts, Modelberry Factory also can be used for pushing and pulling content.
Cons
This method depends on the specific @modelberry block tag with specific inline tags. For existing applications, this tag needs to be added in a comment to all TypeScript models.
Example
This example is taken from the Modelberry Factory website. It shows how ContentfulSomeModel is defined with a title field. Both model and field have various attributes.
A full example repository that pushes and pulls models and content can be found on the Modelberry Factory website.
Wrapping up
Application development has become more agile and by delivering an MVP first, we are able to adopt flexibility and ship faster. But this brings the challenge of maintaining data structures for large scale applications.
To avoid breaking the application based on a flexible data structure, we need:
models to be in sync across separate systems,
the developer team to be updated on all model changes, and
the code base to be updated when models change.
When syncing is simple, therefore, communication to the dev team is simple. The dev team can then focus on updating the code base.
Of these three, syncing the models is the key requirement, and Modelberry Factory helps with that. When using Modelberry Factory, you don't need to maintain a separate script!
