What we've learned from building our API

We have spent the past three years building the Contentful API – an API which is our core product, not just a spin-off project. (If you’ve never heard about Contentful, it’s an API-first content management system, consisting of an API to access the content and a web interface to create and manage it.) During this time we've learned a lot about designing, building and maintaining a public API. We'd like to share our learnings.

Treat an API as a product

First thing that we learned and which we’d like to say boldly and loudly is that an API directly affects developers’ experience of building an API-dependent project. As Matt Gemmell puts it, "APIs are UX for developers."

If someone relies on your API to build something, the API becomes an important tool. Just as any tool, It makes sense to make it great, so that users would love it and enjoy the experience of working with it. If we treat an API not as collections of objects and methods, but as a product which is used by people, then it makes us think about those people, their goals and use cases, and that way we can arrive to a better API.

Humans vs. machines

There is a funny trade-off in API design: 99,99% of the time APIs serve as a way for machines to interact with each other, so at the first glance it feels that it makes sense to optimize for performance. However, APIs are also experienced by humans, and they become recognized and widespread when they are convenient and enjoyable. To achieve that, one would want to optimize for usability. We aimed for the golden mean, balancing simplicity of use with performance efficiency.

Genesis

When we were just starting, the basics were not difficult to figure out: naturally, it's HTTP (what else?); certainly, it's JSON and RESTful – these are all de-facto industry standards. We were also considering JSON schema, which was under development back then, and HATEOAS, and borrowed some ideas and approaches from there. Once these choices were made, we started writing the code.

Fast-forward to the end of 2015: we have two core APIs – one read-only, one read/write (content delivery and content management, that is), millions API requests are processed daily, and it all feels quite nice. It feels like we've made a real product.

Things to think about when building an API

An API is a long-standing contract

Unlike modern apps which live by the "continuous improvement" mantra – deploying new versions every day and unable to imagine life without A/B testing – API vendors aren't lucky enough to enjoy that degree of freedom. Once an API is released, it's next to impossible to remove and change things.

Providing a public API comes with a huge responsibility. There’s little room for experiments. A lot of thought has to be put into every feature. Everything has to be consistent and make sense from v1, both within the feature context, and in correlation to the entire product. It's difficult, but it's so worth it. (You don’t want to maintain several versions of an API – it’s immensely complex and expensive.)

There might be just one user, let's call her Barbara, who relies on one particular feature – but if you decide that the feature is done wrong and needs to be changed or removed, Barbara will be very unhappy when you do that without telling her and make her app crash. Not only Barbara – the users of that app will also be disappointed, not to mention Barbara's manager. They all will have to apologize before their users for something that's not even their fault.

It's best to avoid making all these people unhappy. (Especially Barbara.) Lesson learned: every API change needs to be communicated in advance, so that people would have enough time to prepare for the update.

It's fascinating that what seems like just a minor product improvement on the surface has deep consequences in terms of human reactions.

Naming matters, because the titles stay forever

It's worth taking the time to come up with naming conventions and adhere to them to ensure overall naming consistency.

This is related to the previous thought: after the release methods stay forever, and so do their names. "Let's just quickly implement this" approach can quickly lead to chaos.

Admittedly, coming up with the best names is not easy. For example, for us it was quite a challenge to select just the right name for collections of content. Projects? Workspaces? Folders? Collections? There was a lot of healthy heated debate. Each word triggers certain associations, and probably every person would interpret it somewhat differently.

Initially we named them "buckets", following the Amazon S3 buckets, but later decided to rename them to "spaces" for more clarity – not everyone might know what a bucket is.

We also borrowed a lot of terminology from the content strategy industry, which is interesting: although the product as a whole has a lot to do with content and therefore has to speak the language of the industry – content strategists, producers, editors, writers and the like – it also needs to be understood by developers. Having these two distinct user groups raises some interesting challenges on its own.

Try the API yourself before releasing it to the public

The API owner should be the first to try their own API.

Here are the things we’ve done in that direction: after initial basic Hello World apps we’ve built the web interface entirely on top of our own API, and then created a set of SDKs, which is an essential and ongoing effort for a public API. Read more about both in the sections below.

Provide SDKs

Developers can afford to be lazy. You want them to be able to start working with the API as soon as possible. To simplify their first steps, give them an SDK so that they can skip writing boring boilerplate code part.

It also perfectly resonates with testing your API: building an SDK is a perfect test. If the SDK developers struggle with building it, that indicates problems worth investigating and solving.

We’ve built SDKs for JavaScript, Objective-C, Ruby and Java, and PHP is currently in beta. Undoubtedly, it’s a lot of work, but since it helps every new developer to get started faster, it’s worth the effort.

Write good docs and keep them up-to-date

If we keep the “API is a developer UX” analogy, then it’s safe to compare the API docs to a user interface.

Most likely a developer will take a look at the docs before playing with the API – not after. It takes less time and helps evaluating the API before committing to using it for a real project. If the docs are badly written, poorly structured, out of date or lack some essential information, then chances are that I'll run into troubles when solving a real problem.

Overall, documentation is underestimated. Just like a product, docs need to be properly designed to work for different audiences and serve their goals.

For example, just publishing an API reference is not enough. A reference is fine when you are already familiar with the product and just want to quickly check some specific details – it's okay when you already know what you're looking for. However, you don't want to learn the product by reading the reference. You would like to learn the basic concepts, see best practices, read tutorials, etc.

On a similar note, good API docs can become your content marketing – just the same way a well-designed user interface promotes a product.

Test it carefully

As any software product, an API must be tested to ensure that it works as expected and to enable identifying regressions when changes are made.

APIs bring several particular challenges to the topic of testing, since software that relies on an API might fail because of unexpected changes of the API itself, but also because of changes deriving from the server setup, eventual proxies, CDN networks configuration and so on.

For those reasons in addition to the usual unit and integration tests of our codebase we wrote an entire test suite that accesses the API as a regular client would do via HTTP, and verifying the complete behavior including certain aspects such as HTTP headers.

Since the server setup determines the API behavior, we are using Runscope to continuously test our staging and production environments. It allows us to immediately be notified if any of these environments deviates from the correct behavior.

Design of an API-based product heavily relies on the API design

The Contentful web app is built on top of our API. (That relates to the idea of extensively using your API yourself.) What is interesting is that the design team behind the app can't work independently: certain design changes and improvements are closely connected to the API, to a point when new features on the web app side demand changes in the API design.

In a way, this is just like customer–vendor relationship, with the difference that the customer – the web app design team – is sitting right next to you at lunch. This relationship pushes both teams to collaborate smoother and communicate better, so that every party is aware what the others are doing and everyone is constantly in sync. It also helps understand our customers and their struggles better, since the web app team uses relies on the same API that our external clients are using.

We found that even though API is initially built by engineers, for engineers, involving design thinking and processes can be very helpful, as it helps making informed, human-oriented and research-driven decisions.

Make it fool-proof

It's best to prevent damage and make it a bit more difficult for people to shoot themselves in the foot.

If a user makes a potentially harmful API request and accidentally deletes all their data, you'll have to spend some time to help them recover it. While helping someone is always a good thing, maybe it's even better to help before the damage happens, not after – for instance, by making it technically impossible to delete a big portion of data with a single HTTP call.

Here's an example of this damage prevention in Contentful: you can't delete all the /entries/ in a space. The API will reject this request. You can only delete the entries by deleting /entry/<id>/ one by one. While that is a bit more work for the developer and more requests for the server to process, this task is easily automated, and the payoff, as we hope, is fully justified.

What's next?

These are our learnings. We'd love to hear yours, too.

Here's a question we’ve been considering recently: how to tell a good API from a bad one? Tweet us what you think. We're listening.

Further reading

If you'd like to read more about API design, take a look at these wonderful resources.

• REST APIs must be hypertext-driven
  by Roy T. Fielding, creator of REST and co-author of HTTP standard

• How to Design a Good API and Why it Matters by Joshua Bloch, Principal Software Engineer at Google

• Thoughts on RESTful API Design – Lessons learnt from designing the Red Hat Enterprise Virtualization API by Geert Jansen

• HTTP API Design Guide by the authors of the Heroku API. Quote: "We’re looking for a good, consistent, well-documented way to design APIs, not necessarily the only/ideal way."

• API design for iOS apps components by Matt Gemmell. Quote: "APIs are UX for developers."

• Best Practices for Designing a Pragmatic RESTful API
  by Vinay Sahni, the founder of Enchant. Quote: "An API is a user interface for a developer - so put some effort into making it pleasant."