Sharing data across services – connecting all the serverless dots

Today, very specialized services define the market to prevent developers from reinventing the wheel helping them to ship software faster. Sign up for a service, use their product (most commonly including some API) and enjoy having one problem less concerning development, maintenance and scalability.

Even though software-as-a-service providers usually cover only a specific area in your product, at some point, one service has to be connected to another one because of shared data.

Let’s take Algolia as an example: Algolia specializes in providing a fast search experience. You define what will be searchable and your content could come from everywhere – your database, JSON files or another cloud service like Contentful.

In this article, you’ll learn about what it takes to connect services like Algolia and Contentful directly via webhooks, but it will also show how and why function-as-a-service (aka "serverless functions") will become the perfect glue for modern software stacks.

Sounds good? Let’s jump right in!

Webhooks – the connection of cloud services

One way to connect different services is by setting up webhooks. A webhook is an HTTP request that goes from one service to another, triggered by a specific action – most commonly data creation or update. Phil Hawksworth once called them the "API of the web", and I think that’s pretty accurate.

Unfortunately, the format of webhooks and their payloads can vary. There is no clear standard on how a webhook should look like yet and you cannot always connect one service with another directly via webhooks.

Let’s take Algolia as an example: to push data into Algolia, you need to set specific headers (X-Algolia-Application-ID and X-Algolia-API-Key) in your requests to be allowed to do write-operations. Unfortunately, not every webhook-sending service lets you define or alter the outgoing request headers which makes it impossible to connect particular services.

Fortunately, Contentful as a content infrastructure supports custom webhook headers. Using them, you can establish a direct connection between Contentful and Algolia.

Services in your stack should allow you to set custom public and secret headers.

You’d quickly find out, though, that way too much data is indexed in Algolia. Contentful webhooks include complete content entries which then go directly into Algolia – including metadata and non-relevant information.

What you need is a way to transform the webhook payload and send it to Algolia. "That’s a use case for a serverless function" I hear you say? Yeah maybe... let’s have a look at how this could look like.

"Just write a serverless function for it."

To connect the two services, you can write a function that transforms the webhook JSON payload and forwards it to Algolia to be indexed. What you see below is a JavaScript function that I put together in Webtask. Webtask allows you to write serverless functions without any additional setup.

With this function in the middle, you can now transform the webhook data coming in and index a cleaned-up version in Algolia.

Problem solved – great! Well… almost… it’s actually not that great.

Configurable webhooks – connectors of your service stack

Since I discovered cloud functions, I’ve been following the mantra of "Push functions everywhere", and it’s been working well. This approach brings another burden though.

Cloud functions by themselves don’t require much maintenance regarding operations and setup, but I still have to document code, remember where I deployed functions, and I have to keep my co-workers up to date on how I connected particular services.

A better way to solve this problem is by providing service users with a way to modify and define not only headers but also the complete webhook payload. In Contentful, you can use JavaScript Object Notation pointers (or, in short, JSON pointers) which give you enough flexibility to get rid of metadata and thus the serverless function sitting between your services and Algolia.

The following is a slimmed down default payload Contentful sends when updating an entry. It includes localization information and lots of metadata.

You can now apply a transformation via JSON pointers by accessing the initial payload...

...which leads to the following payload that can be consumed by Algolia right away!

The definition of a webhook transformation has access to a payload object, and you can retrieve the values of this object with the { /payload/… } syntax. This way you can define properties and their values and can cut out the need for a transformation function running in the cloud.

A transformation before the webhook is sent leads to even less complexity than a serverless function and fewer things to remember and document. This possibility is excellent for everyday use cases. If you’re dealing with more complex data structures, a serverless function may still be the way to go, though. You can find an in-depth tutorial about Algolia and Contentful in the Contentful documentation.

Connect all the dots – the power of the service-mesh

Looking at today, the software-as-a-service-driven world we live in expects us, developers, to connect all these services quickly. My advice: when you pick your next cloud service don’t only look at the main features but also evaluate how easy it will be to connect it with your ecosystem.

Ask yourself:
Can you connect services directly to each other?
Will your service be able to authenticate?
Who can access which information?

And if a direct connection is not possible:

How much logic has to go into a serverless function to build the connector?

Services that are not able to talk to each other won’t speed up your development cycles in the long run, but rather become a burden. On the other hand, if your services work well with each other, you can enjoy the new shiny world we live in where it really becomes "just connecting the dots" to build something great quickly.