Engineering at Contentful - Part 1

Engineeringhistory

This post is part one of a series on Engineering at Contentful. We hope to set the scene in this first post with how engineering has evolved from the earliest days—in many ways, typical of a technology startup—through to the current day with a company of over 100 people.

Founding and First Team

Contentful was founded in 2012 by Sascha Konietzke and Paolo Negri (both software developers), growing out of an existing product called StorageRoom which Sascha had created a few years before. The scene was already set for a technology-oriented company due to the technical background of both of the founders. As is very common in the early days of a startup, everything to do with workflow, coordination, what to work on, etc., is vastly simpler than at larger companies—despite being able to fit the whole team around a single table. You can come up with an idea in the morning, brainstorm over lunch, implement in the afternoon, and then have the finished feature by the end of the day!

Sascha and Paolo came from varied technical backgrounds, covering iOS, Ruby and Rails, Erlang and Javascript.

In these early days, both Sascha and Paolo were wearing many hats, working on the product but also talking with customers and investors, trying to raise the visibility of the product and secure its place in the market. Sascha and Paolo came from varied technical backgrounds, covering iOS, Ruby and Rails, Erlang and Javascript. Indeed, many of the initial services were implemented in a variety of these languages, taking advantage of the founders’ strengths and the technological benefits of each given platform.

It was effortless to have impact at this stage, as the codebases were relatively small and all of the context of the whole company—including how the technology worked—was contained in the founders’ heads. Communication and collaboration suffered few overheads, and the biggest problem was finding a market and gaining traction in it.

More Engineers, More Problems

Size: Founders -> 4 people (January 2013)
Communication: 1 to 1 -> collaborative, face to face, informal
Responsibilities: everyone does everything -> each person wears many hats

Seed funding for the company during 2012 brought about the ability to grow the team and the product further. At this size you can still comfortably fit around a single table (well, maybe a large one), yell ideas across the room and enjoy relatively low communication/collaboration overheads. But now, it was not necessarily desirable that everybody is working on the same thing—there were some people working on the front-end authoring experience, and others (including the founders) working on aspects of the backend and infrastructure.

Roles (and technologies) became more specialised. More of the services conglomerated around our current core technologies of Rails and NodeJS, and we had some engineers working specifically on systems relating to accounts, organizations, billing, pricing, while others worked on general infrastructure and content delivery. Some services that didn’t neatly fit into these two technologies were still occasionally in Erlang where it made sense. The founders needed more time to spend on more strategic topics of building up the company and the vision for the future of the product, but now we had more engineers to free them up.

A lot of the workflow was still relatively ad-hoc, testing was best-effort as was internal documentation

A lot of the workflow was still relatively ad-hoc, testing was best-effort as was internal documentation (although we obviously needed to have solid user-facing documentation so that the product was usable). We tried to move as fast as possible and experiment with new product features rapidly, as the end product goal was not yet completely clear. Architectural decisions had to be made in an afternoon instead of a month.

Monitoring and metrics—the ability to understand how and when our user-base was using our product—were in their early stages. Likewise, we lacked the ability to analyze them deeper and drive our product process by KPIs. A lot of the decisions were based on customer conversations and gut feel. Many of the graphs we were looking at were steeply climbing curves (the famous “hockey stick”)—usage growth was extreme.

We have been cloud-native since day one, so this afforded us the ability to move quickly and not worry about provisioning and maintaining our own hardware.

We have been cloud-native since day one, so this afforded us the ability to move quickly and not worry about provisioning and maintaining our own hardware. This came with a lot of benefits, but also led to a lot of 3rd party tooling being required. Economies of scale and consolidation came second to moving fast and gaining product traction. Patterns and best practices were yet to emerge and be taken advantage of.

More Streams of Work

A huge year of growth in 2013 led to Series A funding at the end of the year. This unlocked further possibilities of growing both the team and product from the start of 2014. The team of ten now had to grow, but we had to find the right way of doing it. Furthermore, what had already been a very engineering-heavy team now had to diversify into a company that also contained sales, marketing, business-intelligence, operations, customer support and other roles to support the organization.

Artas joined as the second product manager of the team at the beginning of 2014. César (our first product manager) out of necessity had to take on multiple types of work, and was designing for the parts of the product that were largely to do with the front-end authoring experience.

Andy came on-board to take on the Rails work that previously had been keeping Sascha busy. We already had several independent work streams, which required more concurrent initiative management, even if the engineering “team” was not yet split apart. In terms of things like Scrum workflow, this makes it much harder to say you have a single concrete goal for the team at the end of the sprint. In practice, much like many other startups at this size, the workflow was some kind of “ordered chaos”.

This is not very typical of a small startup and certainly added further pressures to the engineering team.

Since ease of integration and involvement with the ecosystem were priorities from very early on, we already had a public SDK for Javascript. Objective-C, Ruby, and Java/Android followed shortly after. Some were already taking contributions from the community from 2013, which added to the strength of these SDKs but also added the burden of reviewing submissions and fielding bug requests from the public. This is not very typical of a small startup and certainly added further pressures to the engineering team.

The company had more than 20 people by the end of 2014, at which point there were already de-facto teams specifically dealing with the front-end authoring experience, content delivery and accounts/billing concerns as three separate streams of work. We didn’t necessarily all fit around the same cluster of tables anymore, but standups could still happen between all of the developers, and we would all fit in the same open-space area, in what is still part of our office today. Engineering meetings could still fit into one large meeting room.

Introduction of Actual Teams

Size: 7 (January 2014) -> 25 (January 2015)
Communication: collaborative -> team representatives, written decisions, electronic chat
Responsibilities: shared -> specific teams and roles

2015 saw continued growth in the engineering team, although everyone in the company fit on a single floor in the building. An emphasis was placed on working quietly, since it was one big open space shared by everyone. Roles continued to diversify with infrastructure engineers and QA being added around this time as well. At least early in the year, many team rituals were still shared across engineering like standups (conducted virtually with a Heroku app) and retrospectives. By March, we were starting to have retrospectives for each “platform” team—frontend, backend, and ecosystem—usually as each team got to a critical mass of around three or more people. Some teams (e.g. Infrastructure) later in the year grew informally out of individuals who shared a common interest around a topic. From platform teams, we also rapidly evolved into more “topic” areas like authoring and content modelling: cross-functional teams.

Development processes were being continually refined, initially across all of engineering, and later in the individual teams to meet their own needs.

Development processes were being continually refined, initially across all of engineering, and later in the individual teams to meet their own needs. Engineers started becoming more aware of cross-team dependencies and the need to include these in their planning processes. Burdens of development involving many people were being felt, such as large, slow integration testing suites, waiting for someone to pick up pull requests for review, and siloed knowledge.

Some of the first big projects around strategic technical debt management were being kicked-off. We had enough people to support these maintenance efforts, but it was still a challenge to fit them in with the need to continue adding value to the product.

If you look at meeting notes from around this period—especially team retrospectives—you would notice a lot of pain was being felt as a result of the higher coordination costs. Everyone wanted to move just as fast as they did in the past. With more people working at the company (and indeed on the product itself) communication points were being missed, confusion was arising as intentions were not being clarified early enough, and frustrations were rising. Retrospective action points were many and varied. A lot of the things that used to be implicit had to start being made explicit with more concrete processes and ways of working together. Feedback, communication expectations, setting out the workflow process, documentation and adequate planning all featured heavily in what needed to be improved. Institutional knowledge failed to be captured or preserved as the company grew, and incomplete implementations of Agile workflows emerged.

Towards the end of 2015, the company outgrew the single floor that was holding everyone, and engineering teams moved to the floor below, each team with their own room.

Towards the end of 2015, the company outgrew the single floor that was holding everyone, and engineering teams moved to the floor below, each team with their own room. This was the first quarter where we had a roadmap focussed on themes, and with teams named after the theme they tackled (for example Control, Developer Experience, Ecosystem and Platform). This was an explicit pivot away from teams oriented more around technology, to being oriented around aspects of our product and how our customers interacted with it. This is an essential move in the growth of a startup as you outgrow how many people you can have working on any individual component.

This didn’t come without risks of course (for example, potential lack of designers, lack of QA testers, overlap in scope of teams or dependencies on other business units) but we felt that it was a change that needed to be made and this new configuration offered more benefits than drawbacks. The gap between these two sides of the equation are the learnings that a company can make around organizational structure.

Further Growth and Introduction of Engineering Management

Size: 25 (Jan 2015) -> 51 (Jan 2016) -> 71 (June 2016)
Communications: mixed -> reliance on electronic forms as we split office floors
Responsibilities: specific teams & roles -> further specialisations, more teams

Every startup is subtly different, but many hit this kind of scaling challenge - either the CTO, or a VP of Engineering or even Engineering Manager will at some point find themselves with a huge number of direct reports . . .

Early in 2016, the Series B funding round closed, unlocking further room for growth in the team. Paolo at this point had around 25 engineers directly reporting to him and suffering from scheduling overload. Every startup is subtly different, but many hit this kind of scaling challenge—either the CTO, or a VP of Engineering or even Engineering Manager will at some point find themselves with a huge number of direct reports and find it a challenge to cover everything that is required of them. Since there was now a big push to grow the product (and hence the company) more, Engineering Management had to be introduced. Since July 2016, this layer of the organisation has grown from one to four people reporting to Paolo and gradually taking more of the individual contributor burden away from him.

Around this time (mid-2016) there were four teams (4-7 people each) whimsically named after animals: Pikachu, Unicorn, Giant Sloth and Catshark. We failed to specify that they needed to be real (and living) animals, so the lack of rules was exploited in typical engineering style. Each team had their own mission aligned around APIs and authoring experience, enterprise features and accounts, infrastructure and developer productivity, and finally ecosystem and SDKs. Teams consisted of a Product Owner, Designer (if the team was producing front-end features) and engineers with the right skill mix to independently ship features.

As the teams grew organically, we found some that had a reasonably narrow and well-defined scope but others (especially one team that had grown a lot in size) had a very broad scope, with many concurrent initiatives and workstreams. Providing visibility into what was going on in the teams at any given time became the new challenge.

This made it relatively easy for engineers to move around to different teams and indeed, that has happened in the past. Therefore when introducing a management layer to the organisational structure, it was desired to facilitate this kind of team flexibility and ability to move around as needed/desired. The so-called Spotify Model (although it has since evolved and changed) was chosen to promote these aspects.

We generally call our teams “Teams” and not “Squads”, but the shape is the same. We do have “Chapters” which are collectives of engineers from the same discipline (i.e. Backend Ruby Engineers, Backend Javascript Engineers, Frontend Engineers, etc.)—these are groups focussed on sharing knowledge between teams, and surfacing technical debt or other areas of shared concern.

Managers are now present to help with things like career growth, team workflows, organisational structure, technology strategy, and many other areas.

Managers are now present to help with things like career growth, team workflows, organizational structure, technology strategy, and many other areas. We can start to put in place some of the processes, structures, and help direct the culture in ways that facilitate further growth of our engineering organisation. How we build Contentful the product itself, is heavily dependent on all of these elements. How this has developed over the last year and continues to evolve is the subject of further blog posts in future.

Blog posts in your inbox

Subscribe to receive most important updates. We send emails once a month.