Engineering at Contentful - Part 2

This post is part two of a series on Engineering at Contentful. The first part highlighted how we created our first engineering team.

In this post, we cover how our team structures have evolved over the last year, what we still struggle with and in which direction things could go in the future.

Around mid-2016 we had four development teams consisting of around 25 developers. Each had a product owner, and the teams that delivered UI-based features also had a designer. Their names and missions were:

  • Pikachu - Authoring Experience and Content APIs.
  • Unicorn - Billing, Onboarding/Signup and User/Org Management.
  • Giant Sloth - Infrastructure, Internal Productivity and Tooling
  • Catshark - SDKs, Example Applications and Developer Evangelism

While the engineering group had grown considerably over previous years, there was still quite a lot for them to tackle and we had discrepancies in team size: some were only three to four engineers, some over ten. We also had a couple of “floating” singleton teams of engineers responsible for things like Security or Data Processing.

Decision 1: Meiosis or Mitosis?

If you are unfamiliar with the terms, Meiosis and Mitosis describe a couple of different ways biological cells can divide. Similarly, when you want to turn a single team into multiple teams, there are certain decisions you need to make. Do you create a whole “copy” of that team on paper and then staff it up gradually, leaving the original team intact? Or, do you split the existing team in half, creating two small teams that still need to grow a lot to survive?

Learnings

For us the answer has generally been the latter. We have consciously avoided teams that exist on paper only, and teams that are just a couple of engineers with a monstrous backlog. The best policy according to our experiences is to let the original team grow to an almost uncomfortable size, and then split the team into two smaller teams. In doing so we learned the following:

  • Focus for the team will become more difficult, as the size grows and you can tackle more independent streams of work. Especially early in the company journey with broad team missions, in a given sprint you might find several goals and an increasing difficulty to complete them.
  • Some roles don’t naturally scale. One example could be the designer, but more concretely the Product Owner is indivisible. The Product Owner will need to split their time between the two new teams until another product owner can be hired.
  • Physical space. It can be difficult to find a room large enough to hold twelve or more people and still have a productive work environment. Once the team is split, you need to find a second room - this is important because the new teams have to find their own identities. We have not yet considered “co-located” teams, and don’t expect to do this for these reasons above.

Pikachu was the first team to be split in December 2016. The team had a mix of goals around our Authoring experience in the web application and our Platform APIs, and both areas had overlapping technical concerns between front- and back-end work.

The result was one team focusing on Authoring and the other focusing on the Platform APIs, with a single Product Owner sharing responsibilities for both teams.

Decision 2: How to create entirely new teams?

In Q3 of 2016 we started to work on the onboarding experience — that is, how new users are introduced to our product. Because a lot of these responsibilities across multiple teams, it is easy to forget about this crucial aspect of the user experience.

In November the decision was made to form a specific team and initiative around the onboarding experience. It would require a focus on experimentation, fast turnaround times on ideas and learning — and most importantly, making changes to both the UI and backend systems.

But, we didn’t have time to recruit and hire. We decided to start the team with a mixture of seasoned members of the organization together with a few new starters who had not yet become completely attached to the team they were onboarding with:

  • Front-End Engineer from “Unicorn”
  • Back-End Engineer from “Pikachu”
  • Data Infrastructure Engineer
  • New Data Analyst
  • New Designer
  • New Technical Writer
  • New Engineering Manager serving as Technical Product Owner

Fortunately we had several new starters who were keen to tackle the new initiative. We also had an Engineering Manager with a background in rapid iteration and A/B testing willing to cameo as the Product Owner on a temporary basis.

Learnings

Fast Feedback

Fast feedback is critical for rapid iteration — having a data engineer and analyst embedded in the team can be incredibly powerful towards building pipelines and tools for self-service analysis, reserving the manual analysis for more complex questions. The team couldn’t have iterated as fast on ideas without having these roles inside the team.

Split-Focus

However, having such a dynamic mix of skills and competencies in a single team led to multiple independent tracks of work which occasionally were counterproductive. According to Scrum principles, the team should aim to cross-skill and have all members contributing to all topics, but with so many tracks and skills this is a big challenge.

Conflict of Interest

Due to the lack of available Product Owners, we had an Engineering Manager doing double duty as PO and manager. Therefore, all of the engineers in this team were reporting both to the PO and Engineering Manager. In a regular team, there is a healthy balance created by having both product and technology interests represented by two distinct yet equally important roles — when these two needs are represented by the same person we remove that healthy tension, and potentially make decisions that aren’t as strong or balanced.

Product Management is Hard

Being a Product Owner / Manager is a tough job and it was a humbling experience that provided the Engineering Manager with more empathy for our Product colleagues. This experience also gave valuable insights into our Product process and helped to accelerate the understanding of what we do well and where we can improve. While this kind of cross-department cooperation is great to foster, it also stressed the breadth of both roles - something which is a challenge to manage by an individual tasked with both.

Decision 3: Dynamic Teams, Dynamic Projects (or “Gangs of Berlin”)

The start of 2017 was the right time for a revision of our product strategy. We focused on renewing the Developer Experience, as well as identifying clearer missions around a streamlined experience for larger Enterprises. We continued our investment around the onboarding flow, as well as formally acknowledging our distinct requirements for Reliability of the platform, and Security.

Furthermore, we overhauled our product planning process and enabled teams to pitch their own ideas for which projects to work on. Many potential projects were around the Developer Experience area. Most of them were focused on quick wins with maximum impact — taking no more than 6-8 weeks. At hand to tackle them were around 20 engineers in a group that we hadn’t formally split into teams.

We felt that we had overloaded the term “Team”, so a new term “Gangs” was used to differentiate it from our longer-term team structures. The idea was that each initiative would be worked on by the engineers best suited to the task. Once the initiative was completed, the Gang would disband, and a new Gang would form around the next initiative in the plan. It’s a concept quite similar to self-selecting teams, a topic which has gained a lot of interest in recent years.

The concept was quite a radical departure from our previously more structured approach, but seemed on paper to solve the problems we were facing. More importantly, we wanted to genuinely experiment and try a new solution - even if we couldn’t predict the results accurately.

Learnings

On paper, this idea seemed to solve the problem of dynamically tackling a broad product initiative with a large group of engineers and designers. We continued operating in this way for several months at the beginning of 2017 and did complete some important projects during this time: in that sense, we were certainly successful by one measure. We were able to form task forces around the initiatives, brainstorm and plan the idea to solve the problem at hand, and work it through to completion. However, there were still some pain points.

Synchronisation and Certainty

Since at any point in time we might have two or three gangs operating, each working on different projects, they would not naturally complete their work at the same time. If we needed five people for the next initiative, but two were still busy with another initiative for the next sprint, the remaining three would be “idle”. In practice we found technical debt or other improvements to work on in the meantime, but it increased the level of uncertainty for engineers.

We would find people asking questions like “Who am I working with next?”, “Which project is coming up next?” and so on. Having a level of certainty is important for people to feel safe and secure in their working environment, and this was not easy to solve with the Gangs setup.

Team Dynamics

Each new gang that formed was a brand new team. Even though the engineering organization at that point was less than 40 people in total, it is a different experience working with someone closely in the same team to just seeing them in the kitchen and socially around the office.

The Tuckman model of team formation tells us that in the early days of a team, it goes through the “Forming” and “Storming” phases. These are periods during which the team identifies roles, personalities, ways of working and how to deal with conflicts. Gangs that would only exist for 6-8 weeks barely have enough time to get a sense of their surroundings before the project is completed - there is no hope of getting to a streamlined “Performing” phase.

During late 2016 and early 2017 we became aware of our need for more discipline and knowledge around building an Agile organization. Software delivery teams at Contentful generally use the Scrum framework. Ways of working and how Scrum is implemented is one of the aspects of forming a team and again, trying to institute and optimise the Scrum workflow was just not possible within a 6-8 week period. We would only just be starting to identify concrete areas for improvement before the Gang disbanded and reformed with new members.

Visibility and Support

At Contentful we organize engineering structurally as a matrix - cross-functional teams consisting of members across different roles/technologies and management and reporting lines on the technology axis. As an engineering manager, you might have engineers reporting to you who are in several different teams. Now that we added Gangs to the mix, not only was the spread of teams potentially broad, but they changed frequently.

Taking a stake in supporting engineers in achieving their delivery goals, it was hard to know which Gangs needed assistance, whether there were impediments to their success that had become critical, or whether we needed to set up discussions on specific topics to move progress along.

Other departments (like Sales and Marketing) who rely on knowing what is “in the pipeline” so that they can promote new functionality and cater to our customers better, would feel somewhat unclear on the progress. Previously it was easier to make connections between individuals, teams and departments — now it was harder.

Organisational Maturity

One piece of the puzzle when it comes to self-organizing teams (also made very popular by Valve) is the maturity of individuals and the organization. In Valve’s case, they only hire “cream of the crop” engineers who already have a strong sense of direction and understanding of what is valuable to work on. At Contentful we want to boost our diversity and capabilities by having a broad range of experience and skills - from junior to senior members. We also don’t want to leave people in the dark, but provide clarity and certainty with our product strategy. We shouldn’t tell people what to do, but share a compelling and comprehensive vision and strategy for which problems we want to solve for our customers.

We recognized after running with Gangs for several months that we didn’t yet have the organizational maturity to support such dynamic structures. Indeed, we needed to amplify our product strategy a lot more so that everyone understood what they were doing, and why. This is a challenge and a struggle even without dynamic teams!

Ultimately, while we accomplished much during this period, we also learned a lot of the drawbacks and decided we should invest in longer-running teams, and better communication and sharing of our product strategy. So in March and April 2017 we brought in Agile consultants to help us understand where to focus our efforts. Their feedback highlighted our suspicions around dynamic teams and product strategy. We don’t consider it a failure, but a valuable learning experience and worthwhile experiment to have conducted!

Decision 4: Full Matrix or Traditional Hierarchy?

We have a matrix-style structure in engineering relating to our cross-functional teams and reporting lines. We’ve been operating like this under several assumptions:

  • When engineers need to change teams, we want to minimize change and confusion by keeping reporting lines the same.
  • We want to focus efforts on our core technology areas, promoting learning, sharing of knowledge and insights - thus creating Chapters which are led by engineering managers.
  • Responsibilities around feature requirements, delivery and technology are co-owned by the Product Owners and Team Leads in each team respectively.

With a growing engineering management team, we gained more insight into the experiences of engineers and what their concerns were. However, with engineers spread across multiple teams it was difficult to understand the activities in each team. Often we would be sitting with just one team during a typical week, and if another team was struggling with a particular topic you might only find out after the fact, in a 1:1 with a team member.

Learnings

Visibility and Support

It is incredible how much visibility and insights you miss out on when not sitting in the same room as your engineers. Conversations, decisions and challenges are often communicated verbally, and not being present reduces your visibility of them as a manager.

That is why we have decided to tackle this problem by assigning engineering managers to teams to have a more explicit relationship for supporting those teams. Since we are also at a stage where we have sufficient managers to do this, we have usually one or two teams per manager. The goal is to have the manager sit permanently with at least one team, and try to attend as many Scrum rituals as possible - as well as product “pitches”, architectural discussions and so forth.

Being physically present with the team is expected to allow us to understand their needs better, provide better suggestions and feedback regarding their workflow improvements, and ask the right questions about technology implications of work. Where we have direct reports in the team, we hope to gain a better understanding of their daily challenges, learning targets and how we can assist them directly.

Roles and Responsibilities

The official Scrum guide states that cross-functional teams have no “lead” or special roles - that the team is entirely self-sufficient and all members can perform all roles. But, that is easier said than done. So at Contentful we elected to have Team Leads in each team as a natural counterpart to the Product Owner — someone who can act as a first point of contact for technical questions, coordinate work in the team and lead Scrum rituals.

We have found that some responsibilities like delivery, workflow improvement, team health, etc., don’t naturally sit with either the Team Lead or the Product Owner. We’ve hesitated to make either one of these people ultimately accountable for some areas of the team’s processes and this has in some cases led to ambiguity and grey areas. While we don’t necessarily think the engineering manager should be the ultimate owner of these topics (although we might decide this in future) we want to have someone there to highlight concerns (or escalate if necessary) when one of these areas is threatened.

In other words, if we sense delivery is threatened, it’s time to huddle with the Team Lead and Product Owner and gain an understanding of how we as engineering managers can help to resolve this for the team. There are frameworks and models for dealing with these kinds of ambiguities like RACI or DRI but these usually imply a concrete assignment of roles. We are still assessing and learning, and want to evolve our roles further before necessarily drawing that line in the sand. There is, after all, a lot to be said for influence, empathy and support.

The future is now

As we wrap up the last quarter of 2017, we are yet again discussing our product priorities and what organizational structure best fits with how we want to deliver. We are considering all of our learnings gained over the last year and what other companies in our field are doing. There are many interesting possibilities from our variation on the Spotify/Matrix model, traditional hierarchies, Zalando’s Radical Agility, Holacracy as implemented at Zappos, the aforementioned self-selecting teams and many others that we have all experienced at former employers.

We can’t say exactly what things will look like in a few months, but we will approach it with open minds and with the intent to learn - hopefully learnings we can share with you once again in the future!

Join our team

We're looking for smart, enthusiastic, and skilled people to help us build world-class content infrastructure and delight our customers. Check out our open positions and apply today

Blog posts in your inbox

Subscribe to receive most important updates. We send emails once a month.