Since its inception, our Content Management API could only import assets that were hosted on publicly available URLs. Our reasoning behind this choice was that most of them are already available online, and our Web App has a Filestack implementation to circumvent this shortcoming. But today this changes, as we're excited to announce the immediate availability of a new feature, direct file uploads!
With the growth of our customer base, we saw a growing demand for more complex workflows around asset management. Data migration from old systems was a common pain point for our users, as our implementation proposal required them to rely on yet another service on which to host their assets. In our quest for an excellent developer experience, this was something we wanted to tackle, and therefore we started tracking and analyzing current usage. We quickly confirmed our hypothesis: around 40% of all asset uploads come from users working directly with our CMA and hosting assets on third-party services.
Unsatisfied with the current situation, we decided to build this feature right into the platform, to make it even easier to use Contentful in a future-proof way for all your content management needs — no matter if that content is pure text or binary data like images, videos or documents. The team did a fantastic job, and it took only two weeks from idea to prototype, and another two weeks to release it for you.
Before talking about the technical details of the solution, let’s take a look at how easy it is to use this new service in your known Contentful workflow.
To upload a file, just send a POST request with a binary file in the request body to upload.contentful.com.
Now you can create an asset as you normally would do, but instead of passing an external URL to the
upload field, you are feeding to the CMA a link object with the ID of the newly introduced
uploadFrom field to it.
After the asset's creation, the workflow will be the same as ever, so if you're using the CMA directly remember to process and then publish it!
Another tool we just updated to support this new feature is our excellent .NET SDK. Here's a quick look at this new implementation:
Of course, if you are already hosting your assets somewhere public, you might want to still use the old CMA workflow. That's perfectly fine! The new direct file upload system is just a more convenient way, and it will live alongside the previous implementation, so as to satisfy everyone's needs.
For all the details on how to use Contentful to work with assets, our Content Management API Documentation covers everything you need to know.
After the problem space was clear and we knew what we wanted to build, we started to research possible solutions. First we looked into tus.io and Amazon S3’s Pre-Signed URLs feature. The S3 way sounded nice, as especially with Transfer Acceleration, you can achieve very fast upload rates. However, we were forced to drop it because of lackluster developer experience: all AWS APIs are completely different from ours, as they use XML instead of JSON and they provide rather unintuitive error messages. The other alternative, Tus.io, is an open source protocol for resumable uploads, and has plenty of features. Unfortunately, after we analyzed usage patterns related to assets, we realized that we would have needed only a tiny fraction of what tus.io’s is capable of, but we would have had to deal with the overhead of running it in our infrastructure. After some investigation, we decided against both solutions and rolled our own microservice. In general, we try hard not to reinvent the wheel, as we'd much rather stand on the shoulders of giants, so to speak. However, we realized that in this situation the benefits of a custom solution far outweighed its costs.
The service would live in our infrastructure, but we needed to completely separate it from any other systems that we have, as handling long running requests with large payloads is fundamentally different from the fast and small JSON requests/response cycle we normally optimize for. With this setup, we were free to finely tune the new endpoints for handling streaming of binary data, longer timeouts, increased buffer sizes and payload limits.
Another upside of this setup was that we stayed in full control of the API design and how it interacts with the user, so that it will feel immediately familiar to all Contentful users. By having the entire life cycle of an upload process in our control, we can also provide a more convenient and secure experience. As you may have noticed in the example above, the service is just returning arbitrary IDs, which you can only use in the asset endpoints of the CMA to protect your uploads from any abuse. Of course, all new endpoints are protected, and users can only access them with a valid CMA access token.
To achieve all this in a fast way, we started to move forward with our Kubernetes production cluster. This service was an excellent opportunity to get more experience with the setup and still be able to scale to customer demand very quickly. For this reason, all upload endpoints are labeled as beta right now. You can expect a deep dive post on the topic of our Kubernetes setup coming up soon.
If you feel like trying out this new service, jump over to the documentation and upload some files. The team would love to hear your feedback on this. Happy uploading!