Static sites are gaining popularity. Big publishers like Smashing Magazine rely on static site generation to serve content faster. And they do it without worrying about security fixes or scalable server setups. All you need for static site generation is a CI service that kicks off your build and a static hosting provider to serve your generated static files which we then enrich with serverless technologies.
I’m a big believer in the static sites approach, but this approach comes with a set of challenges depending on what you want to accomplish. One problem is to guarantee short build times: file generation takes time, and if you want to generate twenty thousand pages, the build time increases — which leads to frustration and delays in the publishing workflow.
You might say that you won’t run into these challenges with your project or personal website, and I believed the same thing a few months ago. But recently I was facing the problem of builds taking longer and longer. My private website uses Contentful (based on Vue.js). It is deployed via Netlify to Netlify and I was hitting a build time well over 10 minutes — not acceptable.
In this first of two articles on static sites, I will share with you how you can speed up your build process on Netlify with a custom caching layer. The second article will go into the implementation of incremental builds using Nuxt.js.
Why did the build time increase so much in the first place? A few months ago I came across SQIP. SQIP is a new tool by Tobias Baldauf to generate beautiful SVG placeholder images using Primitive. These placeholders can improve the perceived performance of lazy loaded images. Primitive examines the images and generates SVGs that represent the image with primitive shapes which look surprisingly good when you apply a blur effect.
Using these beautiful preview images the user knows what to expect when the image loading kicks in which leads to a better user experience than spinners or random loading graphics.
The way it works is that you place a small SVG graphic below the image that will appear eventually and fade in.
If you’re not interested in implementing these good-looking placeholder images, and only want to read about caching on Netlify, you can jump right to the “Caching for the win” section.
Here is how it works – my images are stored in Contentful, and to generate the SVG previews I go through these steps:
Getting information of all assets stored in Contentful
Download all the images
Generate placeholder SVGs of the images
All the following code sections are small parts of all a longer script which will be linked at the end of the article, and the code makes heavy use of async functions which make the handling of asynchronous operations so much better! As a result, whenever you see an
await somewhere, it is placed inside of an async function in the overall implementation.
Following best practices, the resulting script requires all the dependencies on top of the file whereas in the included code sections I place them right before I use them to make the snippets easier to understand.
Getting all the asset information from the Contentful API is straightforward. I only have to initialize the Contentful SDK client, and the getAssets function gives me the information I need.
First I have to filter all the assets to strip out files that are not PNGs or JPEGs. Then I get rid of all the meta information that I’m not interested in via a
At this point, I have an array
revision and the particular image
url. The collection also includes a
filename property which is the combination of asset ID and its revision.
The connection of these two attributes is necessary because whenever I update an asset, I also want to generate a new preview SVG – this is where the revision number comes into play as it changes in this case.
With this collection of information of all the assets for my site, I continue with downloading all the assets. The download package I found on npm is a perfect fit.
All the asset entries are mapped to promises returned by the download function and everything wrapped into a
Promise.all so that I can be sure that all the images are downloaded to the predefined
IMAGE_FOLDER. This is where async/await shines!
SQIP can be used programmatically which means that you can require the module and you are good to go.
sqip module doesn’t write files to disk though. It returns an object including the generated SVG in the
final_svg property. You may say that I could use the SVG string value and store the SVG directly in the
images collection, but I went with writing the SVG to disk first.
I also use the fs-extra package that provides some convenience methods over the native
fs module, and also maps callback functions to their promisified versions so that I don’t have to make, e.g.
writeFile promises based myself.
This has the advantage that I can have a look at the generated SVGs on my hard drive quickly, and it will also come in handy later in the caching section of this article.
The SQIP module accepts the following arguments:
numberOfPrimitives defines the number of shapes (10 shapes works for me with rather small SVG files but a good preview experience)
mode defines which type of shapes the generated SVG should include (triangle, square, circles, all of these)
blur defines the level of applied blur (I went with no blur in the SVG as I discovered that the result of CSS blur leads to better results)
fs-extra also provides a
readFile function, so I’m ready to flow promises based.
The collection of asset objects gets enriched with the string value of the generated SVG. This string value also adds the asset ID to the SVG so that I can later see what asset was the base for a particular SVG preview image.
The last step – the collection of assets now includes meta information, and also the generated stringified SVGs in the
The execution of the resulting script, including nice logging messages takes two to four minutes on my MacBook Pro for 55 assets (depending on what else is running on my machine).
When it runs on Netlify though, the script execution could easily take five to seven minutes resulting in build times around the mentioned ten minutes
The repeated regeneration is not an optimal approach. With this script, every build would do the same heavy lifting – over and over again. Whenever you repeat operations, may it be image optimizations or other massive computations that take several minutes, it’s time to improve.
The beauty of a continuous delivery pipeline is that things can go live regularly and quickly – ten minutes to bring a typo fix into production is not the environment I want to deal with for my small site.
So how do I sort out this mess?
I could generate the image previews myself and also upload them to Contentful which has the downside of having two assets depending on each other that I need to deal with (the image and the preview) – not an option.
I could commit the preview to the git repository, but I always feel bad committing large assets to git. Big binary files are not what git is made for, and it increases the size of the repository drastically – no option either.
Netlify runs every deploy in a docker container without the possibility to reuse things from the previous deploy (except for dependencies – but I don’t want to misuse the node_modules folder for my own things). My initial solution was an S3 bucket acting as a cache layer during my builds.
The cache layer would hold the downloaded images and generated previews from the previous build, and due to the ID and revision naming convention, a file existence check would be enough to figure out what new assets need to be generated. This approach worked fine but then Phil from Netlify shared a secret with me (be careful though – it’s not documented and usage is at own risk).
It turns out there is a folder that persists across builds –
/opt/build/cache/. You can use this folder to store files across builds which leads to a few additional steps in my script but decreases the time of the SVG generation drastically:
Getting information of all assets stored in Contentful
Check what SVGs have already been generated
Download missing images
Generate placeholder SVGs of missing images
The image folder that I defined in the script now becomes a cache folder (
SQIP_CACHE) depending on the environment.
This way I could run the script on my development machine and place all the files in a folder that is also ignored by git, but when running on Netlify it uses the persistent folder.
images collection I used previously?
I then add another step to the previous script and see if an SVG with the right asset ID and revision combination is available in the cache folder.
If so, read the file and define the
svg property of the image entry, if not, go on.
The generation of SVG files stays the same, except that I can now check if there is already a generated SVG value available like so:
With the improved script I can avoid repeated computation, and the build times on my local machine and Netlify went down to not even one second for repeated builds with a filled cache!
If you want to play around with it, the provided gist includes all you need to generate and cache beautiful image previews with a Contentful example space.
There was one last thing though – caching can be hard and especially when you implement a cache on remote servers which you can not access you should be able to throw everything away and start over again.
In my case running on Netlify, I went for a custom webhook that clears the caching directory before anything happens when this webhook triggers the build.
The addition of the preview cache improved the building experience of my static site drastically. I love the Contentful, Nuxt.js and Netlify setup and now that the build times are at three minutes again I can start thinking about the next improvement – the speedup of the generation of the static HTML files.
Contentful's sync endpoint provides granular information about what changed compared to the last sync, and is a perfect fit for this use case which makes incremental builds possible – a topic a lot of big static site generators struggle with. You can read about that soon. I'll let you know!