Manually download entries and assets into the SDK memory

Overview

When you build your React/NextJS application using Studio and Studio SDK, your app uses data from the Contentful Platform. As content editors keep creating and editing an experience, more and more data entries will be used in that experience. This also happens when the content editor binds entries to components, or binds visual assets to components.

This guide covers a special case: when your component has properties of type Link or Array which allow your component to accept not just a flat (or "shallow") entity or an asset, but an entire “tree” starting with an entry and pulling its associated entries.

For example, you can have a component that pulls in entityOfTypeLink and a “tree” of data stemming from it (up to what we call “level-3”).

function MyComponent({ entryOfTypeLink }){
   // Studio guarantees that it can load up to "level-3"
   // entity passed as prop is "level-2" 
   // NOTE: level-1 is only visible in Studio binding panel UI, 
   //       and is not passed to component.

   // To make references available you will need to resolve them using
   // inMemoryEntities.maybeResolveLink() utility function 
   // as shown in documentation. We skip this step here.
   // ... resolve referenced entities recursively ... 

   const entity = entityOfTypeLink; // L2
   const title = entryOfTypeLink?.fields?.title  // field on L2 entry
   const image = entryOfTypeLink?.fields?.image; // L3 asset
   const imageUrl = image.fields?.file?.url;  // field on L3 asset
   const relevantItem = entryOfTypeLink?.fields?.someRelevantItem; // L3 entry
   const relevantItemTitle = relevantItem?.fields.title; // field on L3 entry

   // this level of nesting is already beyond "level-3" and 
   // is NOT automatically loaded by the Studio and Studio SDK
   // NOTE: field on L3 entry points to L4 entry!
   const relevantItemNextLevel = relevantItem.fields.someRelevantItem; 
}

However, under certain circumstances, you may want to access data beyond “level-3”, meaning that you may want to have your entryOfTypeLink pull in a tree of data, which spans deeper then “level-3”.

Here’s an example of overfetching on simple “level-2” BlogPost entity and “level-3” Author entity:

When your React component that is designed to primarily display a blog post (BlogPost type entry) also needs to display the author’s name and picture (Author type entry), that author’s data will be prefetched by the SDK.

When the Author entry is downloaded, it will download all of the fields of that entry:

• author.fields.title (scalar field)
• author.fields.profilePic (reference to asset)
• author.fields.homeTownPic (reference to asset)

Note, that homeTownPic is not used by your component, but it’s still loaded. In this simple example, the code is overfetching only one reference. However, it can be more complex if an entry has dozens of references. You need to watch out for overfetching when manually preloading entities beyond “level-3”.

To prefetch the data you can manually use this method:

How to download additional entries when using NextJS

In your NextJS page that serves route for Studio-powered pages, usually you will be loading experiences using this pattern:

import Experience from '@/components/Experience';
import { getExperience } from '@/utils/getExperience';
import { detachExperienceStyles } from '@contentful/experiences-sdk-react';
import '../../studio-config';

type Page = {
  params: { locale?: string; slug?: string; preview?: string };
  searchParams: { [key: string]: string | string[] | undefined };
};

export default async function ExperiencePage({ params, searchParams }: Page) {
  const { locale = 'en-US', slug = 'home-page' } = params || {};
  const { isPreview, expEditorMode, mode } = searchParams;
  const preview = isPreview === 'true' || mode === 'preview';
  const editorMode = expEditorMode === 'true';
  const { experience, error } = await getExperience(slug, locale, preview, editorMode);

  const client = createClientWithConfig(preview);

  if (error) {
    return <div>{error.message}</div>;
  }

  const stylesheet = experience ? detachExperienceStyles(experience) : null;

  // experience currently needs to be stringified manually to be passed to the component
  const experienceJSON = experience ? JSON.stringify(experience) : null;

  return (
    <main style={{ width: '100%' }}>
      {stylesheet && <style data-css-ssr>{stylesheet}</style>}
      <Experience experienceJSON={experienceJSON} locale={locale} debug={true} />
    </main>
  );
}

The Studio SDK will automatically fetch data up to “level-3” when called within getExperience().

However, if there’s a need to load deeper levels of Contentful data, you can use the following method. The logic of this method can be found in the utility method fetchAdditionalLevels() .

The essence of the utility is:

• extract leaf links from the experience using the extractLeafLinksReferencedFromExperience() call
• assuming the Experience is loaded by Studio SDK up to “level-3” entities, the leaf links will be the links to “level-4” entities
• recursively download (if you want apart from “level-4” also “level-5”) all of the entities
• insert those manually downloaded entities into the inMemoryEntities cache. This is the same cache where SDK stores all of the Contentful entries (up to “level-3”), which means that you can rely on the same entity resolution algorithm that uses inMemoryEntities.maybeResolveLink() function

• when you call inMemoryEntities.addEntities(entities) the entities will be copied, and their copies will be Object.freeze'ed recursively, as per convention used for all inMemoryEntities cache

import type { Experience } from '@contentful/experiences-sdk-react';
import type { ContentfulClientApi, Asset, Entry } from 'contentful';
import { inMemoryEntities } from '@contentful/experiences-sdk-react';
import {
  extractReferencesFromEntriesAsIds,
  extractLeafLinksReferencedFromExperience,
} from '@contentful/experiences-core';

type EntitiesToFetch = {
  assetsToFetch: string[];
  entriesToFetch: string[];
};

export const fetchAdditionalLevels = async (
  depth: number,
  experience: Experience,
  localeCode: string,
  client: ContentfulClientApi<undefined>,
) => {
  // As first step we extract reference to L4 entities and kick off recursive fetching

  const addToMemory = (entities: Array<Entry | Asset>) => {
    // This function is a placeholder for whatever in-memory storage you are using
    // to store fetched entities. It should add the entity to your in-memory cache.
    // For example, if you are using a custom in-memory store, you might do:
    // inMemoryEntities.addEntity(entity);
    // For this example, we will just log the entity.
    if (entities.length === 0) {
      return;
    }
    inMemoryEntities.addEntities(entities);
  };

  const { assetIds, entryIds } = extractLeafLinksReferencedFromExperience(experience);

  await fetchLevel(depth, { assetsToFetch: assetIds, entriesToFetch: entryIds });

  async function fetchLevel(depth: number, { assetsToFetch, entriesToFetch }: EntitiesToFetch) {
    if (depth <= 0) {
      return;
    }

    const assetItems = await (async () => {
      if (assetsToFetch.length === 0) {
        return [];
      }

      const { items: assetItems } = await client.getAssets({
        'sys.id[in]': assetsToFetch,
        locale: localeCode,
        limit: 1000,
        skip: 0,
      });
      return assetItems;
    })();

    const entryItems = await (async () => {
      if (entriesToFetch.length === 0) {
        return [];
      }
      // Important that we should fetch Entries WITHOUT link resolution,
      // as that's the format that in-memory store expects.
      const { items: entryItems } = await client.withoutLinkResolution.getEntries({
        'sys.id[in]': entriesToFetch,
        locale: localeCode,
        limit: 1000,
        skip: 0,
      });
      return entryItems;
    })();

    // Reduce overfetching: Here you can add custom logic to omit certain fields
    // to reduce overfetching result. Fields have to be omitted manually.
    // entryItems.forEach((entry) => {
    //   entry.fields = {
    //     ...omit(entry.fields, 'allIngredients', 'allAuthors'),
    //   }
    // });

    addToMemory([...assetItems, ...entryItems]);

    {
      const [referencedEntryIds, referencedAssetIds] =
        extractReferencesFromEntriesAsIds(entryItems);

      // narrow down to the ones which are NOT yet in memory...
      const entriesToFetch = referencedEntryIds.filter(
        (entryId) => !inMemoryEntities.hasEntry(entryId),
      );

      const assetsToFetch = referencedAssetIds.filter(
        (assetId) => !inMemoryEntities.hasAsset(assetId),
      );

      await fetchLevel(depth - 1, {
        entriesToFetch,
        assetsToFetch,
      });
    }
  }
};

And now you can use this fetchAdditionalLevels() within the page component:

It’s important to guard the call to fetchAdditionalLevels() like shown below, because this fetching may only work in Preview+Delivery mode (this early prefetching will not work when the experience is loaded in Studio IFRAME because in that mode, experience will be undefined even though there’s no error).

  if (experience) {
    // experience is loaded by getExperience() because it is in Preview+Delivery mode,
    // when it EDITOR+READ_ONLY mode, it return undefined, as experience would be postMessage'd from the Studio.
    await fetchAdditionalLevels(3, experience, locale, client);
  }

In this method, we prefetch up to additional 3 levels (as per first argument to fetchAdditionalLevels(3,…)). Note that if there’s no references going deep, the recursive fetching of additional levels may end early.

• “level-4”
• “level-5”
• “level-6” If you want to only load additional single level, you can use fetchAdditinalLevels(1,…)
• “level-4”

So the only lines added to the page.tsx were:

import { fetchAdditionalLevels } from '@/utils/earlyPreload';

  //... 
  if (experience) {
    // experience is loaded by getExperience() because it is in Preview+Delivery mode,
    // when it EDITOR+READ_ONLY mode, it return undefined, as experience would be postMessage'd from the Studio.
    await fetchAdditionalLevels(3, experience, locale, client);
  }

// And we're creating a contentful SDK client too, via utility function
const client = createClientWithConfig(preview);

and

import Experience from '@/components/Experience';
import { getExperience, getConfig as createClientWithConfig } from '@/utils/getExperience';
import { detachExperienceStyles } from '@contentful/experiences-sdk-react';
import '../../studio-config';
import { fetchAdditionalLevels } from '@/utils/earlyPreload';

type Page = {
  params: { locale?: string; slug?: string; preview?: string };
  searchParams: { [key: string]: string | string[] | undefined };
};

export default async function ExperiencePage({ params, searchParams }: Page) {
  const { locale = 'en-US', slug = 'home-page' } = params || {};
  const { isPreview, expEditorMode, mode } = searchParams;
  const preview = isPreview === 'true' || mode === 'preview';
  const editorMode = expEditorMode === 'true';
  const { experience, error } = await getExperience(slug, locale, preview, editorMode);

  const client = createClientWithConfig(preview);

  if (error) {
    return <div>{error.message}</div>;
  }

  if (experience) {
    // experience is loaded by getExperience() because it is in Preview+Delivery mode,
    // when it EDITOR+READ_ONLY mode, it return undefined, as experience would be postMessage'd from the Studio.
    await fetchAdditionalLevels(3, experience, locale, client);
  }

  const stylesheet = experience ? detachExperienceStyles(experience) : null;

  // experience currently needs to be stringified manually to be passed to the component
  const experienceJSON = experience ? JSON.stringify(experience) : null;

  return (
    <main style={{ width: '100%' }}>
      {stylesheet && <style data-css-ssr>{stylesheet}</style>}
      <Experience experienceJSON={experienceJSON} locale={locale} debug={true} />
    </main>
  );
}

How to download additional entries when using React CSR

The source code for early preloading of entities for CSR app can be found in Page.tsx.

The essence of this method revolves around calling the fetchAdditionalLevels() utility function.

The main changes are:

// we are adding state variable which will signal that early preloading was complete
const [areAllAdditionalLevelsFetched, setAreAllAdditionalLevelsFetched] = useState(false);

// we add effect that does actual loading 
// and effect has a lot of guards
   useEffect(
    function effectFetchAdditional() {
      if (isLoading) {
        return;
      }
      if (experienceLoadingError) {
        return;
      }
      if (!experience) {
        return;
      }

      if (areAllAdditionalLevelsFetched) {
        return;
      }

      async function earlyPreload() {
        try {
          await fetchAdditionalLevels(3, experience, localeCode, client);
          await new Promise((resolve) => setTimeout(resolve, 3000)); // for demo
          setAreAllAdditionalLevelsFetched(true);
      //... 


// ... 
// We add "loading" progress banner, to signal about early preloading 

return (
    <>
      {!shouldShowBannerAboutLoadingAdditionalLevels ? null : (
        <h3
          style={{
            background: 'black',
            color: 'white',
            position: 'fixed',
            left: '0',
            top: '0',
            width: '100%',
            margin: '0',
            padding: '16px',
          }}>
          Loading additional levels... <sup>(won't trigger on hot-reload)</sup>
        </h3>
      )}
// ....     

Here is the full code sample (or find it on github in Page.tsx.):

import './studio-config';
import { useParams } from 'react-router-dom';
import { useEffect, useState } from 'react';
import './styles.css';
import { ExperienceRoot, useFetchBySlug } from '@contentful/experiences-sdk-react';
import { useContentfulClient } from './hooks/useContentfulClient';
import { useContentfulConfig } from './hooks/useContentfulConfig';
import { fetchAdditionalLevels } from './utils/earlyPreload';
import { StudioCanvasMode } from '@contentful/experiences-core/constants';

export default function Page() {
  const { slug = 'home-page', locale } = useParams<{ slug: string; locale?: string }>();
  const localeCode = locale ?? 'en-US';
  const { config } = useContentfulConfig();
  const { client } = useContentfulClient();

  const [areAllAdditionalLevelsFetched, setAreAllAdditionalLevelsFetched] = useState(false);

  const {
    experience,
    error: experienceLoadingError,
    isLoading,
    mode,
  } = useFetchBySlug({
    slug,
    localeCode,
    client,
    experienceTypeId: config.experienceTypeId,
    hyperlinkPattern: '/{entry.fields.slug}',
  });

  // techincally here I should be able to start loading additional levels...
  useEffect(
    function effectFetchAdditional() {
      if (isLoading) {
        return;
      }
      if (experienceLoadingError) {
        return;
      }
      if (!experience) {
        return;
      }

      if (areAllAdditionalLevelsFetched) {
        return;
      }

      async function earlyPreload() {
        try {
          await fetchAdditionalLevels(3, experience, localeCode, client);
          await new Promise((resolve) => setTimeout(resolve, 3000)); // to demo progress banner
          setAreAllAdditionalLevelsFetched(true);
        } catch (error) {
          // you can decide yourself how to handle failed loading
          console.error('Error fetching additional levels:', error);
          throw error;
        }
      }
      earlyPreload();
      return () => {
        console.warn(';;[effectFetchAdditional] Effect cleanup.');
      };
    },
    [
      experience,
      isLoading,
      experienceLoadingError,
      mode,
      areAllAdditionalLevelsFetched,
      client,
      localeCode,
    ],
  );

  const shouldShowBannerAboutLoadingAdditionalLevels =
    mode === StudioCanvasMode.NONE && !areAllAdditionalLevelsFetched;

  if (isLoading) return <div>Loading...</div>;

  if (experienceLoadingError) return <div>{experienceLoadingError.message}</div>;

  return (
    <>
      {!shouldShowBannerAboutLoadingAdditionalLevels ? null : (
        <h3
          style={{
            background: 'black',
            color: 'white',
            position: 'fixed',
            left: '0',
            top: '0',
            width: '100%',
            margin: '0',
            padding: '16px',
          }}>
          Loading additional levels... <sup>(won't trigger on hot-reload)</sup>
        </h3>
      )}
      <ExperienceRoot experience={experience} locale={localeCode} />
    </>
  );
}