Incremental Static Regeneration with Next.js Combines Performance with Accuracy

One of the biggest concerns in modern web development is performance optimization. It’s been well-documented that carving every possible millisecond off page-load times improves the user experience, which can result in deeper engagement and higher conversion rates. Today, we rely on server and browser caching along with content delivery networks (CDNs) to streamline and localize network requests to provide the fastest page loads possible. But for sites that change frequently, how can we ensure that users are also getting the most current content, without fetching new data for every browser request or manually rebuilding the site for each content change? During a recent Next.js application build, we explored this challenge and found that Incremental Static Regeneration (ISR) lets us walk the line, providing best-in-class performance while also ensuring up-to-the-minute content.

The Project: Migration

Recently, the team at Sparkbox wrapped up a project migrating our blog (The Foundry—that’s what you’re reading right now!) away from ExpressionEngine, where it had lived for many years, housing hundreds of articles, each with dozens of customized components that needed to be rebuilt for Contentful’s rich text editor.

As we approached this challenge, the team had three main priorities to consider:

1. End-user performance

2. The ability to quickly preview content changes and promote them to production

3. The ability to quickly preview code changes and promote them to production

The development team settled on a suite of products that would allow us to accomplish these goals: the new site leverages Contentful’s headless “Composable Content Platform,” providing flexibility to tailor our content modeling for ease of management and stability for our development team to build a robust, scalable website with Next.js, the “React Framework for the Web.”

The Problem: Pagination

There were myriad possibilities we explored to solve a set of problems as we adopted a “consultant mindset” even for an internal project. In this case, we were presented with the opportunity to investigate several options that Next gives us for fetching data from our CMS through perhaps an unlikely vehicle: pagination.

By nature, paginated content is determined by a request from the client—usually by the user clicking a link at the bottom of a list of articles, which then adds some kind of flag to the URL in the browser in the form of a query parameter, e.g. page. The site recognizes that indicator, using it to return the appropriate subset of data. Consider the following URL:

https://sparkbox.com/foundry/articles?page=2

Next lets us interact with client-side information like this to detect the values we need, generating accurate content by fetching the data at the time of request. The useRouter React hook returns a query object, from which we can pull out our page variable, passing it on to our data request made with Apollo GraphQL. Here’s a pared-down version of our ArticleList component:

import { useQuery } from '@apollo/client';
import { useRouter } from 'next/router';

function ArticleList({ query }: ArticleListingPageProps) {
  const router = useRouter();
  let { page } = router.query;

  const PAGE = parseInt(page as string) || 1;
  const LIMIT = 10;
  const SKIP = (PAGE - 1) * LIMIT;

  const { loading, error, data } = useQuery(query, {
    variables: { page },
  });

  // replace `null` to render loading or error state if needed
  if (loading) return null;
  if (error) return null;

  // adapt data
  const { articles } = ADAPT_ARTICLES({ data } as ArticleCollectionResults);
  const numTotalArticles = data.articleCollection.total;
  const pagination = {
    numberOfResults: numTotalArticles,
    pageNumber: PAGE,
    resultsPerPage: LIMIT,
  };

  // render the adapted data as a list of articles with pagination
}

This seemed like a reasonable solution, but we quickly realized a problem with this approach. Because we were relying on a query parameter to generate our data, we were depending on a client-side request happening in the browser. Our request was relying on JavaScript to make a request to the Contentful API for every page load. This was preventing that content from ever being cached by the server, slowing down page loads. There was also the downside that no content would be rendered for users who had disabled JavaScript in their browsers, potentially including search engines.

The Pivot: Static Site Generation

In order to cache our paginated content and boost performance, we needed to pivot our approach to fetching the data. Next offers a few ways to do this—the specifics of how each of those functions work is outside the scope of this article, but the Next documentation on data fetching is thorough.

With the goal of moving our data requests out of the client, we decided to lean on the power of Static Site Generation using getStaticProps. As the documentation states, “If you export a function called getStaticProps (Static Site Generation) from a page, Next.js will pre-render this page at build time using the props returned by getStaticProps.”

This meant that we could request a list of articles in Contentful from the server, returning that data as props to the page, allowing the server to fully render the content without requiring a client-side JavaScript request. Huzzah! Here’s a look at how that works for a URL like https://sparkbox.com/foundry/articles

export const getStaticProps: GetStaticProps<{
  articles: ArticleListingProps[],
  total: number,
  variables: QueryVariableProps,
}> = async (context) => {
  const articlesResp = fetchData('GET_ARTICLES', variables);
  const { articles, total } = await articlesResp;

  return {
    props: {
      articles,
      total,
    },
  };
}

However, there was one more hiccup that you may have already anticipated—if our content is being pre-rendered on the server, we no longer have access to a client-side query parameter to determine which “page” of results we should be returning. After some consideration, we decided that using Next’s “dynamic route segments” would provide the pagination functionality that we needed while still following a server-first approach. This meant abandoning the URL query parameter pattern in favor of route segments such as:

https://sparkbox.com/foundry/articles/page/2

Once again to the documentation!

“If a page has Dynamic Routes and uses getStaticProps, it needs to define a list of paths to be statically generated.

“When you export a function called getStaticPaths (Static Site Generation) from a page that uses dynamic routes, Next.js will statically pre-render all the paths specified by getStaticPaths.”

Here, Next’s Static Site Generation functionality got us close to where we wanted to be, but not quite all the way. If we wanted to pre-render every single paginated route possible for every article tag, topic, and author at build time, we would need to include some network requests to the Contentful API during the build process and some logic to determine all the possible scenarios, generating a list of all the paths the server needed in order to create static pages. That request might start with something like this:

export const getStaticPaths: GetStaticPaths = async () => {
  const tags = await fetchData('GET_TAGS', {limit: 1000});

  return {
    paths: tags.filter(tag => tag?.slug).map(tag => ({
      params: {
        slug: tag.slug,
      }
    })),
  };
};

However, even before parsing the returned data to list articles for each tag, breaking that list up by our paginated subsets, and generating a list of paths for each, we quickly abandoned this approach. A network dependence on a third-party API at build time creates fragility and introduces complexity that may be difficult for future developers to troubleshoot if a build fails at any point. Fortunately, Next anticipates this need for flexibility and offers us the fallback parameter.

“getStaticPaths allows you to control which pages are generated during the build instead of on-demand with fallback. You can defer generating all pages on-demand by returning an empty array for paths.”

With this helpful information, we ended up streamlining our getStaticPaths export to just the following few lines:

export const getStaticPaths: GetStaticPaths = async () => {
  return {
    paths: [],
    fallback: true,
  };
}

This solution let us accomplish the following objectives:

• Returning an empty paths array means build times are fast because we’re not including every possible dynamic route that Next would need to generate pagination URLs for all of our article tags, topics, and authors in Contentful.

• Builds are less fragile because we’re not dependent on a network request to a third-party API.

• Passing fallback: true displays a temporary “fallback” state of the page on the very first request, then immediately renders the full page and caches it. Future requests to the same route are given the cached state of the page.

You can read more about how “fallback: true” changes Next’s rendering behavior of the site in their documentation.

The Polish: Incremental Static Regeneration

At this point, we had addressed our performance concerns by moving our data request from the client as the page is loading to the server at build time (for static pages) and at request time with a cached fallback state (for dynamic routes). However, we also needed to make sure our site was always current, accurately rendering the correct content without requiring a manual rebuild on the server every time we published a new article.

This is where Next truly shines.

“Next.js allows you to create or update static pages after you’ve built your site. Incremental Static Regeneration (ISR) enables you to use static-generation on a per-page basis, without needing to rebuild the entire site.

“When combined with Incremental Static Regeneration, getStaticProps will run in the background while the stale page is being revalidated, and the fresh page served to the browser.

“To use ISR, add the revalidate prop to getStaticProps”

With Incremental Static Regeneration, we’re getting the performance benefits of Static Site Generation and caching while automatically regenerating content in the background after a predetermined period of time, so our site remains current up to the minute! This is what we’re returning from getStaticProps now:

return {
    props: {
      articles,
      total,
    },
    revalidate: 60
  };

So let’s break down what’s happening when we combine fallback: true in getStaticPaths with revalidate: 60 in getStaticProps.

• At the first request for a page that hasn’t been pre-generated, the site will display a fallback version of the page without any dynamic content, then immediately replace it with a real-time fully-rendered page.

• On that first request, the server is querying Contentful for a list of articles and caching that response, then returning the cached version instantaneously for all requests happening in the next 60 seconds. 

• The first request that happens after 60 seconds will still receive the cached data but trigger a rebuild from the server in the background. 

• Once that rebuild completes, Next will automatically serve that version of the page for all subsequent requests, restarting the caching and revalidation cycle. Boom! 💥

In summary: getStaticProps works with ISR’s revalidate to provide cacheable, server-rendered versions of our pages, automatic request-time generation for routes or paths unspecified at build time, and automatic re-generation when cached data becomes stale.

The Results

We talked about two concepts in this article that help to provide a positive user experience: web page performance (how quickly the page loads and becomes interactive) and cache invalidation (making sure the page content is accurate and timely). While it’s difficult to illustrate the latter concept in this written format, it is fairly easy to illustrate the former. Using the tools at WebPageTest.org, we can benchmark a page at a specific point in time, then compare data points from a different version of the page.

The following screenshots compare The Foundry landing page as it performed in our legacy scenario (ExpressionEngine) with the same page in our new production environment (Contentful/Next.js). Test results like this will vary, depending on emulated network conditions and device settings, but what is immediately clear is that the page loads almost 3 times faster than before, with significantly faster visual progress. (Side note: while hosting providers can also play a role in page performance, that comparison is not our focus for this article.)

Additionally, when comparing an initial page load with a second view of the same page, you can see the immediate benefits gained from caching the server-rendered page content. Many of the benchmarks shown below have a lower number, some almost by half, indicating a faster response time for the second page request.

What a journey!

It was a fascinating process to discover and address real-world concerns that clients face when addressing their users’ interactions with their website. By iterating over several ideas, attempts, and solutions, we were able to uncover issues that were not initially anticipated, pivoting midstream to an approach that would still meet the objectives laid out in the technical strategy. The end result implements recommended practices for delivering a robust product, prioritizing performance and agility to ensure an exceptional user experience.