Dynamic or Static? Why not both...
Posted: 21st May 2021
In our previous article we discussed some of the methods that can be used to implement caching within our Craft CMS projects. All of these techniques can help us to reduce the overall time required to process a user's request, with static HTML caching providing us with the best performance benefits.
However, there are some situations where HTML caching can get in the way of our application's functionality. Any part of a page which needs to show different content to each user (E.G a logged in state) or that relies on being completely up-to-date (stock levels on an ecommerce site) can make this full caching approach difficult to implement.
Below, we'll explore the main method used to get around this problem, highlight some its pitfalls and introduce a new feature which has recently been added to the Servd Plugin and provides a simple method of mixing static and dynamic content in an easy and performant way.
CSRF Tokens
Likely the first and most common static caching issue you'll run into is with CSRF tokens.
By default, Craft will require CSRF tokens to be sent with any POST requests to your application. The reason why is beyond the scope of this article, but you can read some more about it here. The most important thing we need to keep in mind, is that these tokens change every time a user loads a page and if an unexpected CSRF token is included in a POST request, Craft will reject it.
The correct CSRF tokens are normally included in the HTML sent to the user from the server, however when we cache our HTML responses, the embedded CSRF tokens will be cached too, resulting in incorrect tokens being sent to all of our users.
It's therefore important to ensure that we have some mechanism available to us to update these tokens as required. We need to make sure that these tokens remain dynamic, even when the rest of our page is statically cached.
This isn't a new problem, so a widely adopted solution has been around for a while and can be found included in some of the Craft plugins which help with caching (including the Servd Plugin).
We get around the problem by:
- adding a piece of javascript to the cached HTML which performs an AJAX request back to the application on a URL which is excluded from any caching mechanisms.
- This ajax request simply generates a new CSRF token and returns it to the user.
- Once the javascript receives the new token it can insert it into all the locations in the HTML that contain an invalid, cached token.
This allows us to keep the majority of our HTML content cached, and perform one quick request to the server to generate a new CSRF token.
As an example, here's an example of a controller function that we could add to a custom plugin or module to perform the server-side logic we need:
public function actionGetToken()
{
$req = Craft::$app->getRequest();
return $this->asJson([
'token' => $req->getCsrfToken()
]);
}Other Dynamic Content
This method of retrieving content using AJAX requests to uncached URLs can be extended to other dynamic elements on the page.
If you have a section in the header of your site showing the user's login state, we could pull that in via AJAX. Likewise, if we want to show stock levels on an ecom site, we can use the same technique.
These pieces of dynamic content can be a little more complex to implement though. It's likely you'll want to use a twig template to render them, so the uncached AJAX request will need to be set up to render that twig template.
Craft gives us all of the tools we need to achieve this. We can first set up our dynamic content as a twig template that we traditionally might {% include %} within our other templates, but instead of {% include %}ing it directly we can use the AJAX approach.
In a similar way to the CSRF token process, we can:
- Insert a placeholder node into our cached HTML where the dynamic content should go
- Insert some javascript into our cached HTML which looks for these placeholders and performs an AJAX request which includes the name of the template we want to swap the placeholder for
- The server-side controller handling this AJAX request renders the requested template and returns it
- The javascript swaps out the placeholder for the new, dynamic content it has received
An example of what the server-side process for generating the template might look like:
public function actionGetContent()
{
$req = Craft::$app->getRequest();
$templateName = $req->getQueryParam('templateName');
$output = Craft::$app->getView()->renderPageTemplate($templateName, []);
return $output;
}This method works well for pages with a small number of simple, dynamic elements, but can quickly run into a couple of different issues.
Attack of the AJAX
The benefit of pulling in dynamic content using AJAX is that, although we still have to make an uncached request to the server, the request needs to perform a lot less work than it would if it had to render the entire page.
However, as the number of individual dynamic elements on a page increases, so will the number of AJAX requests, which can lead to reduced overall performance, even when caching the majority of the page.
To understand why this might happen we have to consider all of the steps that take place when a request is sent to the server.
- The incoming request is processed by the webserver and forwarded to PHP
- PHP allocates the request to a worker and begins processing it
- Craft and the Yii framework run their initialisation scripts
- All installed plugins run their initialisation scripts and any other start-of-request logic
- Controller logic runs and/or templates are rendered
- Plugins run any end-of-request logic
- Craft / Yii format and dispatch the result back to the user
- Cleanup
This process occurs for every individual incoming request, including all AJAX requests.
It's easy to imagine a scenario in which a fully cached page actually creates more pressure on the server if it is sending multiple AJAX requests in order to insert dynamic content.
On one site which we recently audited, a single cached page was issuing 14 (!) individual AJAX requests for dynamic content. The total time spent rendering these on the server was more than double the time it would have taken to just render the page without any caching. Also, because all of these AJAX requests were being sent to the server at the same time, they were consuming most of the available PHP workers, blocking the processing of other incoming requests.
This behaviour needs to be very carefully considered when working with injected dynamic content, either manually or using a plugin. There are no plugins we are aware of for Craft which do not fall into this AJAX attack trap, with one exception which you can read about below...
Loss of Context
When you {% include %} a template within another template in twig, the child template inherits the 'context' of its parent, so all of the variables you had access to in the parent are also available in the child.
This avoids us having to explicitly list all of the properties that we would like the child to have access to, and saves our fingers from a lot of additional typing.
When we pull in content dynamically via AJAX we lose this context inheritance.
One option we have is to rebuild the context from scratch within the dynamic content template or its controller, but this can become unwieldy and often results in the need to perform additional database queries just to rebuild a context that we would have otherwise had available to us. We also don't always have all of the information we need to make this viable.
Another is to include a copy of the context within the AJAX request itself, by attaching the variables to the URL or including them in the body of a POST request. This can work well as it allows us to rebuild the context for the dynamic content without performing any other interim processing, but it has one major flaw: it can only pass simple values (strings, numbers, booleans and arrays of these values).
This will limit the data that we can include in the context for our dynamic content unless we work around the problem. We'd likely need to introduce logic to grab entry.id within the parent template and include that in the AJAXed context, then rehydrate entry using its id within the child template.
For any complex templates, you can imagine this becoming tedious! And if you ever decide to disable static caching and switch back to just using standard {% include %}s for your dynamic elements, you'll need to strip all of that extra logic back out again.
Tight Coupling
Let's imagine you're working on a project locally and you've been very diligent and split all of your different components up into different twig templates. In your header, you might have something like this to render a login/logout button:
{-- _components/header.twig --}
<header>
<h1>My Website</h1>
{% include '_components/login' %}
</header>
{-- _components/login.twig --}
{% if currentUser %}
<button>Log Out</button>
{% else %}
<button>Log In to {{ currentSite.name }}</button>
{% endif %}Just before go live, you decide that it would be great to statically cache your pages, but you want to keep your dynamic login button working. So you update your templates to look like this:
{-- _components/header.twig --}
<header>
<h1>My Website</h1>
{{ craft.cachingPlugin.dynamicallyIncludeTemplate('_components/login', {
currentSiteId: currentSite.id
}) }}
</header>
{-- _components/login.twig --}
{% set theSite = craft.sites.getSiteById(currentSiteId) %}
{% if currentUser %}
<button>Log Out</button>
{% else %}
<button>Log In to {{ theSite.name }}</button>
{% endif %}And you duplicate similar work throughout your project for lots of individual portions of dynamic content.
Phew, a lot of work but now we can statically cache our pages!
Unfortunately, the caching plugin raises its prices or becomes unsupported and now all of that work needs to be undone in order to make the project usable again.
This tight coupling of plugins to essential functionality within a project is something which should be avoided if at all possible. One quick way that this can be achieved is through the use of macros which act as an interface between your twig files and the plugins that they use:
{-- _macros/plugins.twig --}
{% macro dynamicContent(template, context) %}
{{ craft.cachingPlugin.dynamicallyIncludeTemplate(template, context) }}
{-- You can use this in local dev if you like --}
{-- include template with context only --}
{% endmacro %}
{-- _components/header.twig --}
{% import "_macros/plugins" as plugins %}
<header>
<h1>My Website</h1>
{{ plugins.dynamicContent('_components/login', {
currentSiteId: currentSite.id
}) }}
</header>
{-- _components/login.twig --}
{% set theSite = craft.sites.getSiteById(currentSiteId) %}
{% if currentUser %}
<button>Log Out</button>
{% else %}
<button>Log In to {{ theSite.name }}</button>
{% endif %}Now you only need to update your plugins.twig file in order to switch to another dynamic content plugin, or to disable it altogether.
It would still be better if you didn't need to think about this at all though!
How The Servd Plugin Helps
Clearly we've put a lot of thought into this. That's because we want to give your projects every possible opportunity to make use of the performance improvements that can be gained through the use of static caching.
So we've been working on a solution to all of the above and it's available to use now in The Servd Plugin.
Introducing: {% dynamicInclude %}
Once you have the Servd Plugin installed this tag becomes available in all of your twig templates. Here's an example of it in use on this very site (it loads in the Pricing info on the home page which changes currency depending on your location):
{% dynamicInclude '_singles/home/pricing' %}Not much to look at, and that's a good thing!
Here's how it solves the problems we've covered above:
Pulling In Dynamic Content
It uses the same AJAX approach that we've been talking about, by injecting placeholders into your HTML with a small piece of javascript at the end of the <body> to replace them with the dynamic content. It also handles all of the server-side logic to render the dynamic HTML too of course.
Attack of The Ajax
All blocks of dynamic content on a page are bundled into a single AJAX request. This vastly reduces the overhead associated with performing requests for multiple pieces of dynamic content. Sprinkle these tags throughout your templates without worrying about choking your server.
Loss of Context
The tag automatically gathers, encodes and compresses the context for each of your dynamic includes individually and uses them whilst rendering their content. This includes any Craft Elements (Entries, Assets etc) and requires no additional logic to do so.
Tight Coupling
The tag can be used with the exact same syntax as the standard {% include %} tag. Switching from static to dynamic is as simple as replacing {% include %} with {% dynamicInclude %}. Switching back again just needs a find and replace.
The plugin also has a simple switch in its settings to disable all dynamic functionality making the tags act just like their static cousins. This allows you to switch off all of those ajax requests while you're working in a local dev environment and caching isn't a problem.
Who Can Use This?
This functionality will work for any project which has the Servd Plugin installed, inside or outside of the Servd platform, and will also work with most other static caching solutions (including CloudFlare edge caching).
We're really excited to see people start using this new feature and hope it helps more Craft sites to adopt the performance improvements associated with static caching, without sacrificing their dynamic elements.
You can read some more documentation in the Static Caching section of our Plugin Docs.
An Added Bonus For Enterprise
If you have an enterprise project running on Servd, you might also have access to our ESI functionality which allows HTML pages to be built from individual pieces, much like the AJAX approach, but the assembly is done directly within Servd's platform, eliminating the round trip time needed to perform an AJAX request.
The {% dynamicInclude %} tag will detect if this functionality is available to you and will switch to using that automatically if it is. Allowing you to mix static and dynamic content, easily and with zero AJAX requests.
🚀
