Name: Servd Hosting
Price range: $$

Documentation

Handling runtime created files that need to persist

Servd uses Ephemeral file systems and Load Balanced instances to provide a range of benefits for all projects, but it comes at the cost of not being able to trust that files written directly to the filesystem will persist for any length of time. So how should we handle runtime generated files that we might want to serve to clients?

Let's start with an example project which downloads some info from an API, stores the data locally and then allows clients to download it. A custom module to handle these features might look like:

public function refreshStoredData()
{
    $data = $this->getDataFromApi();
    $localCopyLocation = \Craft::$app->path->getStoragePath() . '/myfile.json';
    file_put_contents($localCopyLocation, $data);
}

public function sendDataToClient()
{
    $localCopyLocation = \Craft::$app->path->getStoragePath() . '/myfile.json';
    if (!file_exists($localCopyLocation)) {
        return $this->asJson(["error" => "The file does not exist yet"]);    
    }
    return \Yii::$app->response->sendFile($localCopyLocation, "myfile.json");
}

This setup should work fine in local dev or when we are using a non-loadbalanced and non-ephemeral server. However, on services such as Servd it suffers from two problems:

The filesystem will be regularly reset, resulting in the locally stored file being removed on a regular basis.
If the project is load balanced the file will only be downloaded to one of the instances' filesystems, resulting in different results depending on which instance a user's download request is routed to.

We can fix both of these issue by avoiding the local filesystem when storing our file. There's a couple of options for where we do this:

In a shared cache
In a remote asset storage bucket

In A Shared Cache #

Storing the file in a shared cache is easy on Servd - Craft's data cache is automatically mapped to a stateful Redis server, so instead of writing the file to the disk, we can simply add it to the cache!

public function refreshStoredData()
{
    $cacheKey = 'myData';
    $data = $this->getDataFromApi();
    \Craft::$app->cache->set($cacheKey, $data);
}

public function sendDataToClient()
{
    $cacheKey = 'myData';
    if (!\Craft::$app->cache->exists($cacheKey)) {
        return $this->asJson(["error" => "The file does not exist yet"]);   
    }
    $data = \Craft::$app->cache->get($cacheKey);
    return \Yii::$app->response->sendContentAsFile($data, "myfile.json");
}

In an Asset Volume #

For large files, an alternative is to use a remote asset volume (such as the Servd Asset Platform).

You can create a volume specifically for holding these types of runtime files, or you can piggyback on top of an existing volume that you're using for other uploaded assets such as images (but bare in mind that if you piggyback on another volume, re-indexing that volume will detect these files and add them to the volume's file listing).

Once you have a volume set up and ready to use, we can use FlySystem to push and pull the file to the volume.

public function refreshStoredData()
{
    $data = $this->getDataFromApi();
    $localCopyLocation = \Craft::$app->path->getStoragePath() . '/myfile.json';
    file_put_contents($localCopyLocation, $data);

    //Upload it to the volume
    $pathInRemoteStorage = '/runtime/myfile.json';
    $volume = Craft::$app->volumes->getVolumeByHandle('my-volume');
    $filesystem = $volume->getFs(); // Drop down to the filesystem for Craft 4
    $filesystem->writeFileFromStream($pathInRemoteStorage, fopen($localCopyLocation, "r"));
}

public function sendDataToClient()
{
    //Download it from the volume
    $pathInRemoteStorage = '/runtime/myfile.json';
    $volume = Craft::$app->volumes->getVolumeByHandle('my-volume');
    $filesystem = $volume->getFs(); // Drop down to the filesystem for Craft 4
    
    if (!$filesystem->fileExists($pathInRemoteStorage) {
        return $this->asJson(["error" => "The file does not exist yet"]);   
    }

    $fileStream = $filesystem->getFileStream($pathInRemoteStorage);

    return \Yii::$app->response->sendContentAsFile($fileStream, "myfile.json");
}

This allows us to upload and download the file to our remote storage as required, solving our two issues. However it isn't particularly efficient - each time we send the file to a user we have to download it from the remote storage first. We can solve this by simply caching the file contents on the disk:

public function sendDataToClient()
{
    $localCopyLocation = \Craft::$app->path->getStoragePath() . '/myfile.json';
    
    if(file_exists($localCopyLocation)){
        return \Yii::$app->response->sendFile($localCopyLocation, "myfile.json");
    }

    //Download it from the volume
    $pathInRemoteStorage = '/runtime/myfile.json';
    $volume = Craft::$app->volumes->getVolumeByHandle('my-volume');
    $filesystem = $volume->getFs(); // Drop down to the filesystem for Craft 4
    
    if (!$filesystem->fileExists($pathInRemoteStorage) {
        return $this->asJson(["error" => "The file does not exist yet"]);   
    }

    file_put_contents($localCopyLocation, $filesystem->getFileStream($pathInRemoteStorage));

    return \Yii::$app->response->sendFile($localCopyLocation, "myfile.json");
}