Asset File Transfer, Caching, and Storage in AWS

Version: Deadline 10.0

Introduction

One of the many goals of Deadline 10 is to ensure that customers can render on AWS as easily as possible. An essential part of moving to a hybrid or fully AWS-based rendering workflow is dealing with transferring asset files to AWS, and downloading resulting renders from AWS. Fortunately, customer can accomplish this job with Deadline 10 painlessly by using the new features that make dealing with asset files in AWS a breeze!

Simplify Your AWS Render Pipeline

In almost all cases, a rendering job will depend on many external files and textures to produce its renders. When moving to the cloud, these files will not exist in cloud storage prior to rendering. Previously, artists had to manually transfer the required files to cloud storage prior to submitting a render. Artists would also have to manually download the resulting renders after the rendering job was complete. Previously, this presented an extra hurdle in cloud-based workflows. If only the AWS render nodes could directly use the on-premise network file system. Well now they can with Deadline 10!

Use Your On-Premise Network Filesystem in AWS

asset_file_transfer_caching_and_storage_in_aws.png

Deadline 10 offers a service that allows rendering nodes to see your on-premise network filesystem as if it were in the cloud. The files themselves do not need to exist in the cloud in order for the rendering application to use them.

Asset files that are needed get copied to cloud storage prior to the render beginning. Any files that were missed in the initial copy get copied to cloud storage when they are first accessed. This workflow takes away any manual asset gathering steps that an artist would need to do previously. No asset files are missed, and everything renders as if it were local.

As soon as files are uploaded, the files are cached in cloud storage. Once a file is initially cached, that file will never be re-uploaded unless its source has been modified on-premise. This allows the user to render an entire sequence, while the required files are only transferred to cloud storage once.

Automatic Downloading of Rendered Images from AWS

In addition to automatically uploading required assets from the on-premise network, Deadline 10 also automatically downloads the resulting rendered images back to the on-premise network storage. This happens without any artist intervention and begins as soon as a render task is complete. These render files also remain cached in cloud storage which allows future Deadline jobs to use the results of a render without having to upload the files from on-premise.

How It Works

All Deadline 10 AWS-based render nodes have the option to mount a directory which can mirror an on-premise network share. Using Deadline's built-in Path Mapping options, users are able to path map this mounted directory to their on-premise directory root. When this option is enabled, all the communication with the on-premise network, as well as the caching, transfer, and storage of files are handled behind the scenes.

On the user's on-premise side, Deadline 10 provides client software which must be installed. This client software is able to connect with the AWS render nodes and provide access to the on-premise files. Our client software does not require the use of a VPN, and does not require any modifications to the on-premise firewall. Only your AWS-based rendering nodes will be able to connect to it.

Wrapping Up

We hope you enjoy Deadline 10's new cloud rendering features. We are working hard to make Deadline renders on AWS as easy as possible! Let us know what you think of our new filesystem features on the Deadline Forums.