Version: Deadline 7.2, 8.0
Overview - Increasing Render Throughput
As more and more jobs are submitted to your render farm, it becomes important to make use of the tools Deadline provides for maintaining a speedy render pipeline. These tools are both well documented and easy to use! This blog post examines two such tools in depth: Concurrent Tasks and Multi-Slave rendering. Taking advantage of concurrent tasks and making use of multiple Slaves are two ways that Deadline provides a highly efficient, scalable environment for rendering large projects.
Concurrent Tasks and Multi-Slave rendering sound almost like the same concept, but it is important to outline their differences. Concurrent tasks allow a single Deadline Slave to pick up multiple tasks of a job at one time. On the other hand, having multiple Slaves running on a single machine allows for multiple jobs or tasks (or concurrent tasks!) to be picked up at once. Generally speaking, Concurrent Tasks are a good way to increase the efficiency of a single job. Meanwhile, Multi-Slave rendering often improves the efficiency of a multi-job project (multiple Slaves can be used for a single job too). Though there are some tradeoffs to discuss, putting these two tools together results in an incredibly efficient way to perform large amounts of Deadline jobs.
Instructions and Use-Cases
Enabling Concurrent Tasks is a great way to improve the speed of a Deadline job in the context of a single Slave. The number of concurrent tasks corresponds to the number of tasks a single slave may pick up at one time (from a single job). For example, if I have a job with multiple tasks and I set the Concurrent Task number to 3, then the next Slave to pick up the job will pick up 3 of the job’s tasks instead of just 1!
To enable Task Concurrency for a given job, complete the following:
Select your job in the Monitor, right click the job and select the “Modify Job Properties” option. Under “General”, a “Concurrent Tasks” option is available.
Please note that not every plugin supports concurrent tasks, but every plugin that does will have a “Concurrent Tasks” option in their In-app Submitter as well as in the above Monitor option. You can also choose to limit tasks to the Slave’s task limit. This means that a Slave will not pick up more tasks from a job than its limit is set to. Regardless of whether a concurrent task limit is set, no more than 16 tasks can be concurrently performed by any given Slave.
Concurrent Tasks are a smart approach to increasing the efficiency of a single job. For example, if you are rendering a Nuke scene with 30 frames, it will be faster to allow a Slave to pick up multiple tasks at a time instead of only one. Conversely, 3dsMax is highly multithreaded at render time and prefers to exist as a single task on a single Slave on a machine.
Notice in the above screenshot that multiple tasks are being rendered at one time, but only one Slave is currently online. Concurrent Tasks allow your Slaves to render multiple frames at once, thus decreasing the processing time of a Deadline job, assuming your machines have the available bandwidth (network and storage i/o infrastructure restrictions).
When a Slave runs multiple tasks concurrently, only 1 license will be checked out. This means that you can have a Slave render up to 16 tasks at the same time, and still only use 1 Deadline Slave license.
Running multiple Slaves on one machine is another great way to improve the speed of your Deadline workflow. The advantage of using multiple Slaves is the fact that more than one job can be accomplished in a given timeframe. A multi-Slave approach makes up for the fact that a single Slave cannot pick up tasks from multiple jobs. Multiple slaves can also pick up tasks from the same job, increasing Deadline’s efficiency in the context of a single job.
For this blog I will demonstrate how to add an additional Slave to your machine, so you can try this out yourself.
To add a new Slave, you must make sure that permission to “Launch New Named Slave” is enabled for your user group as shown here:
Note that user groups can only be configured in super-user mode.
Once you are able to do so, open your Deadline Launcher and hover your mouse over “Launch Slave By Name”. Select the “New Slave Instance” option and call your new Slave anything you like.
Now you can use your multi-Slave machine to tackle multiple Deadline jobs at once. Multiple Slaves are particularly useful when you have a large batch of independent jobs. For example, if you have 2 Deadline Slaves and 2 Maya render jobs, you can have each Slave render 1 Maya render job and (hypothetically) complete your work twice as fast!
Rendering with multiple Slaves allows multiple jobs to be finished at once. Consider the example below:
In this screenshot, 4 different Python jobs are currently running in the Deadline repository. With a Slave working on each job, the overall render time is greatly reduced! Multi-Slave rendering is a great choice when you have to deal with multiple independent jobs, or a single job with a large number of tasks to complete.
Sometimes your Slaves will wind up converging on a single job. While this means that job will complete quickly, it also means that other jobs will be neglected. You can use Pools and Groups to ensure that specific Slaves work on specific jobs. Check out this Deadline Feature Blog entry for more in-depth information on Pools and Groups.
Similar to how a single Slave running multiple concurrent tasks only uses 1 Deadline Slave license, multiple Slaves running on the same machine will together only use 1 license, provided all the Slaves are on the same operating system. This means that you can have a group of Slaves on a machine work on a single job, and only 1 license will be checked out.
For more information on using multiple Slaves, be sure to visit the documentation.
Combining Multiple Slaves with Concurrent Tasks
Once you have mastered Concurrent Tasks for a single job and multi-job rendering pipelines with Multi-Slave rendering, it’s time to combine both of these concepts and see what we can come up with. If I have a group of jobs which have concurrent tasks enabled, I can deploy multiple Slaves to render multiple jobs at once, and each Slave in turn will pick up multiple tasks at the same time!
In the above screenshot, I have 2 Arnold jobs running and they both support 3 concurrent tasks. I also have 5 Slaves running on one machine and ready to go. Notice that 3 Slaves are now working on the first job and 2 are working on the second. Taking concurrent tasks into account, this means that 9 frames of the first job are being rendered at once, and 6 frames of the second job are also being rendered at the same time. When I combine multiple Slaves with concurrent tasks, not only can I render multiple jobs at once, but I can also render a large number of tasks at once within each job.
When combining concurrent tasks with multi-Slave rendering, it is important to note that the licensing rules mentioned for concurrent tasks or multiple Slaves still apply. This means that multiple Slaves on the same operating system can each pick up concurrent tasks of a job, and only 1 Deadline Slave license will be checked out overall!
Concurrent Tasks and Multi-Slave Rendering are two easy approaches to improve the efficiency of your Deadline experience. However, both of these tools do have tradeoffs that need to be considered.
The first thing to consider is that having concurrent tasks in a job does not always mean the job will be rendered faster. Suppose a Slave picks up 3 concurrent tasks. In most cases, this means that your Slave will complete all 3 tasks in approximately the time it normally takes to complete 1. However, if your Slave is already at its capacity with 1 task, then taking on 3 tasks would offer no worthwhile performance gain over running 1 task at a time. Ensuring your Slave has the processing power to take on additional tasks will help you prevent this drawback. You can set your own limits for concurrent tasks on each of your Slaves by right-clicking that Slave in your Monitor and editing the “Concurrent Task Limit Override” in the Slave’s properties.
The second tradeoff to keep in mind is the fact that the performance of your Slaves is highly dependent on available network bandwidth and storage. This applies both to concurrent tasks and multi-Slave renders. The performance of your machine and your network connection both factor in to the quality of your rendering pipeline. High amounts of multi-Slave rendering and task concurrency can lead to enormous streams of data passing through your network, testing the limits of your network service. Data caching both locally and on your network is one solution to avoid network bottlenecks. In order to get a proper understanding of your machine and network performance, it may be worthwhile to run benchmarks on the applications you will be using in your render workflow.
Even with these tradeoffs in mind, Concurrent Tasks and Multi-Slave rendering are still very strong approaches to keeping your Deadline pipeline moving fast. Provided that your Slaves take advantage of concurrency limits, and are working within their performance capabilities, you can be confident Concurrent Tasks and Multi-Slave rendering together will maximize the efficiency of your Deadline Slaves.
That covers the advantages and reasoning behind concurrent tasks and multi-Slave rendering. Concurrent tasks allow for Slaves to pick up multiple tasks of a job, allowing for faster execution of a single job. Multi-Slave rendering particularly allows for more than 1 job to be picked up at once on the same machine. Multiple Slaves can also be used in the context of a single job. Concurrent Tasks and Multi-Slave rendering work together to create an efficient, streamlined Deadline experience for you and your pipeline!