Version: Deadline 8.0
Have you recently added more Slaves to your farm and are finding that the Database has difficulty handling the load? Is your farm relatively small, but communication with the Database seems to take longer than it should? These issues and more can be resolved by adjusting your farm’s Performance Settings.
The default Performance Settings in Deadline are set for a farm with 200 Slaves. If your farm has fewer Slaves, then decreasing the different performance values will allow you to enjoy quicker response times from your farm! If you have more Slaves, then increasing the time values will prevent Slaves from competing for resources and will improve the response/lock time of your Repository and Database.
Manually Changing Performance Settings
You can configure the Performance Settings to fit the needs of your farm from the Deadline Monitor while in Super User Mode. Just select Tools -> Configure Repository Options -> Performance Settings.
There are three tabs and a button on this page, as you can see. The first tab covers the data that the Monitor reads from the Database.
You can tweak the values (in seconds) for each collection that the Monitor tracks (Jobs, Slaves, Balancers, etc). These collections are used for populating the panels in the Monitor, and the data is pulled from the Database (or the Proxy Server if you are running it). Since these retrievals can be time consuming on large data sets, trying to perform them too often can impact the Database.
For example, if you have 1500 Slaves, perhaps you don’t need to refresh the Slave data every 20 seconds. Instead, you could bump this number up to 30 or 40 seconds to help reduce the load on the Database. In addition, if you aren’t running applications like Pulse or the Balancer, you can increase their update values. These kind of tweaks can allow you to improve the performance of the Database while still frequently updating the collections that are important to your workflow.
As mentioned above, most of the Monitor Performance Settings are tied explicitly to a specific panel. If there are panels you do not use, you can increase the times for those options:
- Number of seconds Between Job Updates -> The Jobs Panel
- Number of seconds Between Slave Updates -> The Slaves Panel
- Number of seconds Between Pulse Updates -> The Pulses Panel
- Number of seconds Between Limit Updates -> The Limits Panel
- Number of seconds Between Cloud Updates -> The Cloud Panel
- Number of seconds Between Balancer Updates -> The Balancers Panel
- Number of seconds Between Proxy Server Updates -> The Proxy Servers Panel
The one exception is the Number of seconds Between Settings Updates option, which is not tied to any specific panel. This option is for retrieving changes to global Deadline settings, such as:
The second tab covers the data that the Deadline Slave reads from and writes to the Database.
The first option, Number of seconds Between Slave Information Updates, corresponds to how often the Slave updates its information in the Database. These updates are important, as other Deadline applications that rely on information about the Slave read that information from the Database. One of the places where this data is pulled is in the Monitor’s Slave panel, so there are effectively two different update intervals for the Slave panel:
- Number of seconds Between Slave Updates: How often the Monitor checks the Database to see if Slaves have updated.
- Number of seconds Between Slave Information Updates: How often the Slave updates the information in the Database.
Number of seconds Between Queries For New Tasks While the Slave is Rendering is used when a Slave is rendering a Job and finishes a Task. Increasing this timeout could help prevent issues where too many Slaves are trying to retrieve Tasks at the same time and are competing for access to the Database. Typically, this value can be left alone, or increased slightly if necessary. Note that changing this setting adds additional overhead to each Task, which can add up fast in the event that your renders are typically fast.
Multiplier to determine seconds between queries while the Slave is Idle does not explicitly refer to a time value, but rather a multiplier to be used when calculating how long a Slave should wait between checks for Jobs when it’s idle. The time that a Slave will wait is calculated by multiplying this value by the number of idle Slaves in the farm. So if you have 200 idle Slaves and the multiplier is 0.15, then the time each idle Slave will wait before checking for Jobs is 30 seconds (200 X 0.15). This timer is one of the only values not affected by Auto Adjust, which we will cover later.
The time idle Slaves wait before checking for Jobs is also influenced by Maximum number of seconds between Job queries while the Slave is Idle and Minimum number of seconds between Job queries when the Slave is Idle. Regardless of the dynamic timeout calculate above, it will never be higher than the maximum value and will never be lower than the minimum value. This system can be very beneficial as it ensures the following:
- If your farm is idle, the multiplier with the minimum and maximum values ensures that the Slaves are not all hammering the Database trying to find Jobs when there are none.
- Since the multiplier is based off of the number of idle Slaves, it will dynamically decrease as the farm gets busier and increase as the farm becomes idle.
The third tab covers the data that Deadline’s Event Plugin system reads from the Database.
The only option here, Number of seconds between checking Event Plugins for updates, is used by the Deadline applications to determine when they need to update their Event information. This process is can be time consuming, as it involves checking files on the Repository machine to see if they’ve changed as well as information in the Database. Most Deadline applications start an Event Sandbox process, so keeping this value large will reduce how often these applications read this data from the Repository. Note that this value is also not affected by Auto Adjust (see below).
Managing all the above options can be time consuming itself! In order to improve your life performance, Deadline has the Auto Adjust button.
Pressing this wonderful button will present you with this delightful screen. Very neat.
You’ll notice that the number of Slaves in the farm is set to 200. As mentioned earlier, the default Performance Settings are set for a farm of that size. If you have more Slaves in your farm, try increasing that Slave count. Let’s assume my farm has 2000 Slaves:
Look at those time values grow! The Previous Value column shows you what is currently set for each timeout and the Adjusted Value column shows what the settings will be changed to for a farm of your size. You can tinker with these updated values right in the Auto Adjust window. Let’s say you do not care about Pulse, Cloud, Proxy Server, Limit, Settings and Balancer updates in the Monitor but Job and Slave updates are important to you:
Will this perform better than our suggested defaults for 2000 Slaves? It easily could! The default values are an educated guess, and your hardware setup may accommodate lower time values or may require higher time values. Deadline will make an initial estimate, but it is up to you to determine what is best for your farm.
Alternatively, if your farm only has 100 Slaves:
Look at the time being saved! Now your Slaves will look for Jobs faster and your Monitors will update their data faster. Again, make sure to monitor your Database to determine how you can fine tune these values.
Wrapping Up Time
So there you have it! Performance Settings are just one tool you can use to improve your farm’s performance. For more information on the topics covered here, and other steps you can use to squeeze a bit more performance out of your farm, check out our Network Performance Documentation.