Groups, Pools & You

Version: Deadline 8.0

Background

Deadline’s Pools and Groups are really nice ways of organizing the Deadline Slaves in your farm. When Jobs are submitted they are put into Pools and Groups. The Slaves will select Jobs to render based on which Pools and Groups they’re assigned to. Today we’re going to learn how to manage both Pools and Groups, but before we get started I suggest you read this great article that explains, in detail, the differences between Pools and Groups, and how they’re intended to be used. This article also covers Limits but this post won’t be covering that.

If you don’t have enough time to read through all of that, here’s a very basic low down.

Groups

Slaves with similar hardware, installed software, etc, should be placed in the same group. The order of the groups DOES NOT matter. Groups do not effect Job Scheduling, rather they are a way of quickly of filtering out a group of Slaves.

For example, Slaves with 16 cores and 32 GB of RAM might be grouped together. Or Slaves with Maya installed will be in the ‘Maya’ Group. Slaves with 3ds Max installed will be in the ‘Max’ Group. Slaves that have both installed can be in both Groups.

Pools

The order of Pools DOES matter, and they are typically used for prioritizing projects. Slaves will prefer jobs from the Pool in the first position and then second etc.

For example, let’s say you’ve got 30 Slaves and 3 projects on the go. ‘Project A’ is really important and needs the most resources so we want 2/3 of our farm to work on it. ‘Project B’ and ‘Project C’ are less important and only need 5 Slaves at any one time. To accomplish this we’ll need to assign 20 Slaves to the ‘project_a’ pool, and 5 Slaves each to the ‘project_b’ and ‘project_c’ Pools respectively. To ensure that we don’t have any idle Slaves we can assign all of the Pools to the Slaves and just make sure we have the correct Pool order set up. The 20 that will forcus on ‘project_a’ might be assigned to Pools like this: ‘project_a’, ‘project_c’, ‘project_b’. 5 Slaves will have their assigned Pools look like this: ‘project_b’, ‘project_a’, ‘project_c’. Finally, 5 Slaves will have their assigned Pools like so: ‘project_c’, ‘project_a’, ‘project_b’.

Permissions

To create and assign Pools/Groups, you’ll need to have Super User permissions. To do this, select Tools -> Super User Mode in the Monitor, and then enter your Super User password (if one is required).

Creation and Assignment

The Manage Pools and Manage Groups dialogs are under Tools option in the Deadline Monitor, or you can right click on one or more Slaves in the Slave List. Note that the right-click option will pre-filter the list of Slaves in the dialog to only show the Slaves that were originally selected.

On the left hand side of both dialogs you will find a list of all of your Pools/Groups. Below that is the option to create new Pools/Groups. Click ‘New’ and enter a unique name for your Pool/Group.

The next step is to assign your newly created Pool/Group to some Slaves. Select the newly created Pool/Group in the list of the left and then select the Slave(s) you wish to assign to that Pool/Group from the list on the right.

Below the list of Slaves is a number of operations you can use to manipulate the Pools/Groups assigned to Slaves. Click on ‘Add’ to assign the selected Pool/Group to the selected Slaves. Note that Slaves can be assigned to multiple Pools/Groups.

There are also options for removing a Pool/Group from a Slave, clearing all Pools/Groups assigned to a Slave, and copying and pasting a Slave’s assigned Pool/Group to other Slaves.

In the Manage Pools dialog, There are a couple Pool specific operations called ‘Promote’ and ‘Demote’. These operations allow you to change the priority of the selected Pool for the selected Slaves (with top priority Pools on the left, and lowest on the right).

For example, let’s say the ‘mobile-024’ Slave has the ‘simproject’ Pool at top priority and the ‘compproject’ Pool at second priority.

If you select the ‘compproject’ Pool and the ‘mobile-34’ Slave, and then press the ‘Promote’ button, the ‘compproject’ Pool will be promoted to first, and ‘simproject’ will fall to second. After this change, ‘mobile-034’ will now prefer ‘compproject’ jobs first, and will only move on to ‘simproject’ jobs if there are no ‘compproject’ jobs left.

The Purge

At some point you might find yourself in a situation where you no longer need a particular Pool/Group. Pools/Groups that have been deleted are not automatically removed from Slaves. Luckily for us there’s an option to Purge obselete Pools/Groups from all the Slaves and from all the Jobs. To do this select the Purge checkbox in the bottom left corner of the Management dialog. Additionally, you can replace the deleted Pool/Group with another Pool/Group by selecting the new Pool or Group from the drop down below.

Here I’m Purging a deleted Group (refered to as an Obselete Group in the dialog) and replacing all the existing assignments of the deleted Group with the ‘openstack’ Group.

Tips

The ‘none’ Pool/Group is a special case. It’s the default Pool/Group that all Jobs get put into if they aren’t assigned to an existing Pool/Group. Normally, any Slave can pick up Jobs in the ‘none’ Pool/Group but you can turn that behaviour off in Tools -> Repository Options -> Slave Settings and enabling Exclude Jobs in the ‘none’ Pool/Group. You might want to consider enabling this feature if you’re worried about a Slave picking up a Job that’s in the ‘none’ Pool/Group right before a Job is submitted into one of the Pools/Groups that Slave is assigned to. The Slave will have to finish that ‘none’ Job before it moves on to the new, probably more important, Job.

Having an ‘All’ Pool can be really helpful to avoid the hassle of assigning every Pool to all of your Slaves. They’ll allow you to manage a lot of your farm very quickly with secondary Pools. If you want your Slaves to focus on a particular Pool but also always be working on something then I suggest assigning your Slaves like this:

Slave: ImportantPool, All

Then all of your Jobs can be submitted in the ‘All’ Pool as their Secondary Pool. A Job’s Secondary Pool is a “fallback” Pool for the Job. Slaves will do an initial pass of the queue using each Job’s Primary Pool. If the Salve doesn’t find anything then it will make a second pass using the Secondary Pool. In our example, Slaves will prioritize Jobs in the ‘ImportantPool’ but if there’s none in there then they’ll start picking up Jobs whose Secondary Pool is All. What’s nice about this is that if you add new Slaves you won’t have to assign all of the existing Pools to them, you only have to assign the ‘All’ Pool. Likewise, if you add a new Pool, you won’t have to assign it to all of your Slaves.

Note: Secondary Pools only work with a Job Scheduling Order where Pools is the most important factor in scheduling. Deadline is set like this by default.

At Your Command

Pool/Group managemenet can all be done through the command line with Deadline Command.

Here are some of the commands for Pools. All of the commands for Groups are the same except there are no ‘Promote’ and ‘Demote’ commands, just replace the word Pool with Group in the commands.

Note: Most of these commands can be used on multiple Pools/Groups using comma separated lists. Delete is singular and only 1 Pool/Group can be used to replace Purged Pools/Groups.

To add Pool(s):

deadlinecommand AddPool <name_of_new_pool1>,<name_of_new_pool2>

Delete a Pool:

deadlinecommand DeletePool <name_of_pool_to_delete>

Purge obsolete Pools:

deadlinecommand PurgeObselete <pool_to_replace_purged_pools> (Optional)

Get a list of Slaves in a Pool or Pools:

deadlinecommand GetSlaveNamesInPool <pool_name1>,<pool_name2>

Set the Pool(s) of a Slave(s). You’ll have to use this command to change the order of the Pools. You can use this command with a cron job to change your Slave’s priorities based on the time of the day.:

deadlinecommand SetPoolsForSlave <slave_name1>,<slave_name2> <pool_name>

Similarly, you can do AddPoolToSlave and RemovePoolFromSlave.

The End

That pretty much covers how to manage your Pools/Groups. Now that summer is almost here, it’s time to grab a group of friends and head down to the pool.