When Will This Job Start Rendering? It Depends...

Version: Deadline 7.2, 8.0

Background

Dependencies in Deadline

Often times jobs have requirements that must be fulfilled before they should start rendering. For example, a previously submitted job is related to the current one and must be completed, or frames in a render job depend on the frames in an export job. The existence of assets like caches might need to be verified or maybe your studio is super unique and has complexity than can only be solved by Python scripting. Jobs that are assigned dependencies will have the status of “Pending” and will not run until the dependencies have resolved, no matter how much you beg.

When Are Pending Jobs Allowed To Run?

You may be wondering to yourself, how will my jobs finish if you don’t let them run? To answer your hypothetical question, we do allow your jobs to run (they would be pretty useless otherwise!) by taking a looking at its dependencies. We accomplish this by running a Pending Job Scan from Pulse, or from a Slave if no instance of Pulse is running, or manually in the Monitor. To read more about configuring Pending Job Scan options please consult your Repository configuration.

This scan will look at each dependency on the currently pending jobs. If all dependencies of the job are met, then the scan will release the job (status will change to Queued). If any dependency for a pending job fails, the job stays in the pending state, and all of the dependencies will be checked again in the next scan. And the next one. And the next one... Until finally the job is deemed fit enough to run!

Note that if you are not running Pulse, and all of your Slaves are always busy rendering (which is a good thing!), then it will sometimes feel like a Pending Job Scan takes forever to release a pending job. To remedy your busy farm having no availability for other duties, it’s recommended to run our Pulse application, or you can always execute the Pending Job Scan via a Monitor on any machine (from the Tools menu while in Super User mode).

How Does One View Dependencies?

Don’t blame yourself if you can’t quite remember which dependencies you’ve given your job. Dependencies can become complex quite fast, which is why we recommend adding the Job Dependency View Panel to your Monitor layout. If it is part of your layout already, why are you reading this section? Skip it! So assuming the panel is not in of your layout, it can be found under “View” > “New Panel” > “Job Dependency View”. Selecting a job will update the view and draw all of its related jobs (nodes) and dependencies (edges) on the graph. You can also reorganize the nodes by moving them around, create new dependencies, and delete old ones.

By default, it only shows job names and the dependency connections, which is why we allow you to show the information that’s pertinent to you. This is accomplished by clicking the highlighted button in the image below.

This will bring up the following dialog where you can choose the properties you wish to display on the nodes.

More info about the Job Dependency View Panel can be found in the Job Dependency View documentation.

You Get A Dependency, And You Get A Dependency! Everyone Gets A Dependency!

If you want to use dependencies, I suppose it would be helpful if I told you how to actually add them to jobs. Well the easiest way to give jobs a dependency is through a submitter by clicking on the super descriptive ”...” button, beside the Dependencies field. This will bring up the following dialog where you can specify which job(s) it will be dependent on.

However, this doesn’t lend itself to customization of the actual dependency. To have more freedom with dependencies for a specific job, you can right-click on a submitted job, select “Modify Job Properties”, and select “Dependencies” on the left. This will be the main screen that’s used when dealing with all the different dependencies. At the bottom you have the option to change the settings for all the dependencies for the job and on the right you the ability to override those settings at the individual level. More info regarding all the dependency options can be found in the Job Dependency Options documentation.

You Promised My Job Would Start! But It’s Still Pending!

So your job has been pending for a while, eh? As I previously mentioned, jobs won’t be picked up by a Slave until the Pending Job Scan clears the job of its dependencies. This means that you currently have a dependency that isn’t resolving and is thus preventing it from being rendered. Now, you could check the dependencies manually yourself or you can be smart about it and use our handy dandy dependency test!

In the Job View Dependency panel, click the “Test Connections” button in panel toolbar and it’ll run all the tests and show you which ones are failing! This will help you visualize why particular jobs are not being released from the pending state. This is important if you use complex workflows such as this one...

The results are colour-coded, with green meaning the test has passed, red meaning it failed and yellow meaning it’s frame dependent and the test for some frames has passed. Hopefully this will also prevent people from trying to make circular dependencies...

Which Dependency is Right for Me?

In the introduction above, we dropped some hints about the different types of dependencies that Deadline supports. Now it’s time to take a look at them! The different types of dependencies are as follows:

  • Job Dependencies
  • Frame Dependencies
  • Asset Dependencies
  • Script Dependencies

Job Dependencies

The most common type of dependency, not just because we need paying jobs to eat, but because they’re built into all of our submitters! So why would you care about how to attach dependencies to jobs? Well let’s say you submit a 2x2 Tile Rendering job and a Draft Assembly job, this means that you have 4 jobs (1 for each tile) and 1 job for assembling the 4 renders from each frame into one larger image. Since it’s literally impossible to assemble images that don’t exist, it doesn’t make sense to allow the job to start until the tile rendering is done. If you’re still with me, we can handle this situation pretty easily by configuring the assembly job to have 4 job dependencies (the 4 tile rendering jobs). When those four jobs are complete, the assembler can begin combining the images. Here’s the graph associated with this example.

Lucky for you, our submitters already take care of Tile/Jigsaw assembly dependencies automatically, so let’s take a look at a manual example. Let’s say that you want to make a QuickTime movie after all the tiles have been assembled. Well, you can do that easily with this type of dependency. In the QuickTime submitter you can specify the dependency in the associated field or, if you forgot to do that, you can modify the job properties afterwards by doing the following. (Alternatively, use our Quick Draft UI to submit a dependent Draft job automatically and handle the dependency).

Navigate to the dependencies for the QuickTime job in “Modify Job Properties”. Under the “Job” tab in “Dependencies”, click the self-explanatory “Add” button and select the assembly job. You’ve now added the job dependency and thus you’ll see the QuickTime job attached like this in the Job Dependency View if you reload the panel (choosing the new QuickTime job, or clicking the toolbar option):

Frame Dependencies

That’s right! Sometimes you need something more flexible than job dependencies. Don’t believe me? Consider the following workflow: You’re using Houdini to export IFDs and then wish to render the subsequent output with Mantra (similar workflows include Redshift, Arnold, etc). I hope you’re thinking, “why not just use a job dependency since you just said how easy it was”, and I’d tell you that you’re right that a simple job dependency would work here, but we can do better.

In Houdini, for every frame that you wish to render with Mantra, there’s an associated IFD file that specifies how that frame should be rendered. So the actual rendering of a frame just depends on the IFD for that frame, not on the whole exporting job. Thus, we can use frame dependencies to handle this more granularly! This means that you’re able to have both jobs rendering at the same time! Hooray for efficiency! To enable this, you can select the “Use Frame Dependencies” option in the Job Properties window under the Dependencies tab I talked about 2 sections ago. Don’t tell me you’ve already forgotten... Anyways, the Job Dependency View will look like this on submission:

After rendering a few frames from the export job you’ll notice that the frames that were exported had their tasks released and rendered while the rest are pending:

However, you may have noticed that Deadline actually already supports this exact workflow described above! When submitting the Houdini job from the Monitor, or integrated submission script, you can specify to submit a “Dependent Mantra Standalone Job” which will do all of this automatically, which saves you the hassle.

Asset Dependencies

I’ve just told you about frame dependences, so how could asset dependencies be different or useful in any meaningful way? Well, the easiest way to convince you is if you use software that isn’t yet support in Deadline (let us know and we’ll add it to the wish list!), but you’d still like a job to start once you’ve received output from that process. You can’t handle this with any of the previous dependencies, but if you know what the output will be, you can use asset dependencies to schedule the job accordingly.

To add this dependency in the Job Properties window, just switch the Asset tab and click Add and specify which file/files you wish it to be dependent on. If Deadline detects frame numbers in the file, it’ll ask if you wish to replace with frame padding. In the previous example, if we replaced the frame dependency with this asset dependency it would like this and have the same result:

How asset dependencies work is that they perform an existence check on the file(s) that you specify. If the Pending Job Scan sees that the file(s) exists and it can access it (as in it has the permissions and isn’t locked by a process), it’ll resolve the dependency. You can even use this to get similar functionality to frame dependencies by specifying a file with frame padding, as it’ll replace the padding with the frames it’ll be rendering.

Script Dependencies

Script dependencies are where things really start to get interesting because if your studio has requirements that the other dependencies don’t handle, you can make it happen if you know a little Python scripting. Deadline handles script dependencies by running the Python script during the Pending Job Scan and releases the dependency only if the script returns True.

Adding a script dependency is like adding any of the others, you navigate to dependencies in the Job Properties window, click Add and select the script you wish to run. An example script a studio might want to execute before a job can render would be to check the file sizes of output, and if it’s within their acceptable level, return True to release the dependency. Here’s an example script that shows this:

from Deadline.Scripting import *

def __main__( jobID, taskIDs=None ):
    if taskIDs is None:
        # Frame Dependencies are therefore disabled
        job = RepositoryUtils.GetJob( jobID, False )
        
        for dependency in job.JobDependencyIDs:
            output_files = RepositoryUtils.GetJob( dependency, False ).OutputFilenames
            for file in output_files:
                if FileUtils.GetFileSize( file ) < 10: # bytes
                    return False
    
    return True

You can also replicate this behaviour by submitting a Python script job and making your job dependent on the script job. You can read more about Job Scripting here.

Putting It All Together

Up until now I’ve been nice and only used examples with one dependency at a time, but they don’t need to be. Dependencies aren’t isolated from each other, and one job can have multiple different types of dependencies, or you can have multiples jobs connected by different dependencies. Using them in conjunction with each other may yield you something like this:

It should be clear now that dependencies are great! They give you a lot of flexibility in determining when jobs run and help you get the most out of your render farm, so be sure to take advantage of them! Besides, no one wants to wake up at 2am to read an email notifying them a job completed and then submit a job because those assets are now ready. If it’s 2am for you now, go back to bed! Deadline’s got this covered!