Deadline 6 is the latest version of Thinkbox Software's render farm management system. It features an incredibly flexible and efficient User Interface across all platforms, combined with a new highly-scalable database backend. The new User Group Management system allows users to be split into groups, with different permissions assigned to each group. Farm administration is made easier with the ability to stream the render log from remote machines.
Note that a new 6.0 license is required to run this version. If you have a license for Deadline 5.2 or earlier, you will need an updated license.
New Database Backend
Deadline now uses a highly-scalable MongoDB Database to store data like the jobs, settings, and slave configurations. The new Database vastly improves Deadline's performance, while at the same time allowing it to scale to higher render node counts than ever before.
A trimmed down file-based Repository still stores files for the Deadline plugins, scripts, logs, and any auxiliary files (like scene files) that are submitted with the jobs. However, since the majority of data now lives in the Database, the demand on the file system and the network is significantly reduced.
Running Pulse is no longer required to improve Deadline's performance. Pulse is only required if you wish to make use of the advanced features like Auto Configuration, Power Management, Slave Throttling, Statistics Gathering, and the Web Service.
Redesigned User Interface
All Deadline applications have been rebuilt using Digia’s Qt development framework, giving Deadline a native look and feel across all operating systems. The new user interfaces were designed to provide greater flexibility and improved performance.
The Monitor introduces a wide range of new features:
- Delayed data loading allows the Monitor to start up quickly, even with thousands of jobs in the farm.
- Data (Jobs, Slaves, etc) is always live. There is no need to manually refresh the data anymore.
- Data is split up into separate panels, which can be re-sized, docked, or floated as desired. This allows for a highly customized viewing experience which is adaptable to the needs of different users.
- New filters make it fast and easy to filter out unwanted data. Filters can be saved, and different filters on the same panel can be viewed at the same time.
- Almost every panel has a graphical representation to visualize the data in a meaningful way.
User Group Permissions
The new User Group Management system allows Administrators to restrict what functionality is available to certain users, as well as make certain features accessible to others without requiring the use of the Super User mode. By default, all users are part of the Everyone group, so default permissions can be set at this level. New groups can be added with different permissions as required.
Slave Log Streaming
Connect directly to the Slave running on a remote machine and view its log in real time. This makes it possible to see what the Slave is doing without having to remote into the machine. The remote log caches the last 500 lines of output, making it possible to see what the Slave did in the past as well.
Slaves now only need access to a license when rendering jobs. If the license server goes offline, the Slave will continue to run instead of shutting itself down. However, the Slave won't be able to render another job until the license server comes back online.
Job Environment Variables
Environment variables can now be set for a job, and they will be applied at render time. During submission, the variables can be set individually, or they can automatically be set to match the current environment. The variables can also be modified from the Monitor. At render time, the variables can be used exclusively, or they can be merged with the Slave's current environment.
Customizable Script Menus
Right-click script menus are now available for almost every list in the Monitor. In addition, the text, icons, and display order for all script menu items can now be customized, and they can even be placed in sub-menus.
Basic Cloud Features
Basic Cloud features are now built into the Monitor, allowing it to connect to different cloud providers and provide a way to control existing instances. Start up, shut down, or even clone existing images to scale the farm up and down as necessary. Currently, only Amazon EC2 is supported, but more providers will be added in the future.
New Job Archiving System
Archived jobs are now removed from the Database and saved to a zip file in the Repository. The zip file will contain the job and task information, as well as the job's auxiliary files. Archived jobs can be imported again from the Monitor. A key difference from the old archiving system is that archived jobs are no longer managed by Deadline, so they no longer impact performance.
New Custom Folder in the Repository
A 'custom' folder has been added to the Repository, and it contains sub-folders for events, plugins, and scripts. Custom plugins and scripts can be dropped into this location, and Deadline will automatically append them to the plugins and scripts it finds in its default folders. If a plugin or script exists in both the custom folder and the default folder, the one in the custom folder will be used.
Nothing in the custom folder will be touched when upgrading the Repository, but it will still be backed up by the Repository installer.
All of Deadline's plugins and scripts now use the native Python bundle that is shipped with Deadline. The IronPython engine has been removed.
- Updated Client applications to use .NET 4 (Windows) and Mono 2.10 (Linux and OSX).
- The Client bin folder is no longer added to the PATH environment variable. On Windows and Linux, a DEADLINE_PATH environment variable is now defined. On OSX, a file called DEADLINE_PATH is created in /Users/Shared/Thinkbox.
- Some changes have been put in place to allow different Major versions of Deadline to run on the same machine at the same time.
- All python scripts that ship with Deadline now use spaces instead of tabs for indentation.
- Fixed some encoding bugs.
- The .NET Runtime installer is now bundled with the Windows Client and Repository installers, and will be installed automatically if it's not already installed.
- The Mono Framework is now bundled with the OSX Client and Repository installers, so it doesn't need to be installed separately.
- The Linux Client and Repository installers now accept a --monoexec command line option that you can use to specify the full path to the 'mono' executable. This is useful if Mono isn't in the PATH when you run the installer with elevated privileges.
- The Repository installers no longer overwrite permissions on the Repository folder.
- The Windows Client installer now adds App Paths registry keys for each of the Client applications.
- The Linux Client installer now has an option to run the Launcher daemon as a specific user.
- The installers no longer allow /usr or /usr/local to be chosen as the installation directory.
- The Repository installer now creates the initial plugin-merge log in the repository root folder.
- Most of Deadline's data is now stored in the Database instead of the Repository.
- To avoid potential permission issues, files in the Repository are only created or deleted. They are never appended to.
- The Client binaries are now stored in the Repository as a zip file. This makes automatic upgrades faster and more reliable.
- Job and Slave report files in the Repository are now compressed using bz2 to reduce disk space usage.
- The scripts folder in the Repository now contains folders for additional lists in the Monitor.
- The Pulse Web Service scripts in the Repository are now in scripts/WebService.
- Added a new 'custom' folder to the Repository to hold custom plugins and scripts.
- There is now a configurable upper limit on the number of tasks that can be created for a job. The default is 5000 and can be set as high as 100,000 in the Repository Options.
- Environment variables can now be set for a job, and applied at render time.
- Job failure detection is now enabled by default for new installations. The default failure limit is still 100.
- When an error occurs for a job, only an Error Report is saved for the job. Previously, both a Log and an Error report were saved, which is redundant because the Error Report contains the full log.
- If a job is suspended, a requeue report will now be created for any tasks that were rendering at the time.
- When deleting a job, only the job and its tasks are deleted immediately. The job's reports and auxiliary files are cleaned up later during housecleaning. This speeds up mass job deletions, as well as job deletions from a remote location.
Client Application Improvements
- The Linux and OSX Client applications now write their logs to /var/log/Thinkbox and /Library/Logs/Thinkbox, respectively.
- Replaced the '()' characters in the Client application log files with hyphens so that they're unix-friendly.
- All Client applications should now log unhandled exceptions to the log files.
- The "SocketOptionName" console warnings that appear on Linux and OSX for all Deadline applications is gone.
- Some keyboard shortcuts in the Client applications have been changed so that they don't replace reserved shortcuts on some operating systems.
- Updated the wrapper shell scripts for the Client applications on Linux to add the absolute path to the LD_LIBRARY_PATH instead of the relative one.
- Improved the resolution of the Client application icons.
- The Launcher notification icon is now on all three operating systems, and it replaces the Launcher interface on Linux and OSX.
- On OSX, the Client applications that are started through the Launcher now show their proper name in the menu bar.
- There are no longer duplicate entries in the Change Repository drop down.
- Added an option to the File menu to import archived jobs.
- There are no longer duplicate entries in the Change Repository drop down.
- Added an option to the Tools menu to configure User Permission Groups.
- Added an option to the Tools menu to remotely connect to Pulse's log.
- Added an option to the Tools menu to perform a full housecleaning, which replaces the individual options for checking for stalled slaves and orphaned limit stubs, while adding additional checks.
- Added a timeout option when sending test emails in the Repository Options.
- When adding a Drive Mapping in the Repository Options, any trailing path separators are automatically removed from the network path to avoid potential errors.
- Added option to the Drive Mapping system in the Repository Options to only map the drive if it is disconnected.
- Changed some list colors and removed the bolding of text.
- Layouts and list filters for the Monitor can now be "pinned" so that you can quickly switch between different layouts and filters.
- Almost all lists in the Monitor now have right-click script menus.
- The script menu items can now be customized in the Repository Options.
- VNC and RDC scripts are now available in many of the right-click script menus. This replaces the built-in remoting menu items.
- Added a new Cloud panel to show your cloud instances. Cloud settings can be configured in the Repository Options.
- Added a new Console panel that shows output as its written to the Monitor log.
- The Remote Command dialog is now a panel.
- The web service password in the user settings is now hidden by asterisks, and there is a field to confirm the password.
Jobs and Tasks
- The Job list now has a Quick Filter panel which can be used to quickly filter on a job's state, pool, group, user, and plugin.
- Double-clicking a job in the Job list will now bring up the job properties dialog.
- Double-clicking a task in the Task list will now open the image for the task and/or remotely connect to the Slave log. This behavior can be configured in the Monitor Options.
- The job properties dialog now shows the job's ID in a read-only text field. This makes it easier to copy the ID from the Monitor if you need it.
- Added options to the Job right-click menu to complete or fail a job.
- Added a column to the Job list to show the number of concurrent tasks for the job.
- Added a bit of extra whitespace in the progress column in the Job list between the complete % value and the X/Y count (where X is the number of completed tasks, and Y is the total task count).
- Added an option to the Task right-click menu to remotely connect to a Slave's log.
- Added an option to the Job Report right-click menu to remotely connect to a Slave's log.
- All reports (errors, logs, and requeues) for a job are now shown in the same Job Report list.
- A progress bar is now shown in the progress column for the Job and Task lists.
- Drag & drop job dependencies no longer trigger if any modifier keys are held down (SHIFT, CTRL, etc). This prevents accidental drops when multi-selecting jobs.
Slaves and Pulse
- Added an option to the Slave Settings to override the Slave's MAC address (if Deadline is retrieving it incorrectly).
- Added an option to the Slave right-click menus to remotely connect to a Slave's log.
- Added an option to the Slave Report right-click menus to remotely connect to a Slave's log.
- Double-clicking a slave in the Slave list will now try to remotely connect to its Slave log.
- The Slave Availability Filter is now called the Job Candidate Filter.
- The Status column in the Slave list now factors the render time into account when sorting.
- The Status Message column in the Slave list is now called Offline Message, and is meant to show why the Slave is offline (manual shutdown, power management, etc).
- When the status of a Slave is unknown, it will appear as Unknown in the Slave list, instead of New.
- When the status of Pulse is unknown, it will appear as Unknown in the Pulse list, instead of New.
- The Last Status Update column in the Slave and Pulse list is now a timestamp.
- Double-clicking a Limit in the Limit List will now bring up the properties dialog.
- The log window has been removed from the Slave user interface. It's been replaced with a button that allows you to spawn the log viewer. This prevents the log from affecting the performance of the Slave.
- Added right-click menu options to Slave log viewer to pause and resume the log.
- Added an option to the File menu to set the license server.
- Fixed a memory leak in the internal slave log.
- If the Slave can't initially connect to the Repository while in -nogui mode, it will keep trying instead of exiting immediately.
- The application plugin is now reloaded if the .dlinit or .py file changes in the Repository. This means a new job doesn't have to be loaded before the changes are recognized.
- Error messages raised by application plugins are now cleaner.
- Slaves only need access to a license when rendering a job.
- The Slave will now recognize if a job's concurrent task count increases while in the middle of a task.
- If an OutOfMemoryException occurs, the slave will try to restart itself. Note that this requires the Launcher to already be running on the same machine.
- The slave now double-checks the timeout setting for a job before throwing an error due to a timeout.
- On OSX, an external application is no longer used to check the bitness of the render executable.
- On OSX, an external application is no longer used to determine free disk space.
- The log window has been removed from the Pulse user interface. It's been replaced with a button that allows you to spawn the log viewer. This prevents the log from affecting the performance of Pulse.
- Added right-click menu options to Pulse log viewer to pause and resume the log.
- Pulse no longer acts as a proxy server for the Monitor and Slave applications.
- If Pulse can't initially connect to the Repository while in -nogui mode, it will keep trying instead of exiting immediately.
- Added some extra debugging output to the Job Repository Scan.
- Proper Python stack traces are now logged if an error occurs with a web service script.
- Submission now fails if two auxiliary files have the same file name but are in different folders.
- The GetJobsFilter and GetJobIdsFilter commands now work properly if searching for different values for the same property (ie: Status=Suspended and Status=Completed).
- The RenderNow command has been renamed to RenderJob, and it now requires a license to render a job.
- Added new command line functions to update specific settings for a job's machine limit.
- Added new command line functions to complete jobs, fail jobs, pend jobs, and release pending jobs.
Power Management Improvements
- Fixed a typo in the emails sent out for the Machine Startup feature.
- The Machine Startup feature for Power Management no longer tries to wake up slaves that are in a job's bad slave list.
- Start commands for Slave Scheduling in Power Management are no longer sent to disabled slaves.
- Deadline Statistics are now stored in the Database. The option to store statistics in a Postgres database has been removed.
- Normalized render times are now included in the statistics that are logged by Deadline.
- The Extra Info 0-9 values for a job are now saved in the job statistics.
- All of Deadline's plugins and scripts now use the native Python bundle that is shipped with Deadline. The IronPython engine has been removed.
- Python "print" statements now show up in the Client application logs.
- For application plugins, a PluginPreLoad.py script can be dropped into the plugin's folder. This script will run before the main plugin script or any of the job pre/post task scripts in the same scope. This will allow it to set the Python environment for those scripts.
- For event plugins, a PreLoad.py script can be dropped into the plugin's folder. This script will run before the main event plugin script in the same scope. This will allow it to set the Python environment for this script.
- For application plugins, the PreLoad.py script is now deprecated, and has been replaced with the JobPreLoad.py script.
- For application and event plugins, all global functions have been deprecated, and can only be used if "DeprecatedMode=true" is set in the plugin's dlinit file.
- The __main__ function for pre/post job and task scripts will now accept an instance of the DeadlinePlugin object as a parameter. If "DeprecatedMode=true" is set in the plugin's dlinit file, no parameters are passed to __main__.
- Added a new AbortRender(string, level) function to the ManagedProcess class to control what happens to the task when the render is stopped. The first parameter is the message, and the second ie one of the following:
- ManagedProcess.AbortLevel.Minor - requeues the task without an error being generated
- ManagedProcess.AbortLevel.Major - requeues the task with an error (like the current FailRender function)
- ManagedProcess.AbortLevel.Fatal - fails the task with an error
- ManagedProcess.AbortLevel.Success - completes the task
- Added a new SetEnvironmentVariable(string, string) function to the ManagedProcess class to set extra variables for that process only.
- Added a new ClearEnvironmentVariables() function to the ManagedProcess class to clear the extra variables. Note that this also clears extra variables set by the DeadlinePlugin's SetProcessEnvironmentVariable(string,string) function.
- Added functions to RepositoryUtils to get job and slave reports.
- Added a function to calculate statistics for a job.
- The LabelControl in the DeadlineScriptDialog now supports hyperlinks. Just specify the <a href="..."></a> tag in the label's text.
- Added new RepositoryUtils functions to update specific settings for a job's machine limit.
- Added RepositoryUtils functions to change the state of a job or task.
- Added a new RepositoryUtils function to change the job submission date.
Event Plugin Improvements
- Added a new OnJobDeleted event to the event plugin system.
- Removed the OnJobArchived event from the event plugin system, since Deadline doesn't manage archived jobs anymore. If the job is deleted when archiving it, the new OnJobDeleted will be called.
- Added option to Shotgun event configuration to disable SSL validation.
Application Plugin Improvements
- Render executable paths for plugins can now be relative. If they are, Deadline searches the current directory and the PATH environment variable to build up the full path.
- All plugins that support local rendering will no longer throw an error if the local output folder is empty. This is to prevent false positives when the renderer is writing output to a different location.
- The slave now suppresses the plugin warning messages that occur if a stdout callback is called while another callback is being processed (they will only appear in debug logging now).
- Updated some plugin icons.
- Added support for 3ds Max 2014.
- Fixed a bug when submitting from the Monitor that messed up the output path.
- Removed reference to Viz from the Monitor submitter (since Viz isn't supported anymore).
- Added support for 3ds Max 2014.
- Concurrent tasks now work with 3ds Max 2013 and later.
- SMTD has been refactored so that it can work as a standalone script or within RPManager.
- The RPManager client proxy script is now shipped with Deadline, instead of being a separate download.
- The RPManager submitter now includes a pass dependency feature that works like backburner.
- Fixed a machine limit bug in SMTD.
- Added a timeout when running the customize.ms script during startup.
- Fixed a bug when exporting a Maxwell setting that caused the final MXI file to be missing lighting.
- Cleaned up some weird file information that was logged after the scene was loaded.
- The Cancel button is now pressed on the "Image I/O" popup instead of Retry. Pressing Retry could "hang" the render if the file can never be saved to disk.
- Replaced Ryan's home folder the 3ds Max startup XML files with the 3ds Max installation folder.
- If 3dsmax crashes, we now try to figure out if it's a backburner-related issue, and include this in the error message if necessary.
- Added popup ignorer for GrowFX popup.
- Fixed a minor bug when setting the initial concurrent task value in the integrated script.
- Added support for Arion's RenderWarrior command line renderer.
- The default verbosity is now 4. It's the lowest verbosity that still shows progress.
- An error is now properly reported when Blender is unable to create the output directory.
- Fixed a bug that could cause Deadline to pass the multipass output path to Cinema 4D, even though it was disabled when the job was submitted.
- Added additional stdout handlers to catch errors.
- Removed a generic error handler from the plugin which tended to catch false positives.
- The executable no longer needs to be an absolute path. If it is relative, Deadline searches the current directory and the PATH environment variable to build up the full path.
- Added support for Composite 2014.
- The Toxik plugin is now called Composite.
- Added basic progress handling.
- Added the option to set the Limit for the Draft event plugin.
- The Draft plugin can now deal with multiple tasks with different frame ranges.
- Added support for Hiero that allows you to submit Nuke jobs from Hiero to Deadline.
- Resuming renders now work with Local Rendering enabled.
- The sampling level for co-op jobs is now computed at render time instead of submission time.
- Added option to ignore missing MXI files during a merge.
- Added option to combine co-op jobs into one job (the default behavior now).
- Added command line option to export extra channels.
- Added cleanup code to take care of extra channel files after a merge.
- Added support for Maya 2014.
- Added better support for the Krakatoa for Maya renderer.
- Added better support for the Octane renderer.
- Added better support for the Caustic Visualizer renderer.
- Added Path Mapping support for .ma scene files.
- When rendering a Mental Ray Export Job, the dependent Mental Ray standalone jobs now contain the layer name in the job name.
- The Arnold verbosity option is now available when submitting Arnold for Maya jobs.
- Fixed a bug where dependent Mental Ray Standalone jobs would have the layer name from the previous job in the job name.
- Fixed an "obsolete -renderType warning" in the integrated submitter.
- Added "Fatal Error:" message to list of fatal errors when strict error checking is enabled.
- Fixed a typo in the JobsDataDirectory property for script jobs.
- Fixed some tile rendering issues for Renderman jobs.
- The Monitor submitter now defaults the MayaBatch option to be on.
- Added a function so that script jobs in the MayaBatch plugin can access the job's ExtraInfo properties.
- Fixed a bug that prevented some scenes from being submitted if they contained referenced layers.
- Added a stdout handler to catch an Arnold error message.
- Reduced the verbosity of the warning messages when Deadline is ignoring an error.
- The "Thinkbox" shelf for the integrated submitter is now called "Deadline". You can delete the old "Thinkbox" shelf.
- Fixed a bug with how output paths were being passed to Deadline when submitting jobs with multiple layers and/or renderable cameras.
- All dialogs shown by the integrated submitter now set the main window to be their parent.
- Added support for modo 7xx.
- Added better progress reporting for modo 6xx and later.
- The job name is automatically set to the scene file's name in the modo submitter.
- Fixed an output pattern bug when overriding the output path that prevent per-pass images from being saved.
- Fixed an error when unloading the modo plugin that would occur if modo hasn't been launched yet.
- Tile rendering now works with Pass Groups.
- In the integrated submitter, DeepWrite and WriteGeo nodes are now supported when using the option to submit write nodes as separate jobs.
- The integrated submitter no longer crashes if there is an enabled write node with an empty output path.
- Shotgun settings for the integrated submitter now show up in their own Deadline tab in Nuke's project settings.
- The integrated submitter now printes out the stack trace if an error occurs while loading the CustomSanityChecks.py file.
- Added tooltips to the integrated submission script.
- The concurrent task setting is now sticky in the integrated submission script.
- You can now toggle the Batch Mode option from the Monitor in the job properties dialog.
- Progress reporting now works when Batch Mode is enabled.
- Increased the upper limit for the Maximum RAM Usage setting when modifying it from the Monitor.
- Added option in integrated submitter to submit write nodes as selected tasks. This can be useful if you have a bunch of write nodes in a Nuke script to output different Quicktime movies. You can enable this option, and bump up the Concurrent Tasks value to allow machines to process multiple write nodes concurrently. Since Quicktime generation only uses a single thread, you can get much better throughput with this option on multi-core machines.
- The REDAlert plugin is now called REDLine.
- There is now an integrated submitter for Rhino 5.
- Added check to ensure the output path contains "/" instead of "\".
- Added support for Softimage 2014.
- The XSI and XSIBatch plugins are now called Softimage and SoftimageBatch, respectively.
- Fixed a bug that caused the integrated submitter to fail after the job is initially submitted.
- Added the Opaque Opacity option to the Monitor submitter.
- Path mapping can now be performed on the contents of the vrscene file. This can be enabled from the VRay plugin configuration.
- The "Buffer Size" label no longer says "Gamma" when configuring the Vrimg2Exr properties from the Job Properties dialog.
- The submitter now lets you submit jobs if the source vrimg files don't exist yet.
- Added support for Vue 11.