Pumpkin

// ERR: Fault Detected

At this point in time, our studio was still using Autodesk Backburner to manage our render farm. While this software worked well for rendering video, for stills it required manually remoting in to each node, disconnecting them from the farm and enabling the "Distributed Rendering" service. As our workload grew and we began to run multiple types of projects simultaneously, this micro-management became more and more frequent.

// Solution

I conceptualized a small desktop app that would allow us to quickly see what mode each render-node was in (renderfarm or "distributed" rendering,) and switch between them. I had experience with UI design in PyQT already from plugin development, but had neither packaged an application for standalone use nor done any networking before, and had to research how to implement these. I decided to use Python early on as it had become something of a standard language in our studio, and learned about the socket library, which would allow me to implement the rudimentary network communications necessary for this.

I developed this into a system where upon opening a connection to one of the nodes the client would send a 32-byte hash as a handshake; if accepted by the node, requests for information and commands could be issued by transmitting 3-byte ENUMs. All communication over the network was handled in this manner, with actions only performed based on the ENUM commands sent, and replies handled within the ENUM structure as well.

Packaging for desktop use was handled using pyinstaller, both for the core application and for an installer/uninstaller. This sidestepped the issue of needing to maintain correct Python versions on each node and (more significantly) each user's PC, as it would be bundled with the rest of the dependencies.

// Breakdown

// Overview

Pumpkin consists of two main elements: a Client and Server. Servers are installed on render-nodes and configured to run in the background on startup, where they listen for connections on a predetermined port. These are responsible for starting and stopping other software on their machine, triggering a reboot, managing lock-state, and reporting the current state of their machine. All actions are performed only on request by a Client. Clients provide an interface for users, where they can manage a list of render-nodes, request status updates, toggle software on and off, and lock a node to prevent conflicts between users.

// Networking / Server

Pumpkin's networking ended up being fairly simple; the hard part was learning to use Python's low-level socket library with no prior experience. Upon opening a connection to a server, the Client first transmits a 32-byte "handshake" hash which is either accepted by the Server (signal.OK is sent and connection is kept alive) or rejected (signal.REJECTED is sent, connection is closed). This serves as a very basic form of security, but more so acts as version control -- the hash is updated with every release, and so prevents undefined behavior from client/server mismatches.

Once a connection is open, all communication is done with 3-byte ENUMs from the signal module, limiting the risk of corrupt or malicious packets. The worst that should be able to happen is a random (but pre-approved) action is performed. The Server will read 3-bytes off of the socket and compare it to a list of commands it can perform, typically confirming to the client with signal.OK when executing a function. Replies are similarly coordinated by sending a signal.READY to the client, waiting for their confirmation and then transmitting data. Each server would only allow one client connection at a time, and this would time-out after a period of inactivity. The connection could also be closed by the client, and usually would be after all requested actions had been performed.

Error checking was a considered and planned feature, but after using the program extensively over our LAN we didn't have any issues with data corruption. Given the low-stakes nature (a corrupt packet may cause a random action to be taken, or more likely for the server to simply close the connection) time was never allocated on this feature.

// Client Application

The client was a bit more complex than the server, with QT GUI instead of a text interface, multi-threading for communicating with servers efficiently, and a system for saving and restoring state. I had some experience with QT already from developing plugins in 3ds Max, but this was (at the time) the most complex setup I had to create. I learned to create custom widget classes for instantiating in the list of nodes, dialog boxes for confirmations, and techniques for updating the UI at runtime without crashing.

The user could populate the list of nodes using PC names or IP addresses, or pre-populate it using an XML file. A connection to each node would be made in a thread, utilizing QT Signals to allow async updates to the GUI. Parts of the UI would be dynamically locked and unlocked to prevent the user from performing additional actions while a node was being updated, and detailed feedback could be found in a Log tab of the UI. At-a-glance status of each action is conveyed to the user the coloring the Node entry in the main UI, with Green for success, Yellow for in-progress, and Red for an error.

Network communication is handled very similarly to the Server, with the main difference being the Client is responsible for initiating contact. After that, they simply attempt to complete the requested action and then release the server on completion.