BeanCounter

// ERR: Fault Detected

We had recently switched our render-farm software to Pulze RenderManager (now RenderFlow) for its interoperability with their SceneManager tool. Unfortunately, this broke the previous system we had at the studio for performing analytics on completed jobs, and was further complicated by a series of updates that made job metadata inconsistent.

// NOTE: Due to a migration to a newer render-farm manager, this project was left in beta. I am currently working to finish it and sanitize an example dataset for publication.

// Solution

I set out to build a desktop application that could perform the same functions as our old system, as well as allow for some user-directed cleanup of the data. After digging into the working directory for the render farm, I determined that all metadata was being stored in JSON files and that these files were retained after completion. From there it was a matter of building out class structures in Python to store the relavent info and doing additional processing to organize it into projects, by render-node, etc. The required features were collecting job metadata (frames rendered, average frame time, etc), automatically organizing them by project, presenting them to the user in a concise manner, and being able to save/load from a database. Requested features were estimating power consumption, cost, and providing insight to individual machine's contributions.

// Breakdown

// Overview

BeanCounter works by scanning the "archive" subdirectory of the Render Manager's working directory for .JSON files, parsing each one into a "Bean" (a single render job) and adding its data to internal memory. These are automatically organized into Projects based on metadata in the JSON, and are flagged for user review if it's inconsistent. Once all jobs are processed the user is asked to resolve or manually sort any flagged jobs, and after that they're presented with a list of all Projects and some high-level statistics. Clicking into any project will display a thorough breakdown of the jobs it includes, sortable by name, date, render time, etc. Alternatively, the user can see a breakdown of how each render-node performed on this project. If the user thinks that a project is erroneously labeled, they can merge it into another. Once they're satisfied with the high-level organization, the projects can be committed to the XML database, merging with those already saved. The full database can then be analyzed in the same way as the newly-added jobs.

// Counting Beans

The BeanCounter expects to be pointed at the current working directory for RenderManager, which includes subfolders for active and "archived" (inactive) jobs. It scans this archive directory for JSON files which store metadata for each job, parsing them and storing them internally as "Beans," distilling the JSON down to 10% their original size in memory. Depending on the type of job (animation, still, or distributed) some fields need to be processed differently. The Beans' internal logic attempts to resolve these as robustly as possible, standardizing them internally for later use. Each Bean also stores a complete record of individual node performance, making aggregated stats on a Project level as simple as tallying up each job within it.

// Sorting Beans

Beans are sorted automatically into Projects, based off of the project entry in the Bean's JSON. This is compared to the prefix of the 3D file that was submitted, if there's a mis-match it's flagged for user review. Due to a persistent bug in the RenderManager software that caused it to read the second section of the filename as the project name, these mis-matches occur often and it's usually safe to treat BeanCounter's interpretation as cannon. In case neither method works to correctly resolve the project name, the user can manually merge one (or multiple) projects together.

Projects are, broadly speaking, just a collection of individual jobs. Using the precalculated statistics stored in each Bean they're able to tabulate this and provide a high-level overview.

// Bean Breakdown

Each Project can be expanded into a thorough breakdown of the jobs within it, or the performance of each render-node that worked on it. The jobs view can be further expanded to show the performance of individual nodes on a specific job. Statistics here on CPU hours and power consumption are based on a config file, assigning names and hardware specs to each IP address, or default values if no match is found. Configs are stored with a valid_on date field and are only applied to jobs that ran after the given date. This allows for hardware changes overtime without invalidating the statistics of older jobs. For calculating power consumption and CPU time, it is assumed that the node was fully occupied (as is often the case) with the task of rendering each of its frames. Thus, these estimates are as simple as multiplying the number of cores in a machine with the time spent working, and likewise for its wattage.