Update 52: Failure

So, I missed my deadline. Even with a conservative ‘double the expected time’ estimate, I still missed it.
It threw me off a little, I’ll be honest. We treat failure so negatively in our society that it’s hard not to let it get to you. ‘Don’t get it wrong’ is so ingrained into our upbringings, our education, and our careers. Easy to see why it sticks. 
Even when the thing that went wrong was missing a mostly arbitrary deadline.

In this update I’ll talk about the shiny new equipment (which some of you have expressed an interest in), then move onto the NSFW previews.

The New Setup


I’d finally gotten my new computer. As mentioned, it came from a system integrator, so the first thing it needed was to come out of the default case and into the rack.
Noctua are the manufacturer of my second favourite fans (the first being you guys <3)

Same chassis as the rest of my nodes, it's gotta match the rest!

The finished build, cool, quiet and monsterous.

I’d been deliberating over which of these machines would serve as my main workstation, then landed on the obvious answer: a KVM switch. That’s exactly what they’re for. Getting the wiring to make sense took a little effort, which put my considerable sketching skills to the test.
Definately didn't revise this a bunch of times

Yes the wiring is a bit messy, but, well, it IS a wiring cubby

Simple USB switch for swopping peripherals that I didn't want to follow the KVM

Cats always seem to know where they're most inconveniently placed at any one time

Whilst everything was out in the open, this was a good opportunity to investigate the issues I’d been having with a node that had been crashing and not recovering mid-render. With a whole new machine joining the fleet and the rack now full, I could simply pull the offending machine, but I was curious what was actually wrong. It’s a pretty fierce machine that I could sell to recoup some of my investment, but I wouldn’t want to sell faulty hardware without disclosing it.
Plan was from the top: Node X4 then Workstation, then second workstation/fun PC

I took the opportunity to strip down, clean and rebuild all the nodes, replacing the bulky coolers that meant every hardware issue required removing the GPU and CPU cooler just to get to the RAM. Not ideal when you’re wanting to do a full test.

Pic taken mid dissassembly, this was the node that was failing

One of the nodes post rebuild, remove motherboard, replace cooler, swop PSU's x4

Now you may have guessed it, but the RAM was the problem. An exciting future within the clutches of a failing RMA system awaits me there.

Tried the sticks in a new machine, instant fail

I took the time to reconsider the OS the nodes ran, stripping them down to a minimal, headless installation. As well as being very efficient, interacting with machines in a terminal makes me feel like a cool hackerman.


I also added some spacers to the sound panel on the front of the server cabinet (still couldn’t get away with the glass, boo) which gave just enough of a gap to keep all the nodes, and most crucially the jet-engine equipped switch, tamed during renders.
The result was a rendering experience that was actually pretty peaceful. I took a moment to admire the calm whirr of the farm as it crunched numbers, rendering scenes as I completed them, with nary a lost frame, crashed node, or anxiety over overheating. It had literally never been as quiet, cool, and efficient. From where I was last year, this was a really triumphant moment.
I spent more time refining my render management system, fixing bugs, adding features, a little bit of tuning. Making sure everything is stable and ready for when something needs to render, or I just want to see a few preview frames in not a lot of time.
Being able to pause a node mid render and resume it, godlike functionality

The Actual Failure


With two thirds of scenes down already, I approached the final scene with gusto. Shiny new super fast machine at my fingertips, the return of a classic setup (with some revisions), and a simple set of sims to do. What could go wrong?
Things started out pretty routine. The sim looked great in Houdini:

When I imported the result into Blender, however, I was getting clipping.
Where you see the lighter shading is where the mesh is clipping through the internal surface, it doesn't look like a big deal here.

However when you add refraction and everything else to it, it makes it look really odd


Clipping happens a lot in animation and is generally fine, since you don’t see the artifacts it produces. In this case though, with a transparent surface, it just looked really odd. This falls into the category of ‘unacceptable failure’ in my eyes.
So I tried running the sim on more accurate, un-remeshed collision geometry. Also no. Then I tried exporting the remeshed sim directly, No again.
Bear in mind that even with the new fancy machine, there are still hours between iterations. I lost weeks to this problem, and I’ll be honest, it wore me down. All that preparation, all that equipment, and I still couldn’t nail it.
The issue turned out to be something completely stupid and entirely unpredictable, fixed with an equally stupid workaround. C’est la vie.

I’ve been piling stress on top of what is already a stressful phase of production. By this point I’ve been staring at the same animation for months, and all I can see is errors. The internal narrative shifts from ‘This is so exciting!’ to ‘Was this ever interesting?’ and ‘Are people going to like it?’

This even has unintentional glitches in it

The only reliable solution I’ve found is to step away for a time, let my barometer reset, and revisit with fresh eyes. Thinking about it further, I’m not sure this is a problem I can, or indeed want to, solve.
If I’m not failing, I’m not trying anything new. I’m not experimenting, I’m not doing the best job I possibly can at that moment. The expansion on the original idea was an attempt to bring you all something new and utterly unrecognisable from the preview I released last year. The lesson I’ve taken from this is that failure isn’t something you should punish yourself for. It’s an indication you’ve grown. It means you’re learning.
More glitches here too...


Luckily enough, my job is mostly done for now. The frames are with sound design and the writers for the finishing touches. Once done, I’ll revisit it one more time for the final stage of polish and presentation, and depending on those timescales this should be out in a couple of weeks.

This sim took WEEKS

Before I go, I do want to say a genuine thank you. To those of you who stick around through the quieter months, who leave comments, who drop ideas into the Discord. I read them, my support team highlights them, the community interacts with them, and more of them feed into the work than you might realise. 
The months where the work is grinding and set back after set back wears you down and progress feels invisible. They're tough, but they're made so much easier knowing there are people on the other side of it who are engaged, who care, and who are willing to wait. That's not something I take for granted.

I promise the wait will be worth it, this one is explosive.