The v0.23.1 Save/Load Bug Breakdown

I promised it and it's finally here, a detailed look at the save/load bug that popped up in v0.23.1 when it was first released. This post is going to be a look into the technical details of Lab Rats 2 and Python that caused the bug, and may accidentally contain some wisdom about code design. If that interests you, read on!

First, let's cover what the bug actually was. For the first two days after the release of v0.23.1 almost all games would crash when you attempted to save. The stack trace produced by the crash very clearly pointed towards an issue with infinite recursion - some part of the program was calling itself with no end.

A bug in the actual code for the save/load system of Ren'py seemed unlikely; one of the reasons I've used Ren'py as an engine is that it handles all of the saving and loading for me seamlessly and so far without fail. The most obvious suspect was the new Limited Time Event system that I had added during the week between the patron-only bug testing release and the public release. Some testing confirmed that the game only entered it's infinite loop when at least one character in the game had a Limited Time Event assigned to them, so clearly this is where the issue was.

What are Limited Time Events, then? LTE's are random events that are tied to a person and trigger when you enter the same location or talk to that person. They have to have a "limited time" to stop rarely interacted with characters building up a ton of random events, which in turn lets me crank up the rate the events are generated so often interacted characters have them often as well. Put another way, the goal was to have these random events trigger roughly proportionally to how often you interreacted with a character. From a conceptual point of view LTE's are very similar to the random events that were already supported by a game, but from a programming point of view had some special issues. Each random event object is a singleton, meaning it exists only once and has the person of interest passed into it when it is triggered. This prevents a bunch of duplicate events from floating around taking up extra memory, and it makes it easy to check if two events in a list are the same. The problem is that these singleton events cannot have a timer attached to them, because that timer would be incorrectly shared by everyone with that event.

The solution to the timer problem was to create a wrapper object. Each wrapper had two things: a reference to the event it represented, and a countdown. When a girl was assigned a LTE this wrapper would let me keep track of when to remove it without having to copy the entire random event object. I wanted these wrapper objects to "look" like random event objects to my code so I wouldn't have to add in special cases everywhere. To achieve this I used one of Python's "magic" functions, so-called because they are used internally by Python and not intended to be called by other things. Generally these functions start and end with double underscores and include things like __init__ and __hash__. The magic function I was able to make use of was __getattr__, which is called any time you ask for an attribute of the object but it doesn't exist. I set the __getattr__ function to take any incoming function call and pass it to the underlying random event. This was an elegant solution that had my wrapper class looking and behaving just like a normal random event, except when I needed it not to.

When tested this all behaved exactly as I expected, so where's the bug? Well it turns out Ren'py's saving system interacts with __getattr__ as well. It probes each instance to figure out what attributes it needs to save, which often results in it asking for attributes that don't exist. Normally this causes an AttributeError to be thrown, which the save module catches and uses to know that there's nothing there. My __getattr__ function didn't throw that error, so the save module didn't know when an attribute was missing or not, so it would just keep asking and asking and asking until it hit the recursion limit and crashed.

The fix was dead simple: before passing a __getattr__ call to the underlying random event the wrapper object first checks to see if that attribute exists in the random event. If it doesn't it throws an AttributeError exactly as expected. It took to research to figure out, but this is the kind of bug I like to run into. Implementing the Limited Time Event system gave me a chance to play around with a cool bit of python, and fixing the bug taught me a lot about how it can go wrong and how __getattr__ in particular often interacts with other classes behind the scenes. Going forward I should probably save major code changes for the next patron release, but I'm still glad I got the system coded up and pushed out as soon as I did.

I hope someone enjoyed this breakdown of this bug! Work on v0.24 continues, and I've just started a batch of new image renders using my re-written rendering automation.