Incrementing Our Way To Glory

As we mentioned in the June update, we spent most of June rewriting how Yarn Spinner calls your commands and functions, on the way to one of our most-requested features ever: actions that can take whatever parameter types you want, including your own. Doing it meant going deep on incremental source generators, and we learned enough painful lessons along the way that we figured we should write them up. This one’s for the programmers, especially anyone building, or thinking about building, a Roslyn source generator of their own.

How it works today#

When Yarn Spinner for Unity or Yarn Spinner for Godot (C#) encounters a command or function in your dialogue, it currently looks the method up in a dictionary of delegates and dynamically invokes it, using C#’s reflection APIs. When an action is registered, we scoop its method up into a delegate and hand it to a library; when your dialogue runs the action, the library combs through its dictionary for a delegate with a matching name, and we dynamically invoke it.

This has worked fine for years, but it has real downsides.

First, dynamic invocation is slow. Your dialogue is rarely going to be the thing causing performance hitches, but we don’t want to develop Yarn Spinner with a “who cares” attitude either.

Second, reflection comes with platform-specific rules about which APIs are allowed where. We don’t currently have problems there, but when you go to make changes it’s very easy to call an API that’s banned on one platform or another without realising, and suddenly you’re deep-diving through platform-specific docs, trying to puzzle out what you did that made it work everywhere but there.

Finally, it’s a lot of code, and in many respects it has to be, because you’re fighting the idea of C# itself. The types of the method and its parameters are gone; what’s left is a weird function you push values into and get values back from. But it’s still a method with a defined shape (C# won’t let you call it with any number or type of parameters), so you end up with code that sometimes feels very dynamic and at other times highly rigid, all in the same place. Arrays of objects going in, an object coming back, fingers crossed that everything got boxed correctly, and even more work again if the command is async. It works, but it’s a mess.

The new approach#

So rather than trying to improve reflecting on methods, and playing issue whack-a-mole as it comes up: why not statically call them?

Instead of dynamically invoking a delegate, we generate ordinary C# code that converts each parameter from the Yarn side and calls your method directly. The design goals:

1. No reflection.
2. Support custom types as parameters in actions.
3. Allow instance as well as static invocation.
4. Simpler action invocation code in the runner.
5. Better diagnostics around problems in actions.

Because the generated code knows exactly which method it’s calling, a lot of things become possible that weren’t before. The big one: any type you like as a parameter. Your method signatures are no longer limited to strings, numbers, bools, and a handful of engine types. If Yarn Spinner already knows how to convert something (say, a GameObject, found by name), or you’ve given it a converter for your own type, you can just use it.

Here’s what that looks like. Say you’ve got a type of your own, and you give it a converter: a static method, marked with an attribute, that knows how to turn a value from your Yarn script into an instance of your type:

public class Thing
{
    public int number;

    [YarnConverter(typeof(Thing))]
    public static bool TryConvertThing(string name, out Thing thing)
    {
        thing = new Thing();
        if (name == "yes")
        {
            thing.number = 9;
            return true;
        }
        return false;
    }
}

With that in place, you can write a command that just takes a Thing:

[YarnCommand("do_thing")]
public static void CommandWithConverter(Thing t)
{
    Debug.Log($"doing some stuff with a thing: {t.number}");
}

…and when your dialogue hits <<do_thing yes>>, the converter gets called and the result is injected into your method. This works for any arbitrary type you want, so long as you define a converter for it. (This is related to an existing feature: you’ve always been able to do this with GameObjects, which are sort of implicitly a string, since they have a unique name in the scene. Now it works for everything.)

On top of that, we know ahead of time whether a method needs awaiting, default parameters can be handled properly, error handling gets simpler, and the generated code is something you can put a breakpoint in and actually follow.

Enter the source generator#

The catch: code like that is specific to your game and your commands. The current system is generic and works with basically anything people come up with; the new one has to be generated per-project. The thing that does that is a source generator: a C# class that reads your syntax and compilation, and adds generated files (not on disk!) into the compilation pipeline.

We already ship a source generator (it’s part of how actions get registered and how .ysls files are produced), so the plan was to extend it to also generate the command and function invoker. The problem: that generator was quite old, and used the previous source generator API. Roslyn now recommends that all source generators be the newer incremental source generators, which perform better, have a more explicit workflow, and play nicer with external editors (fixing a longstanding issue affecting older versions of Visual Studio). The tradeoff is that they make you jump through considerably more hoops, and those hoops were the bits we didn’t understand when we started. Wow, have they caused a lot of headaches.

A quick primer. An incremental source generator has two steps. First, collection: you set up filters that scan through the syntax and compilation for the things you care about; in our case, Yarn actions, found via attribute or via calls to AddCommandHandler/AddFunction. Then, generation: using what you collected, you write out the final invoker, literally the source code that converts values from Yarn into parameters and calls your methods, and the generator adds that virtual file to the compilation as though it were a normal C# file.

That’s the theory. Here’s where the month went.

Issue the First: diagnostics and caching#

Something that was absolutely not clear at the start: you basically should not generate diagnostics from inside a source generator.

To report a diagnostic you need a location (the position in the C# file that’s causing the issue), and a location is something that can’t be cached. Even a single newline or space changes the location of every symbol after it, so if you’re collecting actions and storing their locations, every single keypress invalidates at least one of them and forces the generator to run again. Caching the results of earlier runs is how generators stay performant. Cache is king.

Learning this forced a huge rewrite of the entire generator, splitting it into a generator (which makes the invoking code) and an analyser (which reports problems with your actions).

Issue the Second: the workflow#

The incremental workflow is actually pretty neat: set up filters, collect, generate. The trouble starts when the results of one collection need to influence another, because you don’t control when anything runs. You set up filters, and Roslyn runs them when it needs to. Each collection step is atomic, and the concept of one step happening “before” another isn’t really a thing that makes sense.

An example: custom converters. Now that a parameter can be nearly anything, we need to know what method converts it, but the part of the generator that finds converters runs on its own cadence, independent of action collection. So when an action has a parameter type we don’t recognise, it might be a convertible type, and we won’t know until later. It’s solvable, but you end up carrying around potential actions instead of just checking up front.

Even little things that feel easy, like eliminating duplicate actions, get surprisingly tricky. If we find a command called do_thing, there’s no obvious place during collection to ask “have I seen a do_thing before?”

Issue the Third: analysers are smart#

Analysers are very clever about how often they run; if they can work out that they don’t need to, they won’t. This has interesting implications for error reporting.

During testing, we had a project that depended on another (the other assembly held a converter needed by an action). All fine, until the dependency had an error-level diagnostic in it, at which point the diagnostics in the main assembly silently stopped appearing. It took quite a while to track down, because we were intentionally generating errors on some critical issues, and accidentally preventing the next step from running.

In hindsight it makes complete sense: why bother performing full analysis on an assembly whose dependency has errors? How could you trust the results? But it means we can’t use error-level diagnostics for problems with actions and converters, which is quite annoying.

Issue the Fourth: the docs#

Source generators and analysers are complex, powerful tools, so nobody was expecting hand-holding, but what documentation exists is quite lacking. In particular there’s very little about the philosophy of generators: just how aggressively you’re meant to chase cacheability, and how seemingly-harmless things (like reporting diagnostics during generation) are at odds with how they’re intended to work. The real guidance lives in GitHub issues, where the “oh no, you should never do that” gotchas play out one thread at a time. Puzzling them out took a while.

Issue the Fifth: logging#

The issues above are complex systems with weird quirks. This one is just plain bad. Logging inside a generator is difficult; opening a file is difficult; connecting a debugger is difficult; following what happened is difficult. As far as we can tell, everyone eventually gives up and opens a log file in a way Roslyn can’t complain about, defeating the entire point of the rules that try to prevent it.

Where we’ve ended up#

After all of that, we now have a nearly-working system where you can:

1. Define converters, so your own custom types work as action parameters.
2. Call methods on instances as well as statically.
3. Get much better error messages about any concerns in your actions.

The generated invoker knows about your actions and your converters, and handles correctly invoking your methods when they turn up in your Yarn. And you shouldn’t have to think about any of it: just add the appropriate attributes, and write your dialogue.

This will ship in future releases of Yarn Spinner for Unity and Yarn Spinner for Godot (C#).