Saliency Systems and State Mutation

Recently we had a question about saliency systems, and I wrote up a quick example about a quirk that might not be immediately obvious about Yarn Spinner’s story saliency from the outside. In the end, it turns out that wasn’t at all the issue they were having, but it’s still an interesting problem, so I thought it worth expanding it out into a bit more detail in case other people find it useful.

The Core Issue

The saliency system must first check everything to see what could run, then run what was picked (if anything). If there’s any way that the relevant game state or variables can change during that checking process, you’ve fallen into a trap. This post walks through exactly how that trap works and how to avoid it.

The Problem

Let’s say we have a function that returns something important about the state of our game, and we register it so Yarn scripts can refer to it later:

int counter = 0;
void Start() {
    runner.AddFunction<int>("game_state",() => {
        counter += 1;
        return counter;
    });
}

In this case it’s pretty simple and doesn’t really do anything, but imagine it represents something very relevant about the state of the game our dialogue needs to know about.

It’s important here to pretend that this state is doing things that can’t be easily encapsulated or represented in Yarn variables—we’re just using a single incrementing number here to keep this blog post small and tidy.

This could be moving transforms or updating animations as part of its work, something very non-dialogue based, or alternatively something that would be annoying to do in Yarn directly. Maybe it’s the player choosing which character gets hurt and returns how many hit points they have left—could be anything.

Now in our Yarn scripts we can start making use of this game_state() function:

title: Start
---
Jumping to the node group
<<jump TheGroup>>
===

title: TheGroup
subtitle: Fallback
when: always
---
I am the fallback, this means game_state() didn't return 1 or 2 or 3!
===

title: TheGroup
subtitle: One
when: game_state() == 1
---
The function returned 1
<<jump TheGroup>>
===

title: TheGroup
subtitle: Two
when: game_state() == 2
---
The function returned 2
<<jump TheGroup>>
===

title: TheGroup
subtitle: Three
when: game_state() == 3
---
The function returned 3
<<jump TheGroup>>
===

Fantastic dialogue here, and fully reactive to the current game state, with each possible game state we care about getting its own storylet. So when we run this, we’d probably expect to see this:

Jumping to the node group
The function returned 1
The function returned 2
The function returned 3
I am the fallback, this means game_state() didn't return 1 or 2 or 3!

However, if we actually run this, something strange happens. We instead get this:

Jumping to the node group
The function returned 1
I am the fallback, this means game_state() didn't return 1 or 2 or 3!

Hmm, bit weird. Maybe a cosmic ray hit our RAM and flipped a bit—it can happen, after all. Let’s run it again:

Jumping to the node group
The function returned 3
I am the fallback, this means game_state() didn't return 1 or 2 or 3!

Hang on, why did TheGroup.Three run now, and we still only have a single one being shown before the fallback runs? What is going on? If you run this a lot, you’ll start to see there’s no reason to it. It looks like each of TheGroup.One, TheGroup.Two, and TheGroup.Three have an equal chance of being run, and then the fallback node runs. We’re gonna have to trace this.

Becoming The Yarn Spinner Saliency System

When we hit the first <<jump TheGroup>> inside of Start, the saliency system takes over and will collect all potential candidate nodes in the target node group. In this case, we have four:

• TheGroup.Fallback
• TheGroup.One
• TheGroup.Two
• TheGroup.Three

Now we need to evaluate the conditions for each candidate before we can select one—time to pretend to be a computer.

For each candidate, we need to know three things:

1. PassingConditionValueCount: the number of its conditions pass right now.
2. FailingConditionValueCount: how many of its conditions don’t pass right now.
3. A ComplexityScore: something analagous to how specific its conditions are.

OK, up first is TheGroup.Fallback and its when clause is when: always. OK, easy—that is always true, so it has passed its only condition. As for its complexity, we know when: always has a fixed complexity of 0, and it’s the only clause here, so our first candidate is done:

• TheGroup.Fallback: (PassingConditionValueCount: 1, FailingConditionValueCount: 0, ComplexityScore: 0)

For this blog post, because each storylet only has a single condition on it, it’s going to be easier to keep it all in our heads as a simple true or false for if a candidate’s single condition passed or not. So a nice shorthand for our now-simplified candidate (passed all its conditions and with a complexity of zero) is:

• TheGroup.Fallback: (true, 0)

Next up is TheGroup.One. Its condition is when: game_state() == 1, so let’s run that.

So when game_state() runs, counter gets incremented from zero to one and returns 1. 1 == 1 is true.

Now for its complexity: this is a single expression with no boolean operators, so using the formula from the callout above, it’s num_operators + 1 = 1.

So at this stage, if we were to check our candidates, we have:

• TheGroup.Fallback: (true, 0)
• TheGroup.One: (true, 1)

Everything is looking fine. Let’s keep going.

TheGroup.Two has the condition when: game_state() == 2, and we expect that to fail because it’s the first time this node group is entered. Let’s find out.

OK, so when game_state() runs, counter gets incremented from one to two and returns 2. 2 == 2 is true, and will also have a complexity of 1.

Hang on, what?

And if we do this with TheGroup.Three, when it gets evaluated, we also see that game_state() increments, compares 3 == 3 (which is true), and also has a complexity of 1.

At this point you’ve probably worked out what’s going on, but let’s keep going through. In the end we have the following potential candidates:

• TheGroup.Fallback: (true, 0)
• TheGroup.One: (true, 1)
• TheGroup.Two: (true, 1)
• TheGroup.Three: (true, 1)

Now the saliency strategy takes over. The goal of the saliency strategy is to take multiple potential storylet candidates and pick zero or one of them.

The default saliency strategy is Random Best Least Recently Seen, which will select:

• The Best - one or more candidates with the highest complexity score, which theoretically hold content that is the most specific and contextual to the current game state.
• From those choices, the Least-Recently Seen - effectively penalising points from those you have seen already based on an inverse of how many other pieces of content you’ve seen since. This means even if something is a “Best” candidate, you won’t get the same content repeating itself if possible. The first time a node group runs, this factor doesn’t affect the decision as the penalties are all 0.
• From those choices, a Random choice - so even if the Best is always the same, and even if it’s the first time you’re running this node group so all recency penalties are 0, you will still get a variety of options if there are multiple to choose from.

It sounds like a lot, but let’s apply these rules to our example node candidates.

• TheGroup.One: (true, 1)
• TheGroup.Two: (true, 1)
• TheGroup.Three: (true, 1)
• TheGroup.Fallback: (true, 0)

The very first move it makes is to eliminate any candidates that have failed their conditions, and any that don’t have the highest complexity score. In this case, all nodes passed but the fallback is the least complex and is eliminated:

• TheGroup.One: (true, 1)
• TheGroup.Two: (true, 1)
• TheGroup.Three: (true, 1)

These are the Best candidates.

The next step is to deprioritise any that have been seen already, so that we can choose the Least-Recently Seen. But we haven’t seen any yet, so our list of candidates remains unchanged.

These are the Least-Recently Seen candidates.

At this point we have three equally good options, so the final step means it’s time to pick one at Random. In this case, let’s go with TheGroup.Two.

And we’ve almost finished playing the role of the computer. Now the TheGroup.Two storylet runs, and as a part of that, we do another jump back into the TheGroup node group.

Now when the saliency system runs, the candidates would be quite different than when the checks first happened—the values from game_state() will never match any of the when clauses, so our candidates sent over to the saliency system this second time will look like the following:

• TheGroup.Fallback: (true, 0)
• TheGroup.One: (false, 1)
• TheGroup.Two: (false, 1)
• TheGroup.Three: (false, 1)

So after the first step of the saliency strategy, it will eliminate all but the fallback node, which will be chosen by default.

OK, so what has happened?

You’ve probably seen it—or maybe you saw it at the start—the game_state function mutates saliency-relevant state. Because we have three different calls to it, each time we evaluate the conditions on the nodes we’re modifying the state.

That we didn’t just immediately get the fallback node is entirely a quirk of how I intentionally ordered the nodes in the Yarn file. If we rearranged it so that the ordering in the Yarn script was TheGroup.Three, then TheGroup.One, and finally TheGroup.Two, the saliency system would find and check them in that order, and all three would fail their conditions and be eliminated immediately.

Fixing It

This raises the final question: how do we fix this? There are two ways you can go about this, and neither is realistically better than the other, so go with whichever you find works best for you and your game.

Approach One

The first is to stop mutating state as part of the node being selected. This means the first thing we need to do is modify our game_state function so it doesn’t mutate any state:

int counter = 0;
void Start() {
    runner.AddFunction<int>("game_state",() => {
        return counter;
    });
}

Now when it runs, we’ll always get the same result: the current game state.

But the mutation was clearly an important piece of the game logic—we still want it to change. So we’ll have to make a new way to do this mutation after the saliency process.

The easiest way is to have a new command to do this:

[YarnCommand("mutate_game_state")]
public static void Mutate() {
    counter += 1;
}

And the final piece: in our storylets, we need to call this command to do the mutation as part of the node execution:

title: Start
---
Jumping to the node group
<<jump TheGroup>>
===

title: TheGroup
subtitle: Fallback
when: always
---
I am the fallback, this means game_state() didn't return 1 or 2 or 3!
===

title: TheGroup
subtitle: One
when: game_state() == 1
---
The function returned 1
<<mutate_game_state>>
<<jump TheGroup>>
===

title: TheGroup
subtitle: Two
when: game_state() == 2
---
The function returned 2
<<mutate_game_state>>
<<jump TheGroup>>
===

title: TheGroup
subtitle: Three
when: game_state() == 3
---
The function returned 3
<<mutate_game_state>>
<<jump TheGroup>>
===

Aaaaaand done. Now this works.

The advantage of this approach is it’s very flexible and took almost no time or extra code to make work, but does mean you need to be diligent in calling <<mutate_game_state>> at right time, every time you’re about to jump. There is nothing in the code which attaches it to the node selection process.

Approach Two

The second approach requires a bit more work but does maintain that connection between the two operations and gives you the most control; it will involve making a custom saliency strategy.

Now we don’t want to have to fully recreate all the features of the existing Random Best Least Recently Seen strategy, so our strategy in this example will just be a wrapper around it.

int counter = 0;
IContentSaliencyStrategy existingStrategy;
void Start() {
    existingStrategy = new RandomBestLeastRecentlyViewedSaliencyStrategy(runner.VariableStorage);
    runner.AddFunction<int>("game_state",() => {
        return counter;
    });
    runner.Dialogue.ContentSaliencyStrategy = this;
}

This sets us up to be a wrapper around the normal Random Best Least Recently Seen strategy.

Because we said we’re now conforming to the IContentSaliencyStrategy interface, we’ll need to implement its required methods:

public ContentSaliencyOption QueryBestContent(IEnumerable<ContentSaliencyOption> content)
{
    return existingStrategy.QueryBestContent(content);
}
public void ContentWasSelected(ContentSaliencyOption content)
{
    existingStrategy.ContentWasSelected(content);
    // did we just pick a node from TheGroup? node group
    if (content.ContentID == "TheGroup") {
        counter += 1;
    }
}

Here we’re mostly just forwarding the calls to the existing strategy, but when ContentWasSelected is called with a node from the appropriate group, we perform the mutation we previously did inside the game_state() function. This part only happens after the saliency strategy has done all its checks and picked a node to run, and is now reporting back which one it picked (so it can do things like update those Recently Seen penalties).

This is where a lot of the power and flexibility of this approach comes into play, because you decide what gets passed onto the built-in system before or after selection, or completely change the selection process itself. You can filter or adjust candidates on the fly, mutate or undo mutations, or totally ignore certain pieces of the saliency strategy. You could do something like our Custom Saliency Strategies sample does, and implement a weight node header field. The sky is the limit!

But for this example, it’s fine to just not worry about it and forward everything to the existing saliency strategy.

Aaaaaand We’re Done

Hopefully this has provided a bit of insight into ways to think about saliency, and also helped head off a potential—very annoying to trace—bug.

Expect more posts about Yarn Spinner and saliency in the near future.

In the Meantime, Check Out These Docs

Want to learn more about Yarn Spinner’s saliency system? Here are some helpful documentation pages:

• Saliency Overview - Complete guide to saliency strategies and how they work
• Storylets and Saliency Primer - Introduction to node groups and storylet-based narrative design
• Creating Custom Saliency Strategies - How to build your own strategy (like we did in Approach Two)
• Node API Reference - Technical details on node properties including complexity scores

Header image: Screenshot of Skyrim “Arrow in the Knee” line.