Emergent World Representations Exploring a Sequence Model Trained on a Synthetic Task

https://arxiv.org/abs/2210.13382 Task is comming up with valid othello board moves. Predict othello board state from hidden state activations. State of each cell is empty & black & white. linear prob doesn’t work but one layer MLP leads to 98% accuracy. lets change worldstate manually and see if predictions match our changes. To do this they first change the world model from top five layers like the image below.

how to intervene in one worldmodel, using only the probing model that predicts the worldmodel?

They applied SGD to the activations in a way that the color changes to the other color. After this, the model is giving the answer of the altered worldmodel with a very high accuracy which means this worldmodel is causal to the model prediction and the model Uses this worldmodel for its prediction.

latent saliency maps

They measure how much each cell affects the probability of a prediction. red->important, black square -> destination. For each cell they intervene it and measure the change in the probability.