How LLMs store facts

How might LLMs store facts

Not solved

How do Multilayer Perceptrons store facts?

Different directions encode information in Vector Embedding space.

MLP’s are blocks of vectors, these are acted on my the context matrix

Johnson–Lindenstrauss lemma

Sparse Autoencoder - used in interpretability of LLM responses

See Anthropic posts

• https://transformer-circuits.pub/2022/toy_model/index.html#adversarial
• https://transformer-circuits.pub/2023/monosemantic-features