Visualising Large Language Model Activations

In the field of natural language processing, large and complex neural networks, particularly transformers, have become essential due to their effectiveness in handling text data. However, despite their success, current methods for visualizing how activations of specific network units influence model outputs remain underdeveloped. This gap hinders interpretability and limits researchers’ and practitioners’ understanding of neural network behaviors during training and inference.

This project aims to address this issue by creating a robust visualizer tailored for transformer architectures. By processing training or test data, this project seeks to track and analyze changes in attention scores and other structural components of the transformer network, visualizing their evolution within individual input sentences during inference. The project involves developing innovative visualization conventions to compactly represent the high-dimensional neural network processes in a manner that is analytical and accessible, through the usage of static diagrams.