Musicological Interpretability in Generative Transformers 

2023, Yale University

​

Project Overview

This project investigated what a simple autoregressive transformer model could learn about harmonic structure in music. Working with my colleagues in a collaborative lab setting, I reimplemented a version of NanoGPT to work with symbolic musical input. We then analyzed how the model's internal representations evolved as it consumed more data. 

 

The central question was epistemological and mechanistic: as a transformer learns to generate stylistically coherent music, what kinds of structural abstractions form internally, and how does this emergent knowledge of harmonic grammar compare to the stages of human musical learning? Rather than evaluating output quality alone, we tracked changes in embedding geometry, representational clustering, and functional role differentiation across training.

 

Our analyses showed that harmonic abstractions emerge progressively in a statistical order (reliable harmonic moves stabilize first). These changing abstractions reorganize the model's embedding space in structured ways, clustering into patterns that eventually reflect functional harmonic roles (e.g., tonic and dominant). Although, within these clusters, we found many examples of nuance, where the model stabilized some rare chord transition and placed it in an unexpected functional meta-category. 

 

This work demonstrated that even relatively small generative transformers develop internal representations that align with higher-level structural regularities in musical chord progression data. The project also marked my first sustained engagement with transformer interpretability and representation dynamics.