Abstract

This work examines artificial neural language model behavior through the lens of morphological productivity. Specifically, we ask: given a corpus of linguistic data, (a) can we predict what morphological generalizations a transformer language model trained on this corpus will learn, and (b) is this learned behavior predicted by any existing theories of morphological productivity in humans? To interpret the qualitative behavior of these neural language models, we define a measure of generalization behavior by comparing the relative conditional probability distributions over syntactically-licit completions of uninflected forms. We then use this measure of generalization behavior to compare language model behavior to the predictions of three common models of morphological productivity: Baayen’s P*, Fragment Grammars, and the Tolerance Principle. We offer three main conclusions: (1) None of the three formal models of morphological productivity surveyed are good models of neural language model behavior. (2) Despite this negative result, we observe that the qualitative behavior of neural language models is notably similar to the kinds of predictions made by a different class of formal theories: Maximum Entropy Harmonic Grammar (MEHG) models. (3) We further investigate why neural language models learn the generalizations that they do, concluding that the observed behavior results from models learning a log-prior distribution over next-tokens which is affine in token frequency rather than logarithmic. While this can be interpreted as evidence of model miscalibration, we demonstrate that this relation, which arises as a consequence of the training objective, is identical to the stipulations placed on the harmony measure of MEHG models. Consequently, we show that the similarity between neural language models and MEHG models is not a coincidence: while it is well-known that neural models are a form of the more general maximum entropy models (without harmony), we provide here a novel demonstration that neural language models can be analyzed as highly-parameterized MEHG models whose harmony measure is an affine function of distributional properties of the training corpus.

Bib(La)TeX Citation