Why do function word frequencies vary across individuals? Evidence in favour of The Statistical Approximation Hypothesis

A modern unsolved mystery in applied linguistics is that the frequency of function words can be effectively used to identify the author of a text. Computational methods to identify authors were developed by computer scientists or statisticians who chose to focus on function words because of their high frequency and topic obliviousness (Mosteller & Wallace 1963). However, linguistically speaking, it does not make sense to say that authors differ in their function word frequency because of personal preference. Another reason why this explanation does not make sense is that, as corpus linguistic evidence has always suggested (Sinclair 2004), the basic unit of language is not the word but something larger and more similar to a phrase, a claim compatible with cognitive linguistic and cognitive psychology evidence on language processing (Christiansen & Chater 2016; Langacker 1987).

In this paper I introduce a formal theory of linguistic individuality based on cognitive linguistic principles that could offer an explanation in the form of the Statistical Approximation Hypothesis. This hypothesis says that the frequency of function words reflects the distribution of function words in our repository of linguistic units, which, according to usage-based frameworks, should be unique because of the idiosyncratic process of entrenchment (Schmid 2015). If this hypothesis is correct, then authorship analysis methods based on this principle should outperform or perform equally as methods based on function word frequencies. Preliminary evidence based on analysis of existing benchmark corpora for authorship analysis indeed shows that this is the case. New methods simply based on the presence or absence of n-grams perform as well as a sophisticated machine learning or deep learning algorithms that consider frequency of function words. This result is interpreted as indirect evidence in favour of the proposed hypothesis.