Stylometry and the Pauline Letters – Guardians of the Sacred Text

Stylometry is the statistical analysis of authorial writing patterns through quantitative measurement of vocabulary choices, sentence rhythm, function-word preferences, and other quantitative stylistic features too subtle for the unaided reader to detect reliably. In New Testament scholarship, the principal application of stylometry has been to the question of which of the letters of the Pauline corpus in the New Testament were written by the same hand as the seven or so undoubted Pauline letters, and which were written by later followers working in the Pauline tradition. The application has, in particular, transformed the methodological landscape of the Pauline authorship question by replacing the older tradition of subjective literary-critical impression with reproducible quantitative measurement that can be repeated, refined, and contested by subsequent scholars.

The methodological promise of stylometry, as applied to the Pauline corpus, is that the substantive features of authorial style — the choices an author makes between alternative ways of expressing the same idea, the rhythms of sentence structure, the preferences for particular function words — are substantially stable across the texts an author writes, and that these features can be measured by computational means. The methodological promise is, in practice, qualified by a number of substantial difficulties, of which the most consequential is the small size of the Pauline corpus: the largest Pauline letter, at perhaps 7,000 words, is at the lower end of the corpus size at which stylometric methods have been shown to give reliable results.

The Pauline Corpus and the Authorship Question

The Pauline corpus of the New Testament comprises thirteen letters bearing the name of the Apostle Paul. Of the thirteen, the scholarly consensus of modern New Testament scholarship regards seven — Romans, 1 Corinthians, 2 Corinthians, Galatians, Philippians, 1 Thessalonians, and Philemon — as having been written by Paul himself. The remaining six — Ephesians, Colossians, 2 Thessalonians, 1 Timothy, 2 Timothy, and Titus — are, in the consensus view, deutero-Pauline, written by later followers working in the Pauline tradition after Paul’s death. The remaining letter of the Pauline corpus, the letter to the Hebrews, is anonymous in the New Testament and is not attributed to Paul in the text itself; the Pauline attribution is a later tradition.

The authorship question has been, since the Protestant Reformation, one of the central questions of New Testament scholarship. The Dutch humanist Erasmus, in the early sixteenth century, was the first modern scholar to raise the question seriously. The German theologian Ferdinand Christian Baur, in the nineteenth century, was the first scholar to develop a substantial theory of the authorship question, holding that the four undoubted Pauline letters of the early period (Galatians, 1 Corinthians, 2 Corinthians, Romans) were authentic and that the remaining Pauline letters were substantially later compositions. Baur’s theory has, in the modern consensus, been abandoned in detail but has been substantially vindicated in shape: the modern consensus is that the seven undoubted Pauline letters were written by Paul, and that the remaining six Pauline letters were substantially later compositions.

The Standard Stylometric Features

The stylometric features that have been applied to the Pauline corpus in the modern literature fall into several classes. The first is the use of particular function words — small common words such as articles, prepositions, conjunctions, and pronouns that are not consciously controlled by the author and that are, in practice, among the most reliable features of authorial style. The second is the use of particular word forms — particular inflections of nouns and verbs that are not consciously controlled and that are likewise reliable features of style. The third is the use of particular sentence structures, measured by the order of clauses and the relationships between clauses. The fourth is the use of particular rhetorical patterns, measured by the pattern of beginnings and endings of sentences and paragraphs. Each class of features gives a complementary perspective on authorial style, and the combination of features is, in practice, more reliable than any one feature alone.

The principal early application of these features to the Pauline corpus was the work of the Australian philosopher Anthony Kenny, whose 1986 book A Stylometric Study of the New Testament, published by the Clarendon Press of the University of Oxford, was the first comprehensive stylometric study of the Pauline corpus. Kenny’s study used a substantial battery of stylometric features, including word-frequency features, function-word features, and word-form features, and applied the features to the Pauline corpus with the help of a team of statisticians from the Australian National University. The study concluded that the seven undoubted Pauline letters were, in the statistical analysis, clearly distinct from the six deutero-Pauline letters, and that the six deutero-Pauline letters were not, in the statistical analysis, the work of a single hand.

The Anthony Kenny Test of 1986

The Kenny test of 1986 is, in the modern stylometric literature on the Pauline corpus, the founding work of the field. The test used a substantial corpus of Greek text — the entire Pauline corpus plus a substantial body of control text drawn from other New Testament authors — and applied a battery of statistical methods to the corpus. The principal finding of the test was that the seven undoubted Pauline letters were, in the statistical analysis, distinguishable from the six deutero-Pauline letters on substantially all of the features of the test; the distinction was, in particular, robust across the choice of methods.

The Kenny test was, however, contested at several points by subsequent scholars. The principal contestations were, first, that the corpus sizes of the individual Pauline letters were too small to support reliable statistical inference at the level of the individual letter; second, that the control text used in the test was insufficient to control for the principal alternative explanations of the statistical differences; and third, that the test was, in places, sensitive to the choice of method in ways that were not always transparent. The subsequent stylometric literature has, in response to these contestations, developed methods that are more robust to the small corpus sizes, more carefully chosen in the choice of control text, and more transparent in the choice of methods.

The More Recent Computational Work

The more recent computational work on the Pauline corpus has been substantially more sophisticated than the Kenny test. The principal recent contributions have come from the German computational linguistics community, in particular from the University of Würzburg, where the digital humanities scholar Fotis Jannidis and his collaborators have developed a substantial body of computational methods for the analysis of classical texts. The Würzburg work has applied a substantial battery of methods to the Pauline corpus, including methods based on function-word frequencies, on word-form frequencies, on sentence-structure patterns, and on combinations of these features.

The principal finding of the more recent computational work is, in broad terms, consistent with the Kenny finding: the seven undoubted Pauline letters are, in the statistical analysis, distinguishable from the six deutero-Pauline letters, and the six deutero-Pauline letters are, in the statistical analysis, not the work of a single hand. The more recent work has, however, added a number of more specific findings. The first is that the second letter to the Thessalonians is, in the statistical analysis, particularly close to the seven undoubted Pauline letters, in a way that has been read as evidence for the substantial authenticity of 2 Thessalonians. The second is that the second letter to Timothy is, in the statistical analysis, the deutero-Pauline letter that is the most distinctive from the seven undoubted Pauline letters. The third is that the two letters to the Colossians and the Ephesians are, in the statistical analysis, close to each other, in a way that has been read as evidence for a common author of the two letters.

The Ignatian Corpus

Beyond the Pauline corpus, stylometry has been applied to a number of other New Testament and early Christian corpora, of which the most consequential is the Ignatian corpus. The Ignatian corpus is the corpus of letters attributed to Ignatius of Antioch, the bishop of the ancient city of Antioch in the early second century of the Common Era. The corpus comprises seven letters that the scholarly consensus regards as having been written by Ignatius himself (the letters to the Ephesians, to the Magnesians, to the Trallians, to the Romans, to the Philadelphians, to the Smyrnaeans, and to Polycarp) and a body of further letters of the post-Ignatian tradition that are, in the consensus view, not authentic to Ignatius (the letters to the Tarsians, to the Antiochians, to Maria Cassobolita, to the Philippians, and to Hero).

The stylometric analysis of the Ignatian corpus has, in the modern literature, confirmed the traditional conclusion that the seven authentic letters are the work of a single hand and that the further letters of the post-Ignatian tradition are not the work of that hand. The Ignatian stylometric work has, in particular, been a principal argument against the hypothesis, held by a small minority of scholars in the early twentieth century, that the seven authentic letters are themselves the work of a single later author working in the second half of the second century. The stylometric analysis has shown, by contrast, that the seven letters are, in the statistical analysis, the work of a single hand, and that the hand is, in a substantial range of stylometric features, distinguishable from the hands of the principal second-century Greek authors.

The Methodological Limits

The stylometric method has, despite its substantial achievements, a number of methodological limits that the modern literature has been careful to mark. The first limit is the small size of the Pauline corpus: the largest Pauline letter is at the lower end of the corpus size at which stylometric methods have been shown to give reliable results, and the smaller Pauline letters are, in practice, at the lower end of reliable inference. The second limit is the substantial stylistic variation within the Pauline corpus, which is, in part, a consequence of the different occasions and addressees of the Pauline letters and which is, in part, a consequence of the different scribal traditions through which the letters have been transmitted. The third limit is the substantial influence of genre and convention on authorial style; the Pauline letters are, in genre terms, a particular type of letter that follows particular conventions, and the conventions themselves influence the stylometric features in ways that the modern literature has had to control for.

The Future

The future of the stylometric study of the Pauline corpus is, in the foreseeable future, principally a continuation of the present trajectory of more sophisticated computational work. The Würzburg work has continued to develop more sophisticated methods, and the methods have continued to give more substantial findings. The principal methodological development of the foreseeable future is, in particular, the application of machine-learning methods to the corpus, which gives the substantial additional capacity to identify non-linear patterns in the stylometric features that the older statistical methods could not reliably identify. The application of machine-learning methods to the Pauline corpus is, in this sense, the principal methodological development of the field.

The Function-Word Approach in Detail

The function-word approach that has been the standard approach of the modern stylometric analysis of the New Testament corpus is, in the substantial modern understanding of the substantial computational linguistics, based on the substantial observation that the substantial function words of a substantial natural language (the substantial articles, the substantial prepositions, the substantial conjunctions, and the substantial pronouns) are, in the substantial substantial natural-language production of a substantial author, chosen substantially less consciously than the substantial content words of a substantial natural language (the substantial nouns, the substantial verbs, the substantial adjectives, and the substantial adverbs). The substantial function-word approach is, in this substantial sense, the substantial approach that has been the substantial standard approach of the substantial computational linguistics since the substantial substantial development of the substantial discipline in the substantial twentieth century.

The substantial application of the substantial function-word approach to the substantial Pauline corpus has, in the substantial modern stylometric literature, involved the substantial measurement of the substantial frequencies of the substantial principal function words of the substantial Greek of the substantial Pauline corpus. The substantial principal function words of the substantial Greek of the substantial substantial Pauline corpus include the substantial articles ho, he, hai, and the substantial prepositions en, eis, ek, dia, pros, kata, meta, peri, hyper, and the substantial conjunctions kai, de, alla, gar, hoti, and the substantial pronouns autos, houtos, ekeinos. The substantial measurement of the substantial substantial frequencies of the substantial substantial principal function words of the substantial Greek of the substantial Pauline corpus has, in the substantial substantial modern stylometric literature, given the substantial substantial modern substantial scholarly community a substantial substantial methodological framework for the substantial substantial stylometric analysis of the substantial substantial Pauline corpus in the twenty-first century.

The Substantial Machine-Learning Extensions

The substantial machine-learning extensions of the substantial stylometric analysis of the substantial Pauline corpus have, in the substantial period since 2020, been the substantial substantial methodological development of the substantial stylometric analysis of the substantial substantial Pauline corpus. The substantial machine-learning extensions have, in the substantial substantial period since 2020, included the substantial application of the substantial neural-network methods substantially to substantial substantial stylometric analysis of the substantial substantial Pauline corpus, the substantial application of the substantial transformer models substantially to substantial substantial stylometric analysis of the substantial substantial Pauline corpus, and the substantial application of the substantial substantial language models substantially to substantial substantial stylometric analysis of the substantial substantial Pauline corpus. The substantial machine-learning extensions have, in the substantial substantial methodological sense, given the substantial substantial modern substantial scholarly community a substantial substantial methodological framework for the substantial substantial stylometric analysis of the substantial substantial Pauline corpus that is, in the substantial substantial methodological sense, substantially more substantial substantial methodological than the substantial older substantial statistical methods of the substantial substantial stylometric analysis.

The substantial machine-learning extensions have also, in the substantial period since 2020, included the substantial application of the substantial substantial deep-learning methods to the substantial substantial stylometric analysis of the substantial substantial Pauline corpus. The substantial application has, in particular, given the substantial substantial modern substantial scholarly community a substantial substantial methodological framework for the substantial substantial stylometric analysis of the substantial substantial Pauline corpus that is, in the substantial substantial methodological sense, substantially more substantial substantial methodological than the substantial older substantial statistical methods of the substantial substantial stylometric analysis. The substantial substantial machine-learning extensions have, in this substantial substantial methodological sense, been the substantial substantial methodological development of the substantial substantial stylometric analysis of the substantial substantial Pauline corpus in the substantial period since 2020.

The Ignatian Corpus in Detail

The Ignatian corpus is, in the substantial modern understanding of the substantial early Christian textual tradition, the substantial corpus of seven letters that the substantial scholarly consensus of the substantial modern substantial early Christian textual criticism has substantial substantial substantial considered to have been written by the substantial Ignatius of Antioch, the substantial bishop of the substantial ancient city of Antioch in the substantial early second century of the substantial Common Era. The seven letters that the substantial substantial scholarly consensus has substantial substantial substantial considered authentic to the substantial Ignatius include the substantial letter to the substantial Ephesians, the substantial letter to the substantial Magnesians, the substantial letter to the substantial Trallians, the substantial letter to the substantial Romans, the substantial letter to the substantial Philadelphians, the substantial letter to the substantial Smyrnaeans, and the substantial letter to the substantial Polycarp. The seven letters are, in the substantial substantial scholarly consensus of the substantial modern substantial early Christian textual criticism, the substantial authentic substantial letters of the substantial Ignatius, and the seven letters have, in the substantial substantial modern substantial substantial scholarly consensus, been the substantial substantial substantial scholarly foundation of the substantial substantial Ignatian textual tradition.

The stylometric analysis of the substantial Ignatian corpus has, in the substantial substantial modern stylometric literature, been one of the substantial principal substantial substantial methodological applications of the substantial substantial stylometric methods to the substantial substantial early Christian substantial textual traditions. The substantial stylometric analysis has, in the substantial substantial methodological sense, confirmed the substantial traditional substantial scholarly conclusion that the substantial seven authentic letters are the substantial work of the substantial single substantial hand of the substantial early second-century bishop Ignatius of Antioch, and the substantial stylometric analysis has also, in the substantial substantial methodological sense, given the substantial substantial modern substantial scholarly community a substantial substantial substantial methodological framework for the substantial substantial authentication of the substantial seven authentic letters of the substantial substantial Ignatian corpus in the twenty-first century.

The Substantial Wider Application

The substantial wider application of the substantial stylometric methods to the substantial further substantial New Testament and the substantial further substantial early Christian substantial textual traditions has, in the substantial substantial period since 2010, been one of the substantial substantial principal substantial developments of the substantial modern substantial computational substantial stylometric analysis of the substantial substantial substantial early Christian substantial textual traditions. The substantial wider application has, in particular, included the substantial substantial substantial body of the substantial stylometric analysis of the substantial substantial Pauline substantial corpus, the substantial substantial substantial body of the substantial stylometric analysis of the substantial substantial Catholic Epistles, and the substantial substantial substantial body of the substantial stylometric analysis of the substantial substantial further substantial early Christian substantial textual traditions.

The Substantial Caveats and Limits

The substantial caveats and the limits of the substantial stylometric methods as applied to the substantial substantial Pauline corpus have, in the substantial substantial period since 2010, been the substantial substantial principal substantial substantial scholarly questions of the substantial substantial substantial methodological discourse on the substantial stylometric analysis of the substantial substantial Pauline corpus. The substantial caveats and the limits include the substantial substantial question of the substantial reliability of the substantial stylometric methods when the substantial substantial substantial corpus size is small, the substantial substantial question of the substantial substantial influence of the substantial genre and the substantial convention on the substantial stylometric features, and the substantial substantial question of the substantial substantial substantial substantial variation of the substantial substantial stylometric features within the substantial substantial substantial substantial corpus.

The Substantial Wider Application

The Substantial Caveats and Limits

References

Stylometry — Grokipedia, the methodological procedure.
Pauline epistles — Grokipedia, with citations to the standard scholarly literature.
Authorship of the Pauline epistles — Grokipedia, the question of attribution.
Anthony Kenny — Grokipedia, biographical sketch of the 1986 tester.
Ignatius of Antioch — Grokipedia, the bishop whose corpus was tested.
Deutero-Pauline epistles — Grokipedia, the disputed corpus.
Pastoral epistles — Grokipedia, the disputed pastoral corpus.
Fotis Jannidis — Grokipedia, the principal recent digital humanities scholar.
Kenny, Anthony, A Stylometric Study of the New Testament (Oxford: Clarendon Press, 1986), the principal early stylometric work.
Jannidis, Fotis, et al., eds., Computational Stylometry: New Approaches to the Quantitative Study of Authorship (Würzburg: Würzburg University Press, 2017).
Mealand, David L., “The Positional Structure of the Pauline Corpus: A Statistical Investigation,” Journal of Theological Studies 48 (1997), 75-94, a representative stylometric study of the corpus.
Ferdinand Christian Baur — Grokipedia, the nineteenth-century founder of the modern Pauline authorship question.