Automated Similarity Judgment Program


The Automated Similarity Judgment Program is a collaborative project applying computational approaches to comparative linguistics using a database of word lists. The database is open access and consists of 40-item basic-vocabulary lists for well over half of the world's languages. It is continuously being expanded. In addition to isolates and languages of demonstrated genealogical groups, the database includes pidgins, creoles, mixed languages, and constructed languages. Words of the database are transcribed into a simplified standard orthography. The database has been used to estimate dates at which language families have diverged into daughter languages by a method related to but still different from glottochronology, to determine the homeland of a proto-language, to investigate sound symbolism, to evaluate different phylogenetic methods, and several other purposes.
ASJP is not widely accepted among historical linguists as an adequate method to establish or evaluate relationships between language families.
It is part of the Cross-Linguistic Linked Data project hosted by the Max Planck Institute for the Science of Human History.

History

Original goals

ASJP was originally developed as a means for objectively evaluating the similarity of words with the same meaning from different languages, with the ultimate goal of classifying languages computationally, based on the lexical similarities observed. In the first ASJP paper two semantically identical words from compared languages were judged similar if they showed at least two identical sound segments. Similarity between the two languages was calculated as a percentage of the total number of words compared that were judged as similar. This method was applied to 100-item word lists for 250 languages from language families including Austroasiatic, Indo-European, Mayan, and Muskogean.

ASJP Consortium

The ASJP Consortium, founded around 2008, came to involve around 25 professional linguists and other interested parties working as volunteer transcribers and/or extending aid to the project in other ways. The main driving force behind the founding of the consortium was Cecil H. Brown. Søren Wichmann is daily curator of the project. A third central member of the consortium is Eric W. Holman, who has created most of the software used in the project.

Shorter word lists

While word lists used were originally based on the 100-item Swadesh list, it was statistically determined that a subset of 40 of the 100 items produced just as good if not slightly better classificatory results than the whole list. So subsequently word lists gathered contain only 40 items.

Levenshtein Distance

In papers published since 2008, ASJP has employed a similarity judgment program based on Levenshtein distance. This approach was found to produce better classificatory results measured against expert opinion than the method used initially. LD is defined as the minimum number of successive changes necessary to convert one word into another, where each change is the insertion, deletion, or substitution of a symbol. Within the Levenshtein approach, differences in word length can be corrected for by dividing LD by the number of symbols of the longer of the two compared words. This produces normalized LD. An LDN divided between the two languages is calculated by dividing the average LDN for all the word pairs involving the same meaning by the average LDN for all the word pairs involving different meanings. This second normalization is intended to correct for chance similarity.

Word list

The ASJP uses the following 40-word list. It is similar to the Swadesh–Yakhontov list, but has some differences.
;Body parts
;Animals and plants
;People
;Nature
;Verbs and adjectives
;Numerals and pronouns
ASJP version from 2016 uses the following symbols to encode phonemes: p b f v m w 8 t d s z c n r l S Z C j T 5 y k g x N q X h 7 L 4 G ! i e E 3 a u o
They represent 7 vowels and 34 consonants, all found on the standard QWERTY keyboard.
ASJPcodeDescriptionIPA
ihigh front vowel, rounded and unrounded
emid front vowel, rounded and unrounded
Elow front vowel, rounded and unrounded
3high and mid central vowel, rounded and unrounded
alow central vowel, unrounded
uhigh back vowel, rounded and unrounded
omid and low back vowel, rounded and unrounded
pvoiceless bilabial stop and fricative
bvoiced bilabial stop and fricative
mbilabial nasal
fvoiceless labiodental fricative
vvoiced labiodental fricative
8voiceless and voiced dental fricative
4dental nasal
tvoiceless alveolar stop
dvoiced alveolar stop
svoiceless alveolar fricative
zvoiced alveolar fricative
cvoiceless and voiced alveolar affricate
nvoiceless and voiced alveolar nasal
Svoiceless postalveolar fricative
Zvoiced postalveolar fricative
Cvoiceless palato-alveolar affricate
jvoiced palato-alveolar affricate
Tvoiceless and voiced palatal stop
5palatal nasal
kvoiceless velar stop
gvoiced velar stop
xvoiceless and voiced velar fricative
Nvelar nasal
qvoiceless uvular stop
Gvoiced uvular stop
Xvoiceless and voiced uvular fricative, voiceless and voiced pharyngeal fricative
7voiceless glottal stop
hvoiceless and voiced glottal fricative
lvoiced alveolar lateral approximate
Lall other laterals
wvoiced bilabial-velar approximant
ypalatal approximant
rvoiced apico-alveolar trill and all varieties of “r-sounds” etc.
!all varieties of “click-sounds”