Word frequencies: A comparison of Pareto type distributions

Martin Wiegand; Saralees Nadarajah; Yuancheng Si

doi:10.1016/j.physleta.2017.12.056

Word frequencies: A comparison of Pareto type distributions

Martin Wiegand, Saralees Nadarajah^*, Yuancheng Si

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

Mehri and Jamaati (2017) used Zipf's law to model word frequencies in Holy Bible translations for one hundred live languages. We compare the fit of Zipf's law to a number of Pareto type distributions. The latter distributions are shown to provide the best fit, as judged by a number of comparative plots and error measures. The fit of Zipf's law appears generally poor.

Original language	English
Pages (from-to)	621-632
Number of pages	11
Journal	Physics Letters, Section A: General, Atomic and Solid State Physics
Volume	382
Issue number	9
Early online date	5 Jan 2018
DOIs	https://doi.org/10.1016/j.physleta.2017.12.056
Publication status	Published - 9 Mar 2018

Keywords

Kolmogorov-Smirnov test statistic
Squared error
Zipf's law

Access to Document

10.1016/j.physleta.2017.12.056

Cite this

@article{11c29782487f47b7bb58b2468d2c567c,

title = "Word frequencies: A comparison of Pareto type distributions",

abstract = "Mehri and Jamaati (2017) used Zipf's law to model word frequencies in Holy Bible translations for one hundred live languages. We compare the fit of Zipf's law to a number of Pareto type distributions. The latter distributions are shown to provide the best fit, as judged by a number of comparative plots and error measures. The fit of Zipf's law appears generally poor.",

keywords = "Kolmogorov-Smirnov test statistic, Squared error, Zipf's law",

author = "Martin Wiegand and Saralees Nadarajah and Yuancheng Si",

year = "2018",

month = mar,

day = "9",

doi = "10.1016/j.physleta.2017.12.056",

language = "English",

volume = "382",

pages = "621--632",

journal = "Physics Letters, Section A: General, Atomic and Solid State Physics",

issn = "0375-9601",

publisher = "Elsevier BV",

number = "9",

}

TY - JOUR

T1 - Word frequencies

T2 - A comparison of Pareto type distributions

AU - Wiegand, Martin

AU - Nadarajah, Saralees

AU - Si, Yuancheng

PY - 2018/3/9

Y1 - 2018/3/9

N2 - Mehri and Jamaati (2017) used Zipf's law to model word frequencies in Holy Bible translations for one hundred live languages. We compare the fit of Zipf's law to a number of Pareto type distributions. The latter distributions are shown to provide the best fit, as judged by a number of comparative plots and error measures. The fit of Zipf's law appears generally poor.

AB - Mehri and Jamaati (2017) used Zipf's law to model word frequencies in Holy Bible translations for one hundred live languages. We compare the fit of Zipf's law to a number of Pareto type distributions. The latter distributions are shown to provide the best fit, as judged by a number of comparative plots and error measures. The fit of Zipf's law appears generally poor.

KW - Kolmogorov-Smirnov test statistic

KW - Squared error

KW - Zipf's law

UR - http://www.scopus.com/inward/record.url?scp=85040616807&partnerID=8YFLogxK

U2 - 10.1016/j.physleta.2017.12.056

DO - 10.1016/j.physleta.2017.12.056

M3 - Article

AN - SCOPUS:85040616807

SN - 0375-9601

VL - 382

SP - 621

EP - 632

JO - Physics Letters, Section A: General, Atomic and Solid State Physics

JF - Physics Letters, Section A: General, Atomic and Solid State Physics

IS - 9

ER -

Word frequencies: A comparison of Pareto type distributions

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this