Zipf and the Monkey

A law nobody wrote on purpose

Take any long text, count how often each word appears, and sort the words from most to least common. The most frequent word turns up about twice as often as the second, three times as often as the third, ten times as often as the tenth. Multiply a word's rank by its frequency and you get roughly the same number all the way down the list.

A French stenographer, Jean-Baptiste Estoup, noticed this in 1916 while designing shorthand. The American linguist George Kingsley Zipf made it famous in the 1930s and 1940s, and found the same shape in the populations of cities and the sizes of firms. Plotted with both axes on a logarithmic scale, the pattern becomes a straight line falling at a slope close to minus one. That slope is the exponent people mean when they talk about Zipf's law.

This page is the sample

The chart beside this column is not an illustration. It counts the words of the essay you are reading right now. These paragraphs hold … words, of which … are different. The most common is …, used … times; second comes … with …. The fitted slope is …, flatter than the textbook 1, which is normal for a sample this small: the few most common words have not yet pulled ahead of the rest.

A short essay is a small and noisy sample. The top of the curve wobbles because a handful of words decide it, and the bottom flattens into a long shelf of words used only once. Those words have a name, hapax legomena, Greek for "said once". Here they make up … of the vocabulary. In large collections of text the share is often close to half. Most of the words any writer uses, they use only once.

Enter the monkey

In 1957 the psychologist George A. Miller pointed out something awkward. Imagine a monkey at a typewriter, striking letters at random and now and then the space bar. Every run of letters between two spaces counts as a word. Short words are more likely than long ones, simply because each extra letter is another chance for the space bar to come down first. Count the monkey's words, sort them, plot them, and the line falls in very nearly the same way.

Tick the box above the chart to let a monkey type exactly as many words as this essay contains. Its line appears in the second colour. With all keys equally likely, it falls in a staircase: every two-letter word is as probable as every other two-letter word, so they share a step. Switch the keys to English letter frequencies and the steps blur into a slope that is hard to tell from real prose.

Where the monkey gives itself away

Rank and frequency alone are a weak test. The second chart is a stronger one. It tracks how many distinct words have appeared after each new word is read. Real writing keeps coming back to words it already used, so its vocabulary grows slowly; that growth curve is called Heaps' law. The monkey almost never repeats itself beyond its shortest words, so its vocabulary runs away. In the table the difference shows again: the monkey's favourite words are fragments like single letters, and its list of words used once is far longer than any writer's.

Meaning leaves fingerprints the frequency curve cannot see. A writer repeats a word because the subject requires it. In this essay the word … sits high on the list for no reason except that it is what the essay is about. No random process would put it there.

So what does the law mean?

Nobody fully agrees. Zipf himself argued for a balance of effort: speakers would like a few words that do everything, listeners would like a separate word for every thing, and the curve is the compromise. Herbert Simon showed in 1955 that a rich-get-richer process, where words already in use are more likely to be used again, produces the same shape. Benoît Mandelbrot derived it from the efficient coding of information. Miller's monkey says that some of the shape comes almost for free.

In 2014 the cognitive scientist Steven Piantadosi reviewed the evidence and drew a careful conclusion. The near-straight line is so easy to produce that matching it proves little about any one explanation. What carries information is where real language departs from the simple law, and how those departures differ between languages, topics and speakers.

Count your own

Paste anything into the box on the right: a chapter, an email thread, the lyrics you know by heart, a page of code. The counting happens in your browser and nothing is sent anywhere. A few thousand words already show the line. A whole novel shows it so cleanly that it is hard to remember the curve was ever a surprise.