The Zipf Mystery

There's an entertaining "V-Sauce" video about Zipf's law for anyone wanting to learn a bit more about it and to see other examples of where it shows up: ▶ this video

> You need to know about 20,000 words in any language to sound fluent Are you sure about this? AFAIK the use of words in a language isn't distributed evenly, but according to a Pareto distribution. Thus, you can speak _okay_ knowing only ~500 words, I believe. It won't sound perfect but you'll speak and understand pretty decently. I once wrote a python script which read words from a .txt file full of french words with their english counterparts and then displayed the french word, waiting for you to type your guess. It gave instant feedback which I thought important for learning anything. The file contained the 500 most used french words according to large book surveys. This was in my last year of school and it saved me during tests where I had to write coherent sentences in french. Edit: Here it is! Zipf's law is what it's called.( link ) and here's an interesting video about it: ▶ this video

Your shock is warranted. It's actually a very deep question, called the Zipf mystery: ▶ this video No one really knows why it's true. It's also true across physical constants, not just artificial quantities made up by humans. The same type of distribution pops up in language, in nature, in numbers, popularity of opening chess moves, everywhere.

I won't say it's easy , but it's easier than you might think. Word use follows a power law distribution in all languages ( Zipf's Law, ▶ this video is an excellent vsauce cover of the subject ), but the short of it is that within a relatively short amount of time spent studying you'll be able to pick out common words and build from there, even if you started from zero.

The majority of seeding/downloading will be done with the (minority) most popular files, while only a minority of seeding/downloading is done for the majority of files. For example, go to TPB and look at the distribution of seeders/leechers. ▶ this video

Found the examples of power laws starting on page-5 interesting; all the following follow power law: Word frequency, Citations of scientific papers, Web hits, Copies of books sold, Telephone calls, Magnitude of earthquakes, Diameter of moon craters, Intensity of solar flares, Intensity of wars, Wealth of the richest people, Frequencies of family names, Populations of cities, etc. Terence Tao also has a post on it here from 2009: - Benford’s law, Zipf’s law, and the Pareto distribution link Vsause has video on it too: - ▶ this video

Zipf distributions show up in randomly generated character sequences, Semantics are orthogonal. Zipf thought it was the principle of least effort, the use of language to minimize size of communication and maximize meaning. This heuristic causes the use of language to approach some ideal optimal communication, which subsequently shows up in statistics as Zipfy. The underlying reason is something rooted in information and complexity theory, and will depend on Kolmogorov complexity and systems modeling. To quote vsauce : "...Moreover, Zipf's law doesn't just mysteriously describe word use. It's also found in city populations, solar flare intensities, protein sequences and immune receptors, the amount of traffic websites get, earthquake magnitudes, the number of times academic papers are cited, last names, the firing patterns of neural networks, ingredients used in cookbooks, the number of phone calls people received, the diameter of Moon craters, the number of people that die in wars, the popularity of opening chess moves, even the rate at which we forget." Semantics and syntax are dependent on Kolmogorov complexity, so if you loosen the definitions to use the rules of any given system as syntax and the relationships between atomic units over time as semantics, you could possibly derive a formula that answers the question of what Zipf's law actually is, insofar as there's some universal rule that governs the distributions of units within systems that have entropy. ▶ this video

It's a deeper reason than that. Humans are hardwired to start with short symbols. Picture a group of humans who evolve a new language. That language will likely start with short sounds, and then progressively build up to longer sounds. ▶ this video Anyone who hasn't read A Mathematical Theory of Communication, you're missing out. It's quite readable and quite cool: link

Mass-publishing markets are very strongly given to power-law distributions. We hear that "anybody can be famous" (or successful, or ...), but everybody cannot be famous. Attention and publicity are the ultimate zero-sum games: no matter how large a given talent pool, there will only ever be ten top-ten spots, 100 top-100 spots. All you're doing by enlarging the pool is amplifying the competition for those spots. There's another side to the equation, though, which is the scope of the audience . Prior to modern technological methods, and I'm including here everything from relatively modern printing (say, 19th century onward), as well as phonograph, cinema, radio, television, and sound amplification, the reach of any given work was small . A book might have a printing of several thousand copies (Adam Smith, Wealth of Nations , ~5,000 initial print run, and the cost at ￡5 was roughly a quarter of a working man's annual wage). A theatre might seat (or stand) a few thousand (the New Globe Theatre: 3,000), and the largest cities had populations of about a quarter million (London in 1600, more or less). Transportation was slow and expensive (Smith again: 2 weeks, by stage coach or horse from Edinborough to London -- that's how he travelled to university himself). link As a consequence, entertainment was strongly local . There might be small touring acts, local musicians, and in rare cases, artists with a noble or royal patron (think Shakespeare himself, after a fashion, or Handel,