History of creation[ | ]
The author of the discovery of the pattern is a French stenographer (fr. Jean-Baptiste Estoup), who described it in 1908 in The Range of Shorthand. The law was first applied to describe the distribution of city sizes by the German physicist Felix Auerbach in his work "The Law of Population Concentration" in 1913 and is named after the American linguist George Zipf, who in 1949 actively popularized this pattern, first proposing to use it to describe the distribution of economic forces and social status.
An explanation of Zipf's law based on the correlation properties of additive Markov chains (with step memory function) was given in 2005.
Zipf's law is mathematically described by the Pareto distribution. It is one of the basic laws used in infometrics.
Applications of the law[ | ]
George Zipf in 1949 first showed the distribution of people's incomes according to their size: the richest person has twice as much money as the next richest, and so on. This statement turned out to be true for a number of countries (England, France, Denmark, Holland, Finland, Germany, USA) in the period from 1926 to 1936.
This law also works in relation to the distribution of the city system: the city with the largest population in any country is twice the size of the next largest city, and so on. If you arrange all the cities of a certain country in the list in descending order of population, then each city can be assigned a certain rank, that is, the number that it receives in this list. At the same time, the population size and rank obey a simple pattern expressed by the formula:
P n = P 1 / n (\displaystyle P_(n)=P_(1)/n),where P n (\displaystyle P_(n))- city population n-th rank; P 1 (\displaystyle P_(1))- population of the main city of the country (1st rank).
Empirical studies support this assertion.
In 1999, the economist Xavier Gabet described Zipf's law as an example of a power law: if cities grow randomly with the same standard deviation, then at the limit the distribution will converge to Zipf's law.
According to the findings of researchers in relation to urban settlement in the Russian Federation, in accordance with Zipf's law:
- most cities in Russia lie above the ideal Zipf curve, so the expected trend is a continued decline in the number and population of medium and small cities due to migration to large cities;
- accordingly, 7 million-plus cities (St. Petersburg, Novosibirsk, Yekaterinburg, Nizhny Novgorod, Kazan, Chelyabinsk, Omsk), which are below the ideal Zipf curve, have a significant population growth reserve and expect population growth;
- there are risks of depopulation of the first city in the rank (Moscow), since the second city (St. Petersburg) and subsequent large cities are far behind the ideal Zipf curve due to a decrease in demand for labor with a simultaneous increase in the cost of living, including, first of all, the cost of purchase and rental housing.
Criticism [ | ]
American bioinformatician proposed a statistical explanation of Zipf's law, proving that a random sequence of characters also obeys this law. The author concludes that Zipf's law, apparently, is a purely statistical phenomenon that has nothing to do with the semantics of the text and has a superficial relation to linguistics.
During the election process, voters express their attitude towards certain political figures or parties, casting their vote for this or that candidate or party. The question arises - are there any patterns that describe the distribution of votes between different candidates or parties? If there are no regularities, then any correlation between the numbers of votes received by candidates or parties, as well as between these numbers of votes and, for example, the turnout of voters or the number of invalid ballots, is possible. If there are certain patterns in the distribution of votes, then not all variants of their distribution are possible. Based on the material of many elections in various countries, a statistical relationship was revealed that exists between the numbers of votes received in elections by various candidates and parties. It was found that this relationship is described by the following simple relationship:
If on one axis the number of votes N(i) received by each candidate is plotted on a logarithmic scale, and on the other axis, also on a logarithmic scale, the place i occupied by the same candidate during the election, then the points obtained with sufficient approximation are located along a straight line :
ln N(i) = A - B x lni (1)
The validity of the above equation was confirmed in a series of works by Russian specialists in mathematical political science (Sobyanin, Sukhovolsky, 1995), who analyzed the results of the elections of people's deputies of Russia in 1990, the elections of the President of Russia in 1991 and 1996, as well as data on elections in a number of countries, starting with the election of the President of France in 1848, where Louis-Napoleon Bonaparte won.
This mathematical result is non-trivial in nature. Specialists - physicists, chemists, metallurgists, demographers, ecologists and representatives of many other fields of knowledge dealing with large amounts of statistical data, are well aware that the indicated numerical regularity is of a general nature and describes a situation of "free competition" for the distribution of a finite number of or conditional "goods". It turns out that all conceivable variety of objects, situations and causal relationships does not change the nature of this dependence: as soon as there is free competition, its results in any case fit on the "logarithmic straight line" - only the constant A and the slope of the straight line B change. And vice versa: as soon as there are deviations from the conditions of free competition, the points inevitably deviate from the straight line - and the further, the more significant the "unfreedom factors". So, for example, the "competition" of cities for the number of people living in them leads in civilized countries to just such a dependence. Meanwhile, in the USSR, cities such as Moscow, Leningrad and some other centers deviated significantly from "direct free competition" - due to administrative restrictions associated with the passport regime. Similarly, free competition leads to the same relationship between the size of the largest fortunes and the "place" occupied by their owners in the list of such fortunes - of course, in those parts of the world where such lists exist. The law of distribution of predators by mass known to zoologists is exactly the same (in the absence of anthropogenic factors), and so on.
For the first time, regularities of this kind were established by the Italian sociologist and mathematician V. Pareto, who was engaged in the distribution of the country's inhabitants according to their wealth; subsequently, the American linguist J.K. Zipf, studying the distribution of the frequency of the use of words in texts. Various variants of the ratio written above are called the Zipf-Pareto law. Methods of analysis related to the study of rank distributions are widely used in linguistics, scientometrics, and ecology. Compliance with relation (1) for the electoral process means that there is "free competition" of all candidates who have the opportunity to freely explain their political views and political platform to voters.
The fulfillment of the Zipf-Pareto law for the electoral process means that each of the candidates, each of the parties and political groups of voters voting according to a certain type, has its own political platform, which does not overlap with all the others. Available candidates should cover all possible preferences of voters; then the proportion of voters seeking their choice outside the proposed list of candidates is quite small, and equation (1) describes the distribution of votes with high accuracy. Otherwise, empty "niches" may appear in distribution (1), and the whole analysis becomes more complicated.
Parameters A and B included in equation (1) are calculated based on data on the number of voters who voted for different candidates or for different political groups using regression analysis methods. Parameter A in equation (1) is the logarithm of the number of voters who voted for the leading candidate. The value B, the preference coefficient, characterizes the slope of the straight line (1) and serves as a numerical measure of the homogeneity of the choice of voters. If B = 0, this means that voters have no preference for one party or candidate over another, and that they all received the same number of votes in the election. On the contrary, for large values of steepness B, the outsider parties receive very few votes compared to the leading parties (however, in practice, the parameter B is almost never greater than one). If deviations from the straight line of type (1) are noticed, then under the assumptions made above, this indicates the absence of conditions for free political competition. This can be caused either by the presence of some additional external factors, for example, intimidation of voters by possible political and economic repressions in the event of voting (or non-voting) for a particular candidate, or by direct falsification of election results during the counting of votes in election commissions of various levels. Figure 2 shows a typical graph of the rank distribution of the number of voters in elections in Russia. As can be seen, between the sizes of different groups of voters and the ranks of these groups (i.e., the places of candidates) in logarithmic coordinates (along both axes), there is practically a linear relationship.
The type of distribution of votes cast for different candidates or parties helps to identify electoral fraud. In the simplest case of falsification, if a certain number of ballots filled in favor of some candidate or party are thrown into the ballot boxes, then it turns out that the rank distribution of the number of votes cast for individual candidates is not depicted straight. But if we exclude the data on the candidate in whose favor falsifications were made, then for the remaining candidates (or parties) the rank distribution will correspond to the theoretical one. In the case under consideration, the number of ballots planted can be estimated from the difference between the number of votes received by such a candidate according to official data and the number found from the rank distribution equation after excluding the data related to the said candidate. Figure 3 shows the distribution of votes cast - according to the election commission - for candidates for the post of head of the administration of the Lipetsk region in the elections held in the spring of 1993. This distribution is obviously far from a straight line. In this case, the trial, which took place in 1995, confirmed the existence of falsifications in favor of the candidate who won first place.
Why doesn't Zipf's Law work in Russia? March 11th, 2017
Zipf's Law was first applied to describe the size distribution of cities by the German physicist Felix Auerbach in his work The Law of Population Concentration in 1913. It bears the name of the American linguist George Zipf, who in 1949 actively popularized this pattern, first proposing to use it to describe the distribution of economic power and social status.
In Russia, this law does not work.
Let's go back to 1949. The linguist George Zipf (Zipf) noticed a strange trend in people's use of certain words in the language. He found that a small number of words are used constantly, and the vast majority are used very rarely. If we evaluate words by popularity, a striking thing is revealed: the word of the first rank is always used twice as often as the word of the second rank and three times more often than the word of the third rank.
Zipf found that the same rule applies to the distribution of people's incomes in a country: the richest person has twice as much money as the next richest person, and so on.
Later it became clear that this law also works in relation to the size of cities. The city with the largest population in any country is twice the size of the next largest city, and so on. Incredibly, Zipf's law has operated in absolutely all countries of the world over the past century.
Just take a look at the list of the largest cities in the United States. So, according to the 2010 census, the population of the largest US city, New York, is 8,175,133 people. Number two is Los Angeles with a population of 3,792,621. The next three cities, Chicago, Houston and Philadelphia boast populations of 2,695,598, 2,100,263 and 1,526,006, respectively. Obviously, these numbers are inaccurate, but they are surprisingly consistent with Zipf's law nonetheless.
Paul Krugman, writing on the application of Zipf's law to cities, famously noted that economic theory is often accused of creating highly simplified models of a complex, chaotic reality. Zipf's law shows that the exact opposite is true: we use too complex, messy models, and reality is amazingly neat and simple.
law of strength
In 1999, the economist Xavier Gabet wrote a scientific paper in which he described Zipf's law as "the law of force."
Gabet noted that this law persists even if cities grow in a chaotic manner. But this even structure breaks down as soon as you move on to non-metropolitan cities. Small towns with a population of about 100,000 seem to follow a different law and show a more understandable size distribution.
One might wonder what is meant by the term "city"? After all, for example, Boston and Cambridge are considered two different cities, just like San Francisco and Auckland, separated by water. Two Swedish geographers also had this question, and they began to consider the so-called "natural" cities, united by population and road connections, and not by political motives. And they found that even such "natural" cities obey Zipf's law.
Why does Zipf's law work in cities?
So what makes cities so predictable in terms of population? Nobody can really explain it. We know that cities are expanding due to immigration, immigrants flock to big cities because there are more opportunities. But immigration is not enough to explain this law.
There are also economic motives, since big cities make big money, and Zipf's law works for income distribution as well. However, this still does not give a clear answer to the question.
Last year, a group of researchers found that there are exceptions to Zipf's law: the law only works if the cities in question are economically connected. This explains why the law is valid, for example, for a single European country, but not for the entire EU.
How do cities grow?
There's another weird rule that applies to cities that has to do with how cities consume resources when they grow. As cities grow, they become more stable. For example, if a city doubles in size, the number of gas stations it requires does not double.
The city will live quite comfortably if the number of gas stations increases by about 77%. While Zipf's law follows certain social laws, this law is more closely related to nature, such as how animals consume energy as they grow older.
The mathematician Stephen Strogatz describes it this way:
How many calories per day does a mouse need compared to an elephant? Both of them are mammals, so it can be assumed that at the cellular level they should not be very different. Indeed, if cells of ten different mammals are grown in a laboratory, all of these cells will have the same metabolic rate, they do not remember at the genetic level how big their host really is.
But if we take an elephant or a mouse as a full-fledged animal, a functioning cluster of billions of cells, then the cells of the elephant will spend much less energy on the same action than the cells of a mouse. The law of metabolism, called Kleiber's Law, states that the metabolic requirements of a mammal increase in proportion to its body weight by a factor of 0.74.
This 0.74 is very close to the 0.77 seen with the law governing the number of gas stations in the city. Coincidence? Maybe, but most likely not.
In Russia, the population of the largest city, Moscow, is officially about 11.5 million people. The population of the second city, St. Petersburg, is 5.2 million. As we can see, the ratio of the population of the two cities approximately corresponds to the "Zipf's law". According to it, the third largest city in Russia should have about 4 million people, and the fourth - about 3 million. However, there are no such cities in Russia. In reality, the third city in Russia, Novosibirsk, has a population of 1.6 million people (2.5 times less than the norm), and the fourth, Yekaterinburg, 1.4 million, which is also 2 times lower than the “Zipf” norm.
Why "Zipf's law" does not work in Russia? The American sociologist Richard Florida answers this question in his book The Creative Class. He writes that "Zipf's law" does not work in empires (or countries that have relapsed empires) and planned economies. He names three such countries-exceptions: England (where after London there is not even a second city, 2 times smaller in population), Russia and China.
A study on the "Zipf's law" was also carried out by the Financial University under the Government of Russia. Its conclusion was this:
“The real distribution of Russian cities by population does not fully correspond to the Zipf curve for either developed or developing countries. Part of the real Zipf curve for Russia is located above the ideal one, which corresponds to the distribution of cities in developed countries, and the part below it corresponds to the distribution of cities in developing countries. Thus, according to Zipf's rule, it turns out that in Russia the largest cities and million-plus cities play the dominant role. The deviation of the real curve from the ideal is due to the vast territory of the country and various socio-economic and natural-climatic factors.
Two megacities and small and medium-sized cities (up to 250,000 people) fit perfectly into the type of western urbanization. But large cities and million-plus cities are not.
Findings from another study:
“The revealed trends do not correspond to the assumptions expressed in the literature that the reason for Russia's deviation from the Zipf pattern is the centralized planning of spatial development, which included support for medium and small towns during the Soviet period. The transition to the market was supposed to eliminate these distortions and bring the rank-size relationship closer to the canonical form, however, despite the involvement of market mechanisms in the formation of a space for economic activity, a further deviation from it was observed in the country.
(The circles indicate the population of the regions of Russia)
Those. deviation from the "Zipf's law" in Russia is not the result of a planned economy (as in China), but a consequence of the country's imperialism (when one or two cities play the role of a metropolis).
Based on these trends, the probability of urban development/regression in Russia is as follows:
— Most Russian cities lie above the ideal Zipf curve, so the expected trend is a continued decline in the number and population of medium and small towns due to migration to large cities.
— 7 million-plus cities (St. Petersburg, Novosibirsk, Yekaterinburg, Nizhny Novgorod, Kazan, Chelyabinsk, Omsk), which are below the ideal Zipf curve, have a significant reserve of population growth and expect population growth.
— There are risks of depopulation of the first city in the rank (Moscow), since the second city (St. Petersburg) and subsequent large cities are far behind the ideal Zipf curve due to a decrease in demand for labor with a simultaneous increase in the cost of living, including, first of all, the cost buying and renting a home.
(In the USSR, the "Zipf's law" also did not work - you can see the deviation of cities from the Zipf curve, where they should have been)
Richard Florida in The Creative Class notes another difference between American and Russian cities. In the United States, the concentration of the creative class is in medium-sized cities scattered throughout the country. Thus, the highest proportion of the creative class in cities such as San Jose, Boulder (Colorado), Huntsville (Alabama), Corvallis (Oregon), etc. - in them this share is 40-48%. But the largest city in the United States, New York, is among the middle peasants in terms of the share of the creative class - 35% of the total number of employees and 34th in the ranking, the second city in the country, Los Angeles, is generally 60th. A similar trend is observed in other countries where the "Zipf's law" works (Germany, France, Italy, Sweden, etc.).
In Russia, almost the entire creative class of the country is concentrated in Moscow, while the rest of the cities remain the zone of industrial time of the middle of the 20th century.
All this is terribly exciting, but perhaps less mysterious than Zipf's law. It is not so difficult to understand why a city, which is, in fact, an ecosystem, albeit built by people, should obey the natural laws of nature. But Zipf's law has no analogue in nature. This is a social phenomenon and it has only been taking place over the past hundred years.
All we know is that Zipf's law applies to other social systems, including economic and linguistic systems. Thus, perhaps there are some general social rules that create this strange law, and someday we will be able to understand them. Whoever solves this puzzle may find the key to predicting much more important things than the growth of cities. Zipf's Law may be just a small aspect of the global rule of social dynamics that governs how we communicate, trade, form communities, and more.
P.S. Personally, it seems to me that a law with such approximate assumptions for numbers and a bunch of exceptions is generally difficult to call a law. Just a random coincidence.
What do you think?
sources
Hi all! Recently, more and more often I hear from colleagues about the requirement in the TOR to evaluate the quality of the text according to Zipf's law. And not everyone understands how to edit the text for this law. In today's article I will try to tell you how to improve the parameter in the simplest way, and also clarify why good authors don't really need it.
You can determine the quality of the text according to Zipf's law using several services. But, I think PR-CY is the most adequate, it combines the right formula with a simple and understandable interface. That is what I used in the preparation of this material.
What is Zipf's law
To begin with, it is worth understanding what it is. According to Wikipedia, Jean-Baptiste Estoux formulated this pattern in 1908, this law originally referred to shorthand. The first application of the regularity known to the general public relates to demography, and more precisely to the distribution of population in cities, was used by Felix Auerbach.
The pattern received its modern name in 1949 thanks to the linguist George Zipf. He showed with its help the gradation of the distribution of wealth among the population. And only then the law began to be applied to determine the readability of texts.
How is it calculated
To properly use this law, you need to understand how it works. Let's analyze the formula for the calculation.
- F is the frequency of using the word;
- R is serial number;
- C is a constant value (a number indicating the largest word in terms of the number of repetitions).
In practice, another formula turns out to be more convenient, it looks clearer.
This approach is more convenient, since we have data on the number of repetitions of the most common word. It is from this quantity that they are repelled.
To simplify, in our text the second most frequent word should be twice as rare as the first. Coming in third place, three times and so on.
Text fitting example
The theory has been dealt with a little. It remains to deal with practice. As an experimental text, I took an article from T-Zh. Why from there? Everything is simple. At the moment, this is one of the best examples of the info style loved by many. Well, it was interesting what the text written under the direction of Maxim Ilyakhov would show. I will say right away that the texts for this indicator are at the level, although, having shoveled more than 40 sites, I did not find a single article with poor naturalness at all. Also, I’ll immediately jump ahead and say that the experimental text after fitting became much worse, despite the improved Zipf score, you shouldn’t bother too much with an excessive increase in naturalness.
This is what the analyzer showed us after checking.
Let's take a look at what's in there. As you can see, there is a column with words, as well as incomprehensible numbers. The "occurrence" column (1) indicates how many times the word forms occur in the text. In the Zipf column (2) is the recommended number of entries. Markers 3 and 4 mark ideal indicators for the second and third positions. You should also pay attention to the recommendations, it indicates how many words you need to remove to achieve the perfect combination.
For a better understanding, let's analyze what the analyzer counted. We take the number 39 (C) as a basis, we also need a serial number, pay attention to the 2 (F) position. We take the formula.
Substitute.
F=39/2=19.5
We round up and get 20, this will be the required number of occurrences. This is confirmed by the analyzer. In our country, the second most popular word is used 28 times, respectively, 8 repetitions will need to be removed or replaced.
Having dealt with the principle of the law, we begin to edit. To do this, we delete or replace with synonyms words that have more occurrences than required by Zipf. As a result, we get this picture. 
As you can see, I managed to increase the rate from 83% to 88%. However, the quality of the text suffered significantly. You should not strive to increase this figure to 100%. In fact, if you already have 75%, this is excellent and you should not pervert further.
Useful advice
Pay attention not only to the first lines. Start fitting from the last positions in the list, they often have a greater impact on the overall score than the first ten words.
Zipf and SEO
Now let's move on to why a copywriter needs to know this pattern. When ordering texts, SEOs strive to make them the most convenient for search engines. It is believed (though not clear by whom) that Zipf's law is actively used by search algorithms. It is difficult to prove or disprove this statement. I could not find any sane research and experiments on this topic.
Decided to check it out myself. To do this, I took the issue for such a competitive query “plastic windows”, Yandex took the Moscow issue, I had to conjure in Google, and he also seemed to identify me as a resident of the capital (at least he showed me an ad with Moscow geolocation). I took the first page of the issue, plus 49th place. This is how the sign turned out. 
If you look more closely, you can see that in Yandex the output is more even, if you look at the pattern we are studying. But, at the same time, a higher figure does not guarantee victory in the fight for first place in the top.
Based on this, it can be said that if search engines apply this law, it is only one of the factors. And not the main one.
conclusions
OK it's all over Now. Now you know what the quality of the text according to Zipf's law is, and you can also adjust this indicator. In fact, there is nothing complicated here, everything is quite simple. It is enough to understand the principle of operation of this regularity once.
For the past century, a mysterious mathematical phenomenon called Zipf's law has been able to predict with great accuracy how the size of giant cities around the world will change. The thing is that no one understands how and why this law works ...
Let's go back to 1949. The linguist George Zipf (Zipf) noticed a strange trend in people's use of certain words in the language. He found that a small number of words are used constantly, and the vast majority - very rarely. If we evaluate words by popularity, a striking thing is revealed: the word of the first rank is always used twice as often as the word of the second rank and three times more often than the word of the third rank.
Zipf found that the same rule applies to the distribution of people's incomes in a country: the richest person has twice as much money as the next richest person, and so on.
Later it became clear that this law also works in relation to the size of cities. The city with the largest population in any country is twice the size of the next largest city, and so on. Incredibly, Zipf's law has operated in absolutely all countries of the world over the past century.
Just take a look at the number of the largest cities in Russia. The population of Moscow is approximately 2 times larger than that of St. Petersburg.
Paul Krugman, writing on the application of Zipf's law to cities, famously noted that economic theory is often accused of creating highly simplified models of a complex, chaotic reality. Zipf's law shows that the exact opposite is true: we use too complex, messy models, and reality is amazingly neat and simple.
law of strength
In 1999, the economist Xavier Gabet wrote a scientific paper in which he described Zipf's law as "the law of force."Gabet noted that this law persists even if cities grow in a chaotic manner. But this even structure breaks down as soon as you move on to non-metropolitan cities. Small towns with a population of about 100,000 seem to follow a different law and show a more understandable size distribution.
One might wonder what is meant by the term "city"? After all, for example, Boston and Cambridge are considered two different cities, just like San Francisco and Auckland, separated by water. Two Swedish geographers also had this question, and they began to consider the so-called "natural" cities, united by population and road connections, and not by political motives. And they found that even such "natural" cities obey Zipf's law.
Why does Zipf's law work in cities?
So what makes cities so predictable in terms of population? Nobody can really explain it. We know that cities are expanding due to immigration, immigrants flock to big cities because there are more opportunities. But immigration is not enough to explain this law.There are also economic motives, since big cities make big money, and Zipf's law works for income distribution as well. However, this still does not give a clear answer to the question.
Last year, a group of researchers found that there are exceptions to Zipf's law: the law only works if the cities in question are economically connected. This explains why the law is valid, for example, for a single European country, but not for the entire EU.
How do cities grow?
There's another weird rule that applies to cities that has to do with how cities consume resources when they grow. As cities grow, they become more stable. For example, if a city doubles in size, the number of gas stations it requires does not double.The city will live quite comfortably if the number of gas stations increases by about 77%. While Zipf's law follows certain social laws, this law is more closely related to nature, such as how animals consume energy as they grow older.
The mathematician Stephen Strogatz describes it this way:
How many calories per day does a mouse need compared to an elephant? Both of them are mammals, so it can be assumed that at the cellular level they should not be very different. Indeed, if cells of ten different mammals are grown in a laboratory, all of these cells will have the same metabolic rate, they do not remember at the genetic level how big their host really is.
But if we take an elephant or a mouse as a full-fledged animal, a functioning cluster of billions of cells, then the cells of the elephant will spend much less energy on the same action than the cells of a mouse. The law of metabolism, called Kleiber's Law, states that the metabolic requirements of a mammal increase in proportion to its body weight by a factor of 0.74. This 0.74 is very close to the 0.77 seen with the law governing the number of gas stations in the city.
Coincidence? Maybe, but most likely not.
All this is terribly exciting, but perhaps less mysterious than Zipf's law. It is not so difficult to understand why a city, which is, in fact, an ecosystem, albeit built by people, should obey the natural laws of nature. But Zipf's law has no analogue in nature. This is a social phenomenon and it has only been taking place over the past hundred years.
All we know is that Zipf's law applies to other social systems, including economic and linguistic systems. Thus, perhaps there are some general social rules that create this strange law, and someday we will be able to understand them. Whoever solves this puzzle may find the key to predicting much more important things than the growth of cities. Zipf's Law may be just a small aspect of the global rule of social dynamics that governs how we communicate, trade, form communities, and more.
