Big data bringing Asimov’s Psychohistory from fiction to reality

Isaac Asimov

Isaac Asimov, my favorite science fiction writer and thinker introduced a concept called Psychohistory in his famous “Foundation” novel series.

Psychohistory is the science of predicting the behavior of large groups of people using a statistical model of the psychology of groups using historical data. And for events in the future are primarily the outcome of the what people do, by predicting the behavior of people one can effectively predict the future!

There are some logical requirements or assumptions that are the basis of Psychohistory:

  1. The population of the group should be large enough
  2. The amount of data available about the behavior of people should be sufficiently detailed
  3. The technology to process such large and complex data is available

In the last 60 years since Asimov first introduced Psychohistory in 1950’s,  all these criteria are starting to inch closer to reality.

  • The population of the earth has crossed 7 billion – higher than the 2.5 billion – the population in 1950, but still quite small as compared to the “twenty-five million worlds…each containing a billion or more human beings” (Thanks Bob for the correction)
  • The democratization of data with the availability of cheap data creation tools such as mobile phones has resulted in a huge explosion of data available about people’s behavior.
  • And the third phenomenon of the technology to process such large and complex data is now available.

As a result, it is becoming increasingly possible to make predictions about human behavior based on such data.

Take the example of an interesting experiment done by a group of mathematicians and computer scientists in 2010 based on documents about the war in Afghanistan that were made available via Wikileaks. They extracted historical patterns of violence from these documents and could effectively predict where future events of violence would occur in Afghanistan.

Then there is the famous example of Andrew Pole, an economist and statistician working with the retail chain Target who mined historical buying data to predict which customers had a new baby on the way. The article in NY Times quoted a sensational anecdote of how the pregnancy of a teenage girl was predicted before her parents knew about it!

Another sensational story is of Dan Wagner, the chief analytics officer for the Obama 2012 campaign, who led the campaign’s “Cave” of data scientists. They used data analytics and the experimental method to assemble a winning coalition vote by vote. In doing so, they overturned the long dominance of TV advertising in U.S. politics and created something new in the world: a national campaign run like a local ward election, where the interests of individual voters were known and addressed.

These examples clearly demonstrate how large enterprises and governments are able to use historical data to predict specific outcomes in the future, making it appear closer and closer to Asimov’s Psychohistory.

But there is one thing that Asimov probably did not foresee – he imagined that Psychohistory would be used by a central agency that would predict future events so as to effectively govern the people. However, what is actually happening now is that the ability to analyze historical data is becoming accessible to everyone, and is not restricted to only governments or large organizations with big budgets.

Consider these facts:

  1. A large part of the data that represents human behavior is available to everyone via the Internet (social media, blogs, email, documents)
  2. The technology required to process this data – both software and hardware is affordable (cloud computing) and mostly free (open source)

So it is possible for smaller entities with low budgets to use data to predict outcomes and plan their actions to achieve fruitful outcomes.

Of course, what it takes is some smart people to know how to use this data and technology and get meaningful insights – that’s the mission that (smart?) people like us are trying to actualize 🙂

Click to receive a case study of Predictive Analytics

Related Posts

Share on Facebook0Tweet about this on TwitterShare on Google+0Share on LinkedIn0
  • M.D.

    Ashutosh,

    One of such “smaller entities” to which you refer is “Foundation Encyclopedia Dialectica [F.E.D.]”.

    They have derived a set of seven ‘Psychohistorical-Dialectical Equations”, using the “contra-Boolean Algebra” that their founder discovered, and axiomatized. —

    http://www.dialectics.org/dialectics/Aoristoss_Blog/Entries/2012/5/19_The_F.E.D._Psychohistorical_Equations.html

    However, although it is clear that they support and intend the use of “Big Data”, their initial version of “the psychohistorical equations” are rooted in a different approach than mere statistics and probability theory.

    Their initial equations are aimed at establishing a generalized, deterministic, necessary order-of-appearance, or order-of-progression, of “socio-ontology”, an undergirding, causal “chain of socio-historical phenomena”, around which “Big Data” can supply detailed, quantitative specifics.

    For more about this, see —

    http://www.dialectics.org/dialectics/Welcome.html

    Regards,

    M.D.

    • Thanks for dropping by M.D. and for your very interesting reference to FED and dialectics. I will surely go through it and see if I can understand some of it.

      Regards
      Ashutosh

  • Pingback: Big Data bringing Asimov’s Psychohistory from fiction to reality | Pomelo girl thoughts()

  • BOBinBrooklyn

    You state that:

    “closer to 1 trillion [people] that Asimov had imagined to be spread across the Galaxy.”

    In Prelude to Foundation (1988) Hari Seldon tells the Galactic Emperor that his Empire contains “twenty-five million worlds…each containing a billion or more human beings” (page 12 of the paperback)

    Now 25 million x 1,000,000,000 = 25,000 trillion = 25 quadrillion people at least
    we are never told the exact population — but it’s way more than 1 trillion!

    By the year 20020 A.D. (12,020 G.E. (Galactic Era) )

    Where are you getting 1 trillion from?

    • Bob, you’re right… my bad in putting a number based only on my inaccurate memory. Thanks for pointing this out… am editing the post accordingly.

  • Pingback: Unmetric Unveils Prediction Engine For Social Posts()