BBC Documetary on Big Data

This is a good documentary talking about big data applications and ambitious projects. Being a documentary at times its little dramatic but results in few cases are simply astonishing. 

"In Los Angeles, a remarkable experiment is underway; the police are trying to predict crime, before it even happens. At the heart of the city of London, one trader believes that he has found the secret of making billions with maths. In South Africa, astronomers are attempting to catalogue the entire cosmos. These very different worlds are united by one thing - an extraordinary explosion in data."

Retailers and the Big Data

Somewhere in 2011 I read an article about how much data Walmart collects hourly and how it is used to optimize their business. This incident ignited my interest in big data. I decided to explore more about how retail businesses and big data usage. Here are few interesting finds,

Walmart:

Wal-Mart, a retail giant, handles more than 1m customer transactions every hour, feeding databases estimated at more than 2.5 petabytes—the equivalent of 167 times the books in America's Library of Congress. [1] The magnitude of data is so big that, they recently shifted to 250-node Hadoop cluster. Walmart labs is building their own big data analysis tool and plan to open source them. [2]

The Inception

1662, London

 John Graunt did detailed analysis of bills of mortality and tried to find hidden patterns. He build a model which can predict next bubonic plague in the city.

1880, USA

According to this link here, first big data problem was solved during the census of 1880. A Hollerith tabulating system was used for dealing with huge data of 50 million people. It helped them to complete task of almost 7+ years in just 6 weeks. This was a step ahead in infrastructure required to deal with big data.