Big Data – Pearls or Sand?
I recently read yet another article about how the amount of data is expected to grow to 44 Zetabytes (what exactly is that?) by 2020 which represents 10 times the volume from 2013.
I get it – these are large numbers, even though must of us have a very vague notion of exactly how large. With this kind of growth, one can’t help wondering whether the digital world is going to explode. Surely, we can’t sustain this kind of growth to the next decade, let alone in the long term?
Marketeers, journalists and everyone trying to run their business based on fear of numbers like to keep talking about this and show beautiful graphs and reports on why Big Data is important.
Is it Truly Big Data?
If we take a pause and think about it, we will quickly realize that all data is not created equal. Let’s consider this example.
Advanced kitchen appliances may help me track my energy use every second of every day. If I drill-down into these beautiful charts and see a short spike at say 1 PM yesterday, what does that tell me? What can I do about it? Do I really need this level of detail?
If instead I am shown average consumption per day, over a month, that will tell me what the trend is.
In this example then, having all of the per second data is useless – it is a grain of sand rather than a pearl of wisdom. We can reduce both the amount of data collected (by sampling much less frequently) as well as the data stored by focusing on what the business use case for the data is.
If the data is not useful, it is not Big Data. Big Data requires that we consider what insights we need to run our businesses (or the world) and then build the intelligence to gather, store and analyze the right data that can be put to use. These are the pearls. Discard the rest.