The modern world depends on data. Everywhere you look reams of complex information is used to guide businesses and individuals through life, and people are becoming increasingly concerned with the trail of digital data footprints they are leaving across the web.
From the latest obesity statistics and the thousands upon thousands of daily weather measurements, to the hourly peaks and troughs of what is trending on Twitter, it can sometimes feel like you’re drowning in a sea of unfiltered, unprocessed and unintelligible data.
This is the world of Big Data and businesses are going to have to adapt to survive.
What is Big Data?
Because businesses seem to be reliant on an ever-growing amount of data they are increasingly having to contend with huge datasets that are pushing their capabilities to the limits.
Examples of the amounts of data we are talking about here include modern jet liners generating up to half a terabyte of sensor information for every flight; trading warehouses that deal with petabytes of information and then the Large Hadron Collider – personally producing around 25 petabytes a year.
This is the heart of what Big Data is and the challenges it poses to companies all over the world. The term itself refers to datasets that are so huge that traditional processing applications and database management tools are not up to the job. But it is not just the volume of data that is a problem; it is also the speed it is coming in and the unstructured nature.
The nature of Big Data has created some pretty unique analytical problems, because it does not function in same way as more traditional business intelligence, where the process of summing a known value gives you a result (such as turning order sales to year to date sales figures). The huge nature of Big Data means that a certain amount of guesswork and visual and statistical models are needed, and these will need regular revision.
Where has all this data come from?
The truly jaw droppingly, mind bogglingly answer to the question of why the issue of Big Data has seemingly sprung up from nowhere, is that 90% of the worlds data has been created since 2010.
The reasons for this steep upward curve in the amount of data that is being produced are cited as a combination of a range of technological and social factors.
Firstly, the explosion of demand in the Smartphone market and our continuing crack-like addiction to social media means that people are simply creating more and more data, and that is needed by companies to accurately assess what is going on in the marketplace.
Secondly, the computational ability of the machines and tools that we can now utilise has simply increased to the level where we can begin to see what lies beyond and what kind of technology we require to be able to deal with it.
Image by: Marius B
How your company should react to Big Data
The increasing pace of this build up of data means that businesses are going to have to react pretty quickly or fall behind their more savvy competitors.
For retail companies this is especially important in the realms of marketing and ecommerce/multichannel retail, but for all IT managers this is going to mean steering through a period of transition from traditional IT architecture to something more Big Data friendly.
The first thing that needs to be done is to focus in on the areas of the business where Big Data can be the most useful in improving operations or services. You then need to take a look at your data management and analytics capabilities and see where the holes are, so that you know where to focus your energy and future training.
An important thing to remember is that one of the things that define Big Data is an unpredictability and unstructured nature, so be prepared for some surprises and challenges.
Some useful Big Data tools
Although the challenges of Big Data are only recently starting to weigh heavily on people’s minds, there are already a number of great tools that have been developed to help ease the potentially Sisyphean burden of IT managers and team members.
Here is a brief list of some Big Data tools you may find useful:
- BitDeli can let programmers measure application metrics in Python scripts
- Continuuity can help you abstract the data involved in connecting Hadoop and HBase clusters, whether for yourself or your clients
- Flurry is great for creating and measuring mobile apps and the data they create
Well there you are. An introduction to Big Data that I hope wasn’t too abstract or stressful for everyone involved. Have fun!
What are your opinions on Big Data and what it means for the future of IT?
License: Creative Commons image source