Saturday, October 26, 2013

Is Big Data ‘Bigger’ than just Analytics?

Big Data project involves Engineering practice, Science and Arts, it's an inter-disciplinary pursuit.

Big Data has been defined as: 'high volume, velocity and or variety of information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision-making, and process automation.”. And there are many definitions you can find for Big Data and it is all about using technologies that were not existed 10 years ago that allow you to process great amount of data in various forms and create Value/Insight out of it. 

Analytics is part of the equation, but isn't "Big" data concerned more with the volume and complexity of data available. The statistical and analytic tools available haven't changed dramatically over the last ten years but the different types of data and behavioral interactions that are now captured and stored have multiplied exponentially. The customers know the value of their Big Data; but they have difficulty to just store the data while this is probably less than 1% of the challenge.  "Big" data is more about how captured data is described/codified/structured in a way that enables data to be "remixed" to solve problems, provide new insights or create new services.

There are many challenging pieces in Big Data. Analytics is effectively one piece; but it is only part of the equation. Departments were receptive to the improvements. The technology in these departments swings between two extremes: advances that enable better, faster performance, and equipment that becomes easily inundated by data demand. So there are many other challenging pieces like
1): Sensors technologies
2): Technologies to transport this tsunami of data
3): Technologies to store
4): Technologies to compute and extract the value from the noise
5): Need billions of brains to find new ideas,  "some brains will be engineered"
Clearly there’s an endless field of opportunities to solve important problems
6) Avoid Big Data pitfall, it is called "dark data" or ‘WONR’-Write Once Never Read’ 

Big Data project involves Engineering practice, Science and Arts (This is the most interesting part when you need to be creative and find the question you want to ask). And it’s also about imagination, and having an understanding of the working principles of an industry, vertical, or a problem. Then figuring out a way to improve it or change the mechanism. Analytics is the analysis of historical data done by BI tool. Big Data is the challenge faced by cross-organizational boundary in storing, processing, & reporting of nearly 80% of the unstructured data collected in this world. 

Key Big Data issue is not storage or processing of data. BUT processing "any data type" from source, and able to send to target "requested" data type. It can be any data type too. So the potential issues include:
1). "Open Data Processing" Frameworks/ Architectures
2). Data Quality, a must if you have legacy data to be moved into big data suite
. Data Integration, if they come from different stacks/systems in different formats/type
4). Self Service – So make sure that all above components work together seamlessly. Also make use of monitoring & automation languages or tools.

Big Data is not just the storage and data analysis; it's the complete ecosystem from capture with new instruments or sensors to the emerging neuro-morphic analytics; this is the complete redesign of industry around the focus on extreme data. The drive for more data and the technological advances to support that demand aligned at just the right moment, Now with Big data, you can close this loop of analytics and action as continuous and iterative process, but it's also created an insatiable hunger -- a hype factor -- that has taken the industry by storm, more precisely, it reflects the fundamental shift to converge machine intelligence with human wisdom.   

There's risk in managing Big Data, but there is a much higher risk and failure not making any move in the Big Data direction in particular in this ultra competitive world. Big Data is a reality, more than analytics. 


Post a Comment