Big Data vs. Big Question: Which Comes First

You should first analyze what your business objective is; and be open for the new discovery as well.  

Big Data is still serendipity in most of organizations, and majority of Big Data projects fail to reach business expectation, what’s the best scenario to debunk Big Data puzzle, Big Data vs. Big Question, which one shall come first?

Any analysis needs a purpose; whether Big Data or not, and you usually start with an objective question or hypothesis. Raw on its own has no value in business intelligence. How valuable the data is has nothing to do with the fact whether it falls in the category of Big Data, huge in volume, variety, velocity or voracity. Whether is structured or not. We can still apply analytics, data mining techniques to identify hidden patterns and answer Big Questions with Non Big Data but qualitative and relevant data.

You have to have an idea of what you want to explore. Big Data becomes valuable when it can be converted into knowledge that can answer Big Questions. Setting the machine running and see what comes out is not a viable approach since you still need to tell the machine something to get it running. The important thing is to have a number of questions/hypotheses and test them all - which ones does the data support or disprove? Having a single question might not be enough as your data may never answer it; that is not the fault of the data, just the fault of the question, so you need to try some others. The argument that you need to have a good question to test is based on the reality that questions are potentially infinite. To test all random questions by exhaustive mining would be expensive and that the search for answers, like needles in a haystack, would also be fairly compute intensive.

Most big data projects consume a lot of money and time so you should be very clear on if it is really worth spending that money and when will it exactly give you the ROI. and if that requires a lot of data churning and then step 2) should it be very clear on what output you want from that analysis/churning, and then obviously the cost/time involved in getting that done. You can't be looking for a needle in the haystack, you need to have an idea of what you want to solve first before deciding if big data analytics can do the job. There is no point in simply searching for correlation without a problem or hypothesis in mind; be open for the new discovery, but keep focus on business objectives as well. You may get the correlation but if you fail to find the causation, you big data project fails.

The right buzzword du jour is ACTIONABLE. "Action" being defined in this message as the step that follows knowledge. Uncovering a nugget of knowledge only has value if you can do something with it. If you begin with a question, there's usually an idea of what plan of action you will take depending on the answer. Always keep open mind and curious eyes. Work on discovering unexpected nuggets only when there are no burning questions needing answers. The unexpected nuggets too often have taken a lot of effort just to get the attention they merit because decision making action takers are usually too focused on current objectives to grasp the value of an unexpected nugget.

Sometimes you have to "dig and explore" to find out what you don't easily know or recognize. Framing the right questions are perhaps harder than you think of; know how to formulate questions that have a basis in "predictive" rather than "reactive" thinking. Getting the answer to the "right" questions is the first step toward problem solving. What is worse - Not having the big questions to go with the big data or not knowing what to do next once you have the answers to these big questions? What are the right questions and how long to get an answer has been elusive. Big Data adoption rates and understanding ROI/deliverables are also hard to articulate.

There needs to be a balance. Opportunities trigger the direction and having right problem statement provokes the relevant questions to determine the right solution. This is applicable to all the varieties of data. (1) Start an analysis project with the important questions you want to answer. This will frame everything from the data you collect to the techniques you use. (2) As part of assessing the initial data set, be open to new ideas & findings that come to light either through correlation analysis or simply by evaluating the data. You may be asking the wrong question!

Big data usually brings about a serendipitous pursuit. A focused emphasis on strategic objectives and ensuing strategic data analysis is a much more worthy pursuit. But always be open to the new discovery. Big data is an enabler and not a panacea.

Posted in: Big Data,CIO Debate,Connecting Dots,Leading by Questioning