Intuition vs. Big Data: Think Fast & Think Slow? ~ Future of CIO

“Intelligence is not only the ability to reason; it is also the ability to find relevant material in memory and to deploy attention when needed.” Daniel Kahneman, Thinking, Fast and Slow

“Thinking, Fast and Slow” is a book written by Nobel Prize winner in Economics Daniel Kahneman (DK) which summarizes research that he conducted over decades. The book's central thesis is a dichotomy between two modes of thought: System 1 is fast, instinctive and emotional; System 2 is slower, more deliberative, and more logical. What Kahneman calls “fast” and “slow” thinking, corresponds to what cognitive neuroscience and cognitive science today call “implicit” and “explicit” cognition. Implicit cognition is automatic, unconscious or intuitive (gut feeling) cognition. Gut feeling and another implicit cognition is very good at prediction as long as the environment is highly predictable and stable. From those case studies, it seems people are really good at guessing means, not so good at comparing them.

Now, Big Data even expands the new lens of machine intelligence, to complement human’s thinking, the interesting debate is: Does Big Data make “gut feeling” irrelevant?

1. “Gut Feeling” is still Special

'Gut feeling' depends on how we define it, and it can mean various things. 'Gut feeling', is the key to really solving problems in some cases, such as human factors which can think outside the box.

The Vision thing: An example of where our cognitive abilities seem to outperform any data-driven methods is the vision. Though "gut feelings" and vision might be pretty different animals, it seemingly relevant to elevate the comparison to our cognitive systems' ability to process information into an actionable result quickly, seemingly without conscious cognition. Vision falls solidly into this category and is one area where we seem to be much better than any "computer" thus far. Thus, gut feelings' works very well for highly creative, visionary people, in particular, people able and not afraid to leverage contrarian views. It works badly for some others.

Intelligence of Unconsciousness: Following the Psychologist Gerd Gigerenzer's assumption that "gut feelings" represent "intelligence of the unconscious", he makes the point that many moral judgments, and indeed the moral judgments of institutions, are made without full conscious awareness. In fact, it can be very difficult to express at times why we believe something to be "wrong" or not.

Many social interactions are based on 'senses' of trust (or lack thereof) and other "gut feelings" and not logic. "Big Data" may not be replacing these sorts of judgments. That is not to say that "big data" might not eventually tell us something about when our moral or social intuitions are confirmed or not, that we simply observe the world as it is and make a conscious decision about it

2. Big Data Makes Big Difference

DK would also suggest gut feelings are bad for driving decisions about highly complex interactive systems. In fact, our brains and sensory systems have built-in biases that may subtly (or not so subtly!) influence both our perceptions and decision-making. Statistical prediction and "intuition" will definitely NOT correspond in many cases due to various built-in biases (endowment effect, short-cut heuristics that are more sensitive to examples an imagery, risk-aversion., etc.) in human decision making. In other words, it is certainly true that analytic methods will be superior in many cases.

Culture & Decision Process Matter: The result will possibly be a spectrum of relevance relative to the prevalent business culture & decision-making process. Some companies and cultures are and have been data driven for as long as they have existed, they will be on the extreme side of the spectrum, Analytics & big data will just take them to new levels of science-driven business management. On the other side of the spectrum, serious conflict may arise if big data leads to completely new insights or the need for serious business re-structuring that conflict with what the gut says

Big Data is what sets to make "gut feeling" increasingly relevant. 'gut feeling' means the form of human intelligence as expressed behaviorally as 'subjectivity' - made 'operant' through 'self-reference' as 'sentiment' data-capture.. The Scientific Study of Subjectivity -is already over-and-beyond Bayesian statistical understandings of subjectivity; it demonstrates the structure of 'gut feeling' as mathematical 'fit' within specifically contextual human decision-making processes. "Gut feeling" finds its expression in behavioral action, as expressed in hierarchical form through natural language processing- enables the modeling into Artificial Intelligence.

More brains, and less Gut: Then as the data accumulates, the importance of some gut feeling will dwindle to be replaced with the data-driven results. Since we are talking about big-data, the observations should quickly overwhelm the original gut reactions. So the predictions are like societal progress - over time, you see more brains and less gut.

Some keys for communicating unintuitive results are:

1) Make really sure your are right. If other people start finding errors in the analysis you present
then you can lose credibility quickly.

2) Socialize your results before you get to the big meeting. This will help with and also means
that at least some folks in the big meeting will already be convinced.

3) Break down the analysis into parts that can be easily understood. This may mean you have
to simplify your analysis and maybe weaken the results, but getting something done is better than
not getting something done because you can't explain it.

4) Choose your battles. Convincing people of something that does not fit their intuition can be
hard, painful and time consuming. Remember you have other stuff that needs to get done as well
and the goal is to successfully execute on as much as possible to deliver the most value possible.

3. Hybrid Solution is Superior

Basing a decision on data, experience and previous failures is better than just data no experience or no data and much experience.

Are you looking at the "signal" or the "noise?" Did you find any of sample instances? How do you know? How do you know that you don't know? There are many instances when looking at the results of an analysis and it just didn't "smell" right - this isn't an ‘either’ ‘or’ question. - We should definitely and clearly rely on analysis of that data (assuming rigor), and equally as importantly, remember that even our subjective experience has value too

Machine Intelligence converging with Human Wisdom: Combine technology with Intelligence talent, meaning how to use machines- advanced technology and algorithms analysis together with the human source- who can understand trends and analyze them while understanding the market needs. There are many things machines can do, yet they do lack the human factors such as creativity, and so, businesses can provide up-to-date ongoing information while using models and technological machinery, not undermining the human source which brings creativity among other important qualities.

Measurement error (the traditional "hard" science concept) is much more complicated in the social metrics arena. Human intuition is a decent thing to use to calibrate a model's real world-ness. But making sure your data is measuring what you want, and measuring it in a relevant way should always be one of the first steps in building a model (this step may be revisited throughout the design/build process).

Leverage both pieces accordingly: The value from a gut feeling is probably setting the context of the problem up and making sure we are using the right data. Secondly, It’s always important to use more than one source of information analysis to drive decisions. Just using one method, gut feeling or data driven, to drive decisions is bad risk management. Any application of data sciences without a balance of 'gut feeling' or business acumen would not do justice to the application of insights. The data can only tell you what has happened and what may happen but it won't apply experiences and principles.

Data is factual and driving factors with which one can produce reports, charts and prepare concise material for analysis. The process to make decision is much more dependent on the skills, experience and belief system (from a data perspective) of the decision maker. Those who have been brought up on 'gut feeling' and corporate politics need overwhelming evidence to follow the data. Those of a more analytic/scientific bent need much less persuading and will follow where the data leads; but with the sapient touch, some of which turn out to be essential aspects of human nature, perception, cognition, choice, and satisfaction

Key Checking Points:

-Do we believe what Big Data tells, period?
-Do we go with intuition and use Big Data only if it supports the Gut?
-The "middle way" is to look closely at the results of the analysis:

-See if the data tells a cohesive story (no serious ambiguities).
- See if the data conflicts with intuition for this environment.
- If there are conflicts, go back and re-examine the data.
- Accept the conflicts and search for explanations.
- Make sure these results are reasonably transparent so they can be presented to decision makers
- Socialize the results broadly.

In summary, don't ignore gut feelings, but on the other hand don't be a slave to them. Gut feeling is important but this needs to be supported by good logic / data. Big data has ability to provide good reasoning and it can support when somebody having gut feeling, but unable to convince with a good data. In fact, people have gut feeling about any case can provide a good input for big data analytics, hence there is no reason for gut feeling becoming irrelevant. No matter how valuable the Data is, we still need the human factor in making final decisions,