Big Data is hot trend, but Big Data project has very low success rate, so the real question is how organizations deal with the situation. Do they continue to pour resource into a failing project, or do they shut it down early to learn what they can get from the experience and move forward? What are the key factors to be considered in order to drive Big Data project success?
1. Have a Business Goal in Mind
It needs to be clear what the outcomes of the project will be, who can/will benefit and how they can/will benefit. It is critical that at the end of the day you have the right team in place to use the insights that come from the projects, what better decision will you be able to make as a result of the analysis and how will you drive and track those better decisions,
It is understandable that many BDA projects are started but in a competitive internal environment, so most of these projects have no defined business driver or requirement. As long as Big Data projects continue to be treated as pure IT projects, the failures will continue to pile up. Getting the machinery running is generally not the issue, it is the failure to deliver value because the end-users and core decision makers are ill equipped to "get it" and can not extract and use the inherent value in the mass of data at their fingertips. Converting big data, into understandable, digestible, useful and actionable information is the business purpose from Big Data project.
2 Assume Data Imperfection & Immature Technology
Data will be incomplete and less than pristine (duplicates, incompleteness etc.) in the Big Data analytics, and the analytics must accommodate the real state of the data. By the time a project gets going, there's probably around 30% more data (both in volume and diversity) available. The analytics will need to be refined and adapted because you don't know everything when you start. Your framework for developing and deploying the analytics should support an iterative development approach
Lack of resources, high data volume, new technology that is not production ready, variety of data sources and high data volume is plaguing the industry. This is also encouraging innovation at a very fast rate, but it will take time for industry to mature and stabilize and then the success rate will increase and will come more in line with industry standard success rate of projects or even higher.
3. Big Data Talent
The pitfalls of most of these projects is because Big Data seems like a disruptive technology and the old users of the systems are not equipped with these new concepts and will go a long way to see it fails.
There are generally two distinct skill-sets, technical on the architecture side and the analytical skill set to create action based on the information. You need a solid team for both to have a shot. If not you wasted time, money, effort and potentially damaged prospects for the next projects coming down the pike. For example if your new project can tie your
and other support systems all up in a nice wrapper and you can see how
purchases, the loyalty program and your customer service activities can help
someone put their customers into groups, it doesn't do any good if there isn't
someone in marketing or in a specialized analytical role/group that can DO
something with that information, like recognizing triggers that a customer is
more/less receptive to a certain kind of marketing/up-sell/cross-sell messaging
or that there seems to be a risk of the customer leaving. CRM
"The statistician is no longer an alchemist expected to produce gold from any worthless material offered him. He is more like a chemist capable of assaying exactly how much of value it contains, and capable also of extracting this amount, and no more. In these circumstances, it would be foolish to commend a statistician because his results are precise or to reprove because they are not. If he is competent in his craft, the value of the result follows solely from the value of the material given him. It contains so much information and no more. His job is only to produce what it contains." Ronald A. Fisher
4. Collaborate Effective Processes with Right Talent
Poor planning and coordination are organization and project management related, however, the nature of most big data projects requires the ability to read and rapidly process variety of data, high volume data sources, making coordination critical. Things get complicated very quickly as the structure of these data sources and the business needs continue to evolve. If organizations are unable to staff the right kind of data scientist(s) who have all the skills to take the project forward, and understand the business needs, they end up staffing a lot of individuals to fulfill the skill set needs. This creates a communication and coordination issues...communication touch points increase with the square of number of individuals involved in the project .. and we know how things go from there.
That has remained constant, or at least linearly increased, is the number of prospective users of data who seek magic over math, confuse computational speed with statistical precision, and generally would prefer the answers they seek over the answers that past analysis has given them regardless of scale.
On the business side, Businesses haven’t yet fully understand how to redesign business processes so that they can take advantage of what big data can potentially tell. Poor cooperation, poor planning and lack of skills should be categorized under organizational failures rather than Big Data centered project failures though.
5. The Big Data Life Cycle Management
The life cycle of big data is to evaluate capacity planning for Big data services through timely intervals and expected growth cycles. The velocity and a metric for relevant time sensitivity is important in the collection processes of what you are analyzing and for what purpose pursuant to the predictive analytics needed to support the model coupled with levels of complexity.
For a couple of decades, IT has been pushing towards a "single source of truth" in data management. A lot of money and technology is aimed at getting everything to line up and agree. There is a whole "Master Data Management" industry evolved out of the desire for consistency and correctness - never mind the legal requirements to get the data "right". Big data often threatens all this and makes a lot of IT shops nervous, so projects get sidetracked, slowed or abandoned because it takes the new angle and multi-dimensional lens in capturing insight upon Big Data.
Until organizations get an operational context that can respond to new capabilities, the project can be technically successful, but never deliver enough business value to gain support. In that situation you probably won't get to do another one.... Big Data has all tough characteristics, how to build the Big Data modeling framework as a systematic approach to master volume, velocity and complexity needs to be thought through.
Mastering Big Data is more as journey, not a one time project,. It is the nature of the beast that "noise over signal" dominates every epistemological effort. A certain amount of project failure ought to be tolerated if innovation is being sought. With considering all key factors listed above, organizations can make progress in driving Big Data success.