Three Big HOWs to Tame Big Data

Big data is data that is too large to process using traditional methods.

In its May 2011 report on big data, the McKinsey Global Institute projected a 40 percent annual growth in global data—a doubling of data volumes every two years. Much of this data is complex, unstructured data that doesn’t submit readily to analysis. In fact, big data refers to the volumes of mostly complex, unstructured data whose analysis could yield significant dividends in terms of competitive business advantage, but cannot be analyzed by the existing structured data oriented infrastructure & framework.

Therefore it lies the chal­lenge of big data, at Three WHYs for Big Data, we explored why Big Data is Big Matter, we also further articulate Three Big WHATs to identify Big Data Challenges, and now continue the series on how to overcome the barriers to tame the Big Data?

 1. How Big Is Big Data?

While experts agree that Big Data is big, exactly how big is a matter of debate. IDC forecasts a roughly 50 percent annual growth rate for what it calls the world’s “digital universe,” more than 70 percent of which IDC estimates is generated by consumers and over 20 percent by enterprises. Between 2009 and 2020, the digital universe will swell by a factor of 44 to 35 zettabytes, or 35 million petabytes, IDC predicts

Big data is data that is too large to process using traditional methods. It originated with Web search companies who had the problem of querying very large distributed aggregations of loosely-structured data

 2.  How to start first step to tame Big Data?

If you’re creating large data sets, which most of businesses do today,  you have no choice but to embrace big data management. First Steps to Managing Big Data can be overwhelming and full of confusion, especially if the organization has a lot of existing data and data continue to grow and flow exponentially.

A) Start with business Strategy/goal: As we put in the Three Big Whys for Big Data, Big Data is the means to end, not the end. Start with end: the purpose, the business goals, the strategy;

B) And then develop a Big Data Strategy/map in which you classify:

The types of data that are important to the organization.

Various aspects of your data, such as whether it changes a lot (complexity & variety)  and how fast it’s growing (velocity).

Is much of your data critical information

Prioritize the business problems Big Data can help solve

C) The right Tools and Architecture: using traditional data warehouse solution to search through such vast amounts of often unstructured data can takes weeks, if not months. Hadoop, an open source ecosystem of data management tools and technologies that can help organizations tackle the broad problem of big data analytics, here is another nice article to recommend 7 top tools for taming Big Data.

D) Data Security & Governance: However, IT and business managers should keep cautious when it comes to implement Hadoop, which requires a fair amount of practices and serious attention & solutions to the security of massive amounts of data/ information stored in distributed clusters and potentially in public clouds

3. How to Build the Right Mindset and Culture to gain wisdom from Big Data quantitatively and qualitatively?

Extracting business value and wisdom from big data remains elusive for many organizations. For most companies today, data are abundant and readily available, but not well used. Nearly one in four survey respondents says the vast majority of its company’s data are untapped. Another 53% say they only use about half of their valuable data. Yet 73% say that data collection in their organization has increased over the last year

 The biggest challenge that business faces to manage Big Data today is more cultural than technical. Data oriented business culture need be well established to help every level of organization to make data-driven, better and faster decision. Technically, the Big Data solution need combine computer science, mathematics, statistical analysis, data visualization and even social science to gain insight and foresight about what customers need and what’s the future trend.

 There is a strong link between effective data management strategy and financial performance. Statistically, companies that use data most effectively—strategic data management of Big Data—stand out from the rest. Fifty-three percent of customers say their organizations achieve higher financial performance than their peers, compared with 36% overall.

Big-data analytics must be business led with the right mindset, time to value, and with effective data mining tools, as not all projects will be successful at finding the needle in the haystack. The organizations that can do it successfully will out beat the competitors and become the intelligent business in 21st century.

Twitter: @Pearl_Zhu

Posted in: Big Data,Business Intelligence