Is Self-Service Reporting a Biggest Challenge for Big Data

The key is that you are able to create/maintain a system that has 'right data available at right time.’
Fundamentally, if you use BI for decision making, the cost of failure will determine what you can spent. Any executive that is responsible for ROIs should understand it and realize that the risk is not in the ability to graph a data set, but in the risk of using bad data. Given the skills required to validate these types of models and quantify the risk, you need a real data science capability. So is self-service reporting one of the biggest challenges for Big Data?

Self service recently has been available to management through ease of use of analytic and visualization tools. One of the major challenges in Self Service  is not only how quickly data is ingested from various data sources, but making sense and presenting meaningful collaborative information in a matter of seconds. Making sense includes correlation of various events, detection of causalities, identification of anomalies, ability to dynamically ask questions about specific situations that can span zeta-bytes of data, seeing an impact in real time etc… Claiming all the necessary capabilities, but failing to scale makes Self Service unusable. The challenge is the expertise of management to understand the following fundamental items:

1). Data quality

2). Source of Data

3). Result trends

4). Causality (many documented failures)

5). Scope (range of your data and prediction)

6). Sampling quality (geographic and time dimensions need special attention)

7). Validation (hypothesis testing typically requires additional experiments)

The key is an integrated platform for data management with an enhanced visualisation layer as the kicker. Things are changing faster than you think…Too long have you had silos for all the bits with so much resource and time and cost to pull it all together. Fast and quality self service is important as it can solve many business users problems immediately and quickly given an idea of the art of the possible and a view of what's needed to really solve the data cleaning piece. Of course for the real disparate data and data quality challenges, you need a fast and cost effective continuum from self service analytics to detailed data intelligence analytics (BI on your data itself), then to the integrated data management and visualisation all rolled up on one product. Filtering by data source would be nice. Perhaps even a data driven development framework talking the coding out of building customized solutions to the problems you are facing on the ground would also help.

Self Serving Reporting is tied to internal decision systems driven by the management culture of companies - It is a supportive technology not a disruptive technology - Big Data & Big Modeling & Big OLAP are transforming innovations that require a unique approach - First, you need to know what you are doing? why your are doing it? what is the goal? what is the expected value creation? how will this value be measured ? and then there is the change in culture to adapt and use. Self Service reporting in BI or Big Data is not just a challenge; but it is a complete paradigm shift. In simple terms, the envisioning behind the Self Service Reporting is to enable the ‘business users design/build their own reports by choosing the required subject area, its components and creating their own data-views (slice-n-dice as per their visualization).’ To get such self-service BI system in place is quite a daunting task. The very essence of minimizing the IT cost such that the demand of creating new reports or means of analyzing different data does not necessarily need IT professionals is quite a myth. The fast-paced business demands are so rapid that you constantly need to a) Refactor the code, b) Maintain data-integrity and quality c) Change the database schema and its underlying data views such that the desired drill-downs are made possible. So, take the simple example of an executive trying to make a decision on business expansion plans. The quality of that prediction will be heavily dependent on the way existing data is correlated to demographics, infrastructure capabilities, and seasonalities. Self-serving that type of analysis has very little to do with drawing a trend graph, but everything to do with data curation and validation in the context of the question. Not rocket science but not self-service BI either.

The key is that you are able to create/maintain a system that has 'right data available at right time.’ The ROI factor is an important attribute for considering Self Service BI systems. All these requires: a) Business Users (domain experts) to constantly work with these BI technologist in creating and maintaining the underlying business intelligence systems b) It requires constant training for the business users on the chosen technology stack such that they are able to manage and create their own objects, reports for required analytics. Because more often, Data Scientists have little or no knowledge of situations they would be asked to address. They will be buried by the required effort and number of requests. In many cases, by the time Data Scientists complete addressing some of the reports, the situations may change and the created reports become obsolete, resulting in a waste of time. But most importantly the actual user that needs answers is left in dark. It is a never ending game where the users almost never get the needed information timely. They also do not have any ability to dynamically change their questions and see business behavior from different angles

Self service reporting is a challenge, has been a challenge and will likely be a challenge in the years to come. The purpose is tied to internal decision systems driven by the management culture of companies, laser focus on creating business value, and accelerating business speed.

Posted in: Big Data