The Correlation between Big Data and System Thinking

Perhaps there is a point of convergence where Big Data and Systems Thinking begin to lean on each other to understand complex situations, such as digital ecosystem.

Big Data and System Thinking have many “nature” correlation. For example: Can System Thinking be effectively applied as an epistemological perspective, and its methods used to define the algorithms needed for the automated analysis of Big Data, including the definition of the required ontological framework? Or  if resources allow it, is a Big Data also not a bad system to add as resource to the system intelligence analysts. But then that is where the question comes in: Can the connection between the BD and the ST approach be more formalized in a methodology, or should it be left to the arbitrary sense of creativity and intuition of the analysts?

System Thinking could be such ontological framework for Big Data mining: Big Data is just like any other data, only, by using computers instead of just human’s five senses, we can gather an awful lot of it. When using the word Big Data, it’s not just referring to the data per se, but the process of collecting, collating and analyzing, in short Data Mining.  But there’re big issues about Big Data, the smaller issue is the volume of data, the larger issue is the multiplicity of conditional relationships among the bits of data, and the even larger issue is the rate of morphing of the extent, variety and ambiguity of the data. Big Data is typically rife with multiple interrelationships. Accordingly, processing time increases as the square of the number of relationships per data element unless you are using some kind of associative processor. Big Data can be envisaged, accessed and have interfaces built and application which utilize it, which are informed by various frameworks and understandings. This process needs to be automated, because there is simply way too much data to sort through, impossible to sort out manually, or, at any rate, way too time consuming. So Data Mining is nothing but a method, to be applied within an epistemological perspective. Being automated requires algorithms, which in turn require data scientists to describe explicitly an ontological framework. And systems thinking might be one such framework, all that needed is to define what exactly one means by systems thinking in such a context.

While Big Data can uncover correlations between data, it doesn't reveal causation: Sometimes, it doesn't really matter, but other times, it might — in ways we’re not always aware of. But when Big Data works hand in hand with System Thinking, there are better chances to discover both WHAT and WHY. Having more data, and more ways to process it, means that we can develop different kinds of theories and models, but even so, with more data, we would perhaps have to rely more on correlation using complex modeling rather than on our instincts to identify causal relations. So perhaps there is a point of convergence where BD and ST begin to lean on each other to understand complex situations - especially where the data sets are huge - in a more comprehensive way.

The connection between the Big Data and the System Thinking approach can become more formalized in a methodology: Is it possible to connect a big data approach to a systems model? By its very nature, Big Data contains an enormous number of specific types or forms of information and it would be quite hard to build a single system model that could accurately reflect this. On the other hand, building information system models to illustrate how to manage the contents of a big data repository is entirely feasible. The fact that Big Data methods are by no means immediately flexible (the structure, the semantic engine, and so forth are not changeable in a second, and require quite a lot of effort to set up), though the nature of the connection between the two methods is far from trivial. System Thinking can help in the formulation of the ontology on the basis of which the automated analysis of Big Data is carried out, in particular, by identifying classes, subclasses, elements and attributes as parts and relationships of a system model. So the opinion that doing this can allow for an organic growth of the ontology (and the consequent increase in complexity and level of definition of the data mining algorithms), while retaining control over the whole high level picture, with the consequent increase in reliability and soundness of the final reports, which are the true objectives of the whole project.

Understanding Big Data is in effect understanding properties of a global Web. The shift is from envisaging the web as a glorified information system to envisaging it as an infrastructure of hyper-connectivity; the digital paradigm also moves from which information was king, to one in which reciprocal connectivity rules.The overarching purpose of technology should be to empower people to transcend the current level of thinking that created the existing problems in the first place to the next level of thinking by leveraging Big Data, System Thinking and Creativity. So the domain logic of Big Data systems should account for the fundamental need rather than merely support status quo. Hence, Big Data has to evolve Big Thinking as well.

Posted in: Big Data,Connecting Dots,Systems Wisdom