Sunday, July 21, 2013

Can Modeling Let the Complexity Shine through

All models are wrong, some models are useful  -George .E.P. Box

Model is by definition an approximation of reality - why would someone want to model reality in full detail and precision. A model is developed to help you with something - answer a question to some degree of precision that is useful or let the complexity shine through with a certain degree of insight. A model will never be perfect, but a model can still be useful if one knows what to use it for. So even highly chaotic systems can be modeled and should be modeled. Modeling serves many purposes beyond forecasting –not the least of which is to device adaptive strategies for highly chaotic environments.

Problems with Modeling: Most problems with models are that people make simplistic assumptions, sometimes fail to make these assumptions explicit or assume that future can be predicted by simply looking at past history. Users of the models don’t bother to look at assumptions, assume that they live in a linear world where everything is normally distributed and proceed to use a point prediction made by a model.

Many “soft factors” can be modeled and should be modeled to guide better decisions: When you choose not to model them, you are essentially saying that they have no impact – so it is still there in your model, but with a multiplier of 0. If soft factors are important to a problem, then a model should incorporate them – yes the results will not be very precise and you will not get a point prediction, but you are likely to have fewer errors than if you completely ignored those factors. There is a large body of work in system dynamics modeling and agent-based modeling where soft factors are routinely modeled and used for many applications. The problem with all the "soft" modeling is that when it comes to reality, and making real-time control decisions with optimization, not marketing or lab type estimates, one tends to want a "good estimate" or a more correct estimate", or a prediction that is "favorable most of the time", note that continuous reference to time, and the "fuzziness" in the colloquial sense. Eventually in the real world, if these predictions are not "safe" enough,  stuff blows up. So then you "constrain" or "limit" solutions with a safety net that is in many cases "intuitive" or "based on experience" to the point where the solution may become over-constrained or useless.

Forecast is Useful: You simply can’t model reality properly, too many permutations, unknowns, interactions, uncertainties, non-linearity, with "singularities" etc. It is true that you cannot make point predictions, but that is not the reason for doing forecasting. The point of forecasting (which implies a range of outcomes) is not to predict what precisely will happen, but to help you make decisions and take actions today. Point predictions are wrong, but forecasts can be useful. The principles to do modeling:
  • Ignore soft factors in models at your own peril. 
  • Job of models are not prediction or perfection; it is to guide actions today. 
  • It is possible to model very complex system behavior via very simple components
  • Understanding the limitations of each model you use is, therefore, the key. Using models without understanding their limitations exposes you to big surprises.


Post a Comment