The problem with complex problems is that they are open, with diverging views among stakeholders.
The industry study suggests that half of all large IT projects defined as those with initial price tags exceeding $15 million, massively blow their budgets. On average, large IT projects run 45 percent over budget and 7 percent over time, while delivering 55+ percent less value than predicted. So CIO as IT and business leader: How much do you trust your budget estimates? To which degree, can you say you actually have sort of method or approach that would enable you to systematically deliver proper (right, accurate, etc) estimates?
There is a clear difference between a budget and an estimate. Estimates are always ranges and are but one consideration in developing a budget. A budget is a single number used for tracking progress. But you first need to secure a proper budget, or you'll be in risk of adding to the failure statistics. Estimates are one thing. Budgets are another. Your terminology confuses the issue and thus increases the probability of poor estimates. Research shows that high under- estimation rates are not due to lack of reasoning skills, but actually a bias, the same bias that makes you successful in the most routine task, but that fails to separate wrong gut estimates from reasoned, well - based statistical procedures.
Developing cost and duration estimates for a project is a challenge. But it is easier if you recognize that estimates are but one input to the budgeting process. And it's easier still if you have a proper project life-cycle and descent systems architecture. The real issue is the lack of validity of claims about the possibility of predicting the cost in a systematic way for really complex problems. Is it more information really useful regarding how much costs estimates you will need? Regardless the tool you choose for the estimate and the information you are provided (be it too much or too little), your estimate for complex problems will be nothing else than a guess backed by the tools and the fact that nobody else knows how much it should cost. Many would say, "if we had full vision of what´s in the path for us, we could build much better estimates." But you cannot have a full vision of a large complex project from the start. It has never been possible to have that. It will never be possible to have that. So asking for it is unrealistic. But you can develop a full vision in steps with a well-defined project life-cycle.
Agile budget planning needs to be “hybrid” to mix top-down estimate with bottom up adjustment. For many years, IT organizations know that complex projects should be broken up into smaller pieces, the phases of a project life-cycle, and sub-projects whose main purpose is to provide proof- of-concept, or to otherwise develop a better understanding of the unknowns. However, Agile budgeting needs to take mixed approach, besides bottoms-up adjustment, the top-down approach means that you determine how much you are willing to invest in solving a problem or delivering a capability, and then you deliver against that until enough of the problem is solved to satisfy your needs or the money runs out, at which point you have gotten your highest priority needs met for the money that you had. Many great organizations already deliver accurate estimates consistently, even in software development. It requires:
* A well-defined project life-cycle approach (this can include agile)
* Well-defined scope, consistent with what's possible for the current phase (or iteration)
* Reviewing the estimates at the end of each phase (or iteration)
* The willingness to spend some time learning how to estimate
* The willingness to spend some time developing the estimates
*The willingness to defend your estimates when subjected to political pressure
The structure of much of budget estimating for big IT projects will tend to under-estimate the time and cost. if you had full vision of what´s in the path for the complex problem solving, you could build much better estimates. The problem with complex problems is that they are open, with diverging views among stakeholders. Some stakeholders will not see a problem at all. You estimate the time and cost for all the project activities that you can think of and include a consideration of the interconnections. However, there are almost always things you did not consider or problems that pop up. In a sense, a critical path is the shortest possible time to completion, not the expected time to completion. All you need is a number of stages each with some probability of unanticipated problems to end up with some unanticipated problems being almost a certainty. So generally speaking, you can systematically define and estimate, thus plan any problem, project and phenomenon, regardless of its size, but not always from the start. It has to be done in pieces. You have to recognize that you can't predict the final total with much accuracy at the start. And if you are in IT, you have to recognize that this phenomenon is NOT unique to large, complex IT projects. It is true for large, complex projects in EVERY domain.
The facts say that most big projects are losing budgetary grip and that the estimates didn´t take into account of those slippages at estimation time. In hindsight, we will always find an explanation for failure. The point is how to learn from the failure. That you are not fighting against the lack of skills or best practices or intelligence, but against your own lack of human ability to avoid your biases and to question if the best practices, concepts, and tools you used yesterday are applicable today. And it is the time to reflect and develop your next budget management practices to make continuous improvement.