The cost-benefit analysis of mega government projects often turns out to be wrong, reveals a research paper
13-Jan-2022 •Anand Tandon
The National Infrastructure Pipeline website opens with the following statement: "The National Infrastructure Pipeline (NIP) for FY 2019-25 is a first-of-its-kind, whole-of-government exercise to provide world-class infrastructure to citizens and improving their quality of life. It aims to improve project preparation and attract investments into infrastructure." The pipeline estimates a total expenditure of Rs 111 trillion between 2020 and 2025. A break-up by sector of the projected capital expenditure is given in the chart 'Sector-wise break-up of capital expenditure...'.
Most large projects tend to overshoot costs. For example, India is currently executing the largest number of metro rail projects in the world. One such project, now complete, is the 72 km metro constructed by L&T in Hyderabad. As against the original planned investment of about Rs 164 billion, the overrun is approximately Rs 50 billion. Importantly, the project was to break-even once passenger traffic increased beyond one million per day. Pre-COVID, peak traffic achieved remained at half of that level. Currently, it is at 25 per cent of break-even. One of the consequences of this is that new investors are difficult to find for increasing the network, which itself has deleterious effects on traffic volumes.
This raises the question on how the cost-benefit analysis of the project was carried out and if that was correct. A recent paper (Sep-Oct 2021) by Bent Flyvbjerg and Dick Bester of the University of Oxford titled 'The Cost-Benefit Fallacy: Why Cost-Benefit Analysis Is Broken and How to Fix It' has some important insights to offer.
Biases in cost-benefit analysis of public investment projects
A cost-benefit analysis is a means of evaluating relative attractiveness of projects when there is a slate of projects to choose from. But what if the estimates of cost and benefits suffered from systematic biases that made them a case of 'garbage in, garbage out'? Flyvbjerg and Bester find it to be indeed the case.
For their study, the professors used data from 104 countries, spread over 1927-2013 and included investments across bridges, buildings, rapid bus transit, dams, power plants, rail, road and tunnels. Data on actual costs and benefits are difficult to get. The study was careful to include all data that could be considered reliable from any published source. Construction costs were taken as 'actual cost', while first-year benefits were taken as proxy of 'benefits'. This was compared to the estimated costs and benefits at the time of making the investment decision. The results can be summarised in the two histograms showing the relative frequency of cost and benefit overruns, respectively, pooling all investments (N = 2,062).
The paper explains, "If estimates of costs and benefits were largely accurate and unbiased, both histograms would be more or less symmetrically distributed around 1 (on the x-axis) with thin tails (technically, we would expect the log of the distribution to be exactly symmetric around one). In fact, each histogram is highly asymmetrical, not distributed around 1, and fat-tailed (to the left for benefits, to the right for costs). On average, cost overruns are compounded by benefit shortfalls instead of being mitigated by benefit overruns. The findings are overwhelmingly statistically significant (p < 0.0001)."
Quoting further from the paper, "Cost overruns are highly inaccurate and biased for public investments, ranging from an average cost overrun for roads of 24 percent to dams at 85 percent, in real terms. The fact that the data show bias is a crucial finding, because whereas errors cancel out, biases compound. Biases - and especially biases with fat tails, as here - are therefore notably worse news than errors in public investment planning". On benefits, the authors conclude "benefit estimates are also inaccurate, though less so than cost estimates. Bus rapid transit and rail investments have significant benefit shortfalls, on average respectively 58 percent and 34 percent."
The conclusion is bad news for using cost-benefit analysis for public investments. An interesting finding was that across all investment types - from bridges to roads and tunnels - in all cases, 'costs' were underestimated and 'benefits' overestimated. The ex-ante benefit-cost ratio was found to be overestimated by between 50 and 200 per cent and this was true across geographies and time. Optimism runs deep when estimating benefits and frugality when estimating costs!
Why does the bias persist?
Costs beyond plan are often explained away thus: unexpected developments when ground is broken to start the construction process, changes in input prices and delays caused by legal or political influences. What is inexplicable is that despite experience showing that situations such as these almost inevitably arise in every project, forecasts continue to remain bullish. This reflects behavioural bias more than estimation errors. The authors conclude that the advice to public investment planners and managers should be "your biggest risk is you".
The authors suggest various ways to counteract this bias. One possibility is to adjust forecasts using historical data for similar projects. Another could be by penalising forecasters who make large errors, while rewarding those who are more accurate. The third method could be to use 'unbiased experts', along with public representatives to decide on priorities - on the basis that in a democracy the public's concerns, even if unreasonable, should be taken up by policymakers.
Implications for India
While all kinds of feasibility studies and analyses are carried out in India before a project is sanctioned, the ones that see the light of day are inevitably those that seem important to the political ruling class. At one level, that could be classified as a reflection of what the public wants. What is also clear, however, is that in a democracy, only a small minority of individuals or institutions have a view on any issue that they feel strongly about and are willing to advocate, while the majority are silent or indifferent. Consequently, the vocal minority often determines economic and development agenda. Large projects that are the favourites of the political class are most likely to perform the worst in terms of cost-benefit but continue to garner political patronage. A pipeline of large projects will only ensure that capital deployment is carried out ineffectively and the GDP multiplier from such investments needs to be tempered down.