I’ve been thinking a great deal recently about how policymakers can address the challenges of stalled investment in the energy markets. My post yesterday on system tightness underlines the importance of the challenge, so I was pleased to come across this excellent paper from Michael Hogan at The Regulatory Assistance Project (RAP).

The paper outlines, in a way accessible to those unfamiliar with energy market mechanics, the different approaches available to address what is often described as the “missing money” problem, in order to generate appropriate investment signals. I will summarise the key points here.

What is the “missing money” problem?

Historically energy markets were dominated by virtually integrated state monopolies, who often subsidised the cost of electricity during periods of shortage through higher prices at other times. When the markets de-regulated and the monopolies broken up, regulators focused on stimulating competition, as without effective competition, markets become vulnerable to abusive pricing practices.

In theory, prices in commodity markets reflect the price at which supply and demand is balanced, however electricity markets have two features which undermine this process. The first is that electricity is difficult and expensive to store, increasing the market’s vulnerability to abusive withholding of supply. The second is that electricity demand is relatively inelastic as retail prices do not vary greatly in time meaning that demand does not respond to changes in wholesale pricing, except possibly over long time horizons.

To combat these challenges, regulators introduced various interventions, including price caps, which begin to separate prices from costs. Other price-distorting mechanisms include procurement of emergency reserves which can enter the market at artificially low prices. While each of these interventions had sound motives, and effective competition is an absolute pre-requisite for proper price formation, the effect of these interventions is to distort prices, which can interfere with investment signals.


Other distortions come from measures to promote wider political agendas, for example decarbonisation, where the subsidies for renewable generation have not been matched by early retirement schemes for other plant, leading to significant overcapacity and depressed prices. Other policies can support continued use of assets that would otherwise be forced out of the market, to protect local employment for example, which leads to further inefficient resource allocation.

Why does it matter?

“The consequences of misallocation extend far beyond simple questions of fairness between groups of investors. It can create structural incentives to invest in a mix of resources ill-suited to the underlying needs of the system, particularly a low-carbon power system. It can obscure the true value of energy storage and flexible demand as supply becomes less controllable. As a result, the business case for innovation can be seriously compromised and consumers can face significantly higher costs for reliability.”

Policymakers aim to deliver a reliable energy system at the lowest (reasonable) cost. They can also have other agendas such as decarbonisation, hence the famous UK trilemma: security of supply, affordability and decarbonisation.

As governments have pursued decarbonisation policies, traditional sources of generation have been increasingly pushed out of the merit order, leading to extensive mothballing programmes as thermal plant cannot compete with subsidised generation with zero marginal cost. Emissions regulations have further led to withdrawal of coal capacity, albeit at a faster rate than anticipated, particularly in the UK as a result of the Carbon Price Floor. As a result, notwithstanding the data above on EU-wide overcapacity, the UK is facing a very tight supply margin over the coming years.

As the markets decarbonise, they are undergoing fundamental changes and creating challenges for system operators to manage around supply intermittency and frequency control. This is creating opportunities for new types of demand-side response, for example remotely controlled charging of electric vehicles, and grid-integrated space and water heating.

“Investors and innovators will need to see a business case for developing these new, more varied strategies for increasing system flexibility. Regulators and consumers will need to see that they are getting value for money. Resources good at providing some kinds of flexibility tend to be not so good at providing others. The dynamism of effective energy pricing represents the most transparent means of signalling what kind of flexibility is needed, revealing what it is actually worth to consumers and bringing forward the most cost-effective responses. Whether or not we pursue a low-cost pathway to a low-carbon power sector will depend on how well we cultivate or replicate that functionality.”

(quotations taken from the RAP paper)
So what can be done?

The RAP paper identifies three approaches to addressing the missing money problem:

  1. The most obvious approach is to remove market rules which distort price formation;
  2. Use administrative measures to inject the missing money into energy and balancing services prices; and
  3. Capacity pricing mechanisms outside the energy and balancing services markets can be adopted.

In removing market distortions, regulators have a number of options including modernising the rules around the procurement and pricing of balancing services, setting gate closure times closer to delivery, or introducing local pricing to reflect actual areas of grid congestion.  

In practice there can be good reasons why some of these measures may not be desirable, in which case regulators can turn to various administrative approaches to adjusting market economics. The choice is between measures that seek to adjust the market prices of energy and balancing services, or whether a separate mechanism outside the markets should be adopted.

In the first case, system operators can adjust market prices up or down to the level at which the system operator determines the cost of meeting demand in future balancing intervals, taking account of forecast demand and available supply over the coming hours and days. By ensuring the price reflects the full cost of meeting demand reliably in any dispatch interval, this mechanism imitates price formation in a fully functional market, and provides incentives for market participants to respond before shortages become acute.

When a system is short of either capacity or flexible capacity, as the UK market currently is, the prices of energy and balancing services become higher and more volatile – yesterday being a great case in point. A market that fully reflects these costs will deliver the investment signals required to deliver new sources of capacity or flexibility, and will stimulate the sort of risk management approaches such as long-term bilateral agreements that will support investment cases.

It is instructive that some of the most progressive energy markets in the world have begun to adopt this approach. ERCOT has been seeing good results with the policy it implemented in 2014, and PJM has also recently adopted a similar approach. In November 2015 the UK introduced a version of this as part of the Electricity Balancing Significant Code Review, in the hope that it will eventually render the troubled Capacity Market redundant.

The final approach described in the RAP paper is the introduction of a capacity market through which providers are guaranteed payments for the provision of capacity, with prices typically being set through auctions. In this case the system operator establishes the price consumers would be willing to pay for capacity over a period of months or years.

Capacity markets tend to favour flexible generation that can ramp up and down quickly thereby avoiding the need to run in low-priced periods outside the shortage events. This was reflected in the UK experience where small OCGTs and diesel generators were very successful in the auctions at the expense of the new CCGTs which had been sought by the government.

The experience of most capacity markets has highlighted the difficulties in designing the rules to deliver the desired outcomes, leading to scheme re-design, often increasing complexity, and political interference.

As capacity markets are designed to deliver capacity based on estimates of demand months or years into the future they suffer from forecasting difficulties and risk over-procurement of resources. They also fail to distinguish between the flexibility offered by different sources of capacity, a facet that is of growing importance with the increasing penetration of intermittent generation in power systems. For this reason, the paper’s author concludes that capacity markets are the least desirable mechanism available to policymakers, doing a relatively poor job at ensuring system reliability at the lowest reasonable cost.

In order to stimulate the required innovation and investments to deliver affordable and secure electricity supplies, policymakers should remove market-distorting practices and ensure that the price of energy and balancing services is fully reflective of the cost for delivering reliable supply in each dispatch period.

Subscribe to the Watt-Logic blog

Enter your email address to subscribe to the Watt-Logic blog and receive email notifications of new posts.