Measuring Reliability Using Conditional Exceedance Probabilities

Simon Mason

IRI
Columnia University

Given the inherent uncertainty in forecasts of the atmosphere, the degree of confidence in any forecast can be of considerable interest. However, it is important to indicate whether there is any useful information in the variability in the forecast confidence. In the context of probabilistic forecasts of variables measured on a categorical or ordinal scale, such as precipitation occurrence, or temperatures exceeding a threshold, forecasts typically are verified by comparing the relative frequency with which the target event occurs given different levels of forecast confidence. A procedure for assessing the reliability of forecast ensembles of a continuous variable is proposed. It is an extension of the binned probability histogram. Individual ensemble members are treated as estimates of percentiles of the forecast distribution, and the conditional probability that the forecast precipitation, for example, exceeds the amount observed (the conditional exceedance probability) is calculated. A diagram showing the conditional exceedance probabilities for ranked ensemble members is suggested as a useful method for indicating reliability when forecasts are on a continuous scale, and various statistical tests are suggested for quantifying the reliability.

Last Update: April 24, 2005