A novel statistical idea: 'Down-Up' sequences that 'capture' small tail probabilities

Reliable estimation of tail probabilities is important in fields from finance, to geophysics, to meteorology, to the design of ships, and to optics.

A new paper published in Mathematics and Statistics presents the novel statistical idea of “Down-Up” sequences which “capture” small tail probabilities with surprising precision without knowing the underlying probability distributions. In Estimation of Small Tail Probabilities by Repeated Fusion, ISR-affiliated Professor Benjamin Kedem (Math) and his colleagues Lemeng Pan, Paul J. Smith and Chen Wang describe the idea, its implementation, and its usefulness in the estimation of small tail probabilities using limited amounts of data.

They show how to estimate any threshold probability from data below or even far below the threshold through repeated fusion of the data with externally generated random samples. This is referred to as repeated out of sample fusion (ROSF). A comparison of the approach with peaks-over-threshold (POT) across different tail types shows that ROSF provides more precise point and interval estimates based on moderately large samples.

The ideas presented in this paper can be extended in a number of ways, such as using “fake” data from distributions other than uniform, and using different fusion mechanisms together with appropriate inferential methods other than the semiparametric method used in the paper. Different ways of connecting X0 and the fusion samples can be explored, other than by means of their distributions as expressed by the density ratio model.

Published February 11, 2020