Suppose that we take every forecaster who has a negative (beats the crowd) accuracy score on closed questions related to an open question. Suppose we assign them a weight equal to the sum of their accuracy scores on related questions (call this the net accuracy score), divided by the sum of all accuracy score weights of all included forecasters, so that the sum of weights for all forecasters adds up to 1.
Now suppose we do two things:
- Scatterplot forecast on X axis and individual net accuracy scores, in hopes of seeing some clustering of more accurate people around a particular conclusion.
- For each forecast, sum the net accuracy scores of forecasters making that forecast. This constitutes an empirical probability density function for the open question. We hope that it peaks around a particular forecast.
And the winner is…
Nope and nope. I’m going back to liking the double median filtration idea, which I will present separately.