To do: Exponentially decay old forecasts when assessing forecaster accuracy

Forecasting tournament probabilities are overly stable compared to prediction market probabilities due to stale predictions.  [1] This is a problem.  The solution is to exponentially decay old forecasts in favor of more recent ones, so that old and singleton forecasts fade over time.

How this would work when assessing forecasters is to weight all predictions of forecasters reverse exponentially, so that a forecaster who forecasts infrequently will see influence decline over time in assessing the crowd prediction at each time point and in assessing a subset of the crowd over time.

The sum prediction would be divided by the sum of weights at each time point, so that you wouldn’t get an artifact like a market with 1 forecaster seeing 1 forecast decay over time when there are no other forecasts.

[1] https://www.gjopen.com/comments/comments/62078

Advertisements

One thought on “To do: Exponentially decay old forecasts when assessing forecaster accuracy

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s