To do: Measures of foresight

For purposes of crowd aggregation I think GJP exponentially tails old observations.  That is pragmatic for prediction of open questions.

I want to do hindsight is to measure foresight.   Brier accuracy, not foresight per se, because it is not time-weighted.

So over the lifetime of the question (which will I will assign to be natural lifetime i.e. from now to when it happened, not the question period which is arbitrarily long), I will retrospectively discount with continuous compounding so that the final observation has a low weight, say 50%, versus initial observation weighted 1.  So find some subset of questions that many people have answered.  Measure their flat Brier scores and order them in terms of sum accuracy over multiple questions (accuracy = Brier – Crowd Brier).  Then do retrospective Brier weighted by various windows such as linear, quadratic and exponential and inverse exponential and inverse quadratic.  See which windowing scheme leads to an ordering on individuals which is closest to the flat sum of accuracy ordering.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s