# Retrospective on crowd forecasting in the Sejm question

Recall the GJOpen question “How many seats in Poland’s Sejm will PiS (Law and Justice) win in the upcoming parliamentary elections?”  125 people forecasted this question, making 374 forecasts, or an average of 3 forecasts each.   This question had 3 branches Majority, Plurality and Not a Plurality.  Almost everybody predicted Majority vs Plurality so it was essentially a binary question. The aggregate forecast over time is shown by GJOpen as follows:

The exact distribution of how many predictions each forecaster made is as follows.  We have over half the forecasters making just 1 prediction, more making 2 or 3, and a few hearty souls with a bunch:

It is fun to look at the prediction trajectories for all users at once, imagining each forecaster’s predictions as a kind of Monte Carlo path.  Here’s that picture (I’m fudging all 3-way predictions into a single number where 0 represents “Plurality” and 100 represents “Majority”).  Also I’m assuming that the last prediction is on the closing date of the question.  Also I’m using “prediction time” as the clock, where the first prediction ever made on GJP is at prediction time 1.  There are roughly 130 predictions a day made on GJP, to give you a scale:

This is a very cool picture which I would like to enter in the Guggenheim.  What it tells you is that, individually, forecasters were all over the map.  As a group, the final consensus just prior to the election was Majority 31%, Plurality 69%.  As it happens, this was not helpful: The election ended in a Majority.  A hopeful reading of the prediction is that in 1 in 3 cases, Majority will occur.  That’s not what most betting people want to hear.

It is not easy to pick the outliers from the above picture, i.e. people who were very right or very wrong.  Since the crowd was biased in favor of Plurality, anybody betting Majority risked looking like a crank.  So what we can say here is that it is hard to single out anybody as being particularly biased in favor of one or the other outcome.  Let’s just say there was a Plurality Camp (in the wrong, finally, but following the conventional wisdom) and a Majority Camp (in the right, but maybe cranky).  What were they thinking?

One way to guess what people were thinking is to look at the words they use in their rationales.  Let’s put people in Majority Camp that bet > 50% Majority, otherwise Plurality Camp.  This gives rise to two word clouds, as follows.  The Plurality cloud:

and the Majority cloud:

You can’t tell much about these pictures except that the Majority folks were thinking novely about prospects for coalitions and winning, and the plurality people were thinking about the majority.

You may wonder what the distribution of Brier scores was, and whether number of predictions or number of upvotes were predictive of good Brier score.  Here is (approximate, I am computing over prediction time) Brier score as a function of number of predictions.  Looking at this picture, I would guess that number of forecasts is negatively correlated with accuracy, except that if you just make 1 forecast, it looks more like a coin toss:

Here is Brier score as a function of number of upvotes.  Here is seems that upvotes are somewhat negatively correlated with accuracy.  If you have no upvotes, you are at least within a coin toss of a good score:

Finally, here is the distribution of (prediction-time based) Brier scores for our 125 forecasters.  The mean was 0.85 with a standard deviation of 0.43.  By this criterion, 53% or 66 forecasters beat the crowd, by which, for the sample size, I’m going to interpret as a pretty high number of guessers:

The really key question in the end, giving the sources referred to, is what was missing from either those sources or a model of Poland which would have predicted the actual outcome better.  We should redact the texts of all majority-favoring and plurality-favoring rationales, and work from there.

Below is a list of the Majority-supporting and Plurality-supporting URLs.  You can do the source analysis and research yourself as to what content was favored by either camp.  The Plurality people hit up Wikipedia 16 times and cited a lot more references.  The Majority people were a bit more laid back.   Other than Wikipedia, Bloomberg, Reuters, NY Times and The Guardian were popular sources:

Majority URLS

Plurality URLS