Retrospective on crowd forecasting in the Sejm question

Recall the GJOpen question “How many seats in Poland’s Sejm will PiS (Law and Justice) win in the upcoming parliamentary elections?”  125 people forecasted this question, making 374 forecasts, or an average of 3 forecasts each.   This question had 3 branches Majority, Plurality and Not a Plurality.  Almost everybody predicted Majority vs Plurality so it was essentially a binary question. The aggregate forecast over time is shown by GJOpen as follows:

sejm

The exact distribution of how many predictions each forecaster made is as follows.  We have over half the forecasters making just 1 prediction, more making 2 or 3, and a few hearty souls with a bunch:

users_per_npreds

It is fun to look at the prediction trajectories for all users at once, imagining each forecaster’s predictions as a kind of Monte Carlo path.  Here’s that picture (I’m fudging all 3-way predictions into a single number where 0 represents “Plurality” and 100 represents “Majority”).  Also I’m assuming that the last prediction is on the closing date of the question.  Also I’m using “prediction time” as the clock, where the first prediction ever made on GJP is at prediction time 1.  There are roughly 130 predictions a day made on GJP, to give you a scale:

prediction_paths

This is a very cool picture which I would like to enter in the Guggenheim.  What it tells you is that, individually, forecasters were all over the map.  As a group, the final consensus just prior to the election was Majority 31%, Plurality 69%.  As it happens, this was not helpful: The election ended in a Majority.  A hopeful reading of the prediction is that in 1 in 3 cases, Majority will occur.  That’s not what most betting people want to hear.

It is not easy to pick the outliers from the above picture, i.e. people who were very right or very wrong.  Since the crowd was biased in favor of Plurality, anybody betting Majority risked looking like a crank.  So what we can say here is that it is hard to single out anybody as being particularly biased in favor of one or the other outcome.  Let’s just say there was a Plurality Camp (in the wrong, finally, but following the conventional wisdom) and a Majority Camp (in the right, but maybe cranky).  What were they thinking?

One way to guess what people were thinking is to look at the words they use in their rationales.  Let’s put people in Majority Camp that bet > 50% Majority, otherwise Plurality Camp.  This gives rise to two word clouds, as follows.  The Plurality cloud:

plurality

and the Majority cloud:

majority

You can’t tell much about these pictures except that the Majority folks were thinking novely about prospects for coalitions and winning, and the plurality people were thinking about the majority.

You may wonder what the distribution of Brier scores was, and whether number of predictions or number of upvotes were predictive of good Brier score.  Here is (approximate, I am computing over prediction time) Brier score as a function of number of predictions.  Looking at this picture, I would guess that number of forecasts is negatively correlated with accuracy, except that if you just make 1 forecast, it looks more like a coin toss:

score_vs_forecasts

Here is Brier score as a function of number of upvotes.  Here is seems that upvotes are somewhat negatively correlated with accuracy.  If you have no upvotes, you are at least within a coin toss of a good score:

score_vs_upvote

Finally, here is the distribution of (prediction-time based) Brier scores for our 125 forecasters.  The mean was 0.85 with a standard deviation of 0.43.  By this criterion, 53% or 66 forecasters beat the crowd, by which, for the sample size, I’m going to interpret as a pretty high number of guessers:

score_density

The really key question in the end, giving the sources referred to, is what was missing from either those sources or a model of Poland which would have predicted the actual outcome better.  We should redact the texts of all majority-favoring and plurality-favoring rationales, and work from there.

Below is a list of the Majority-supporting and Plurality-supporting URLs.  You can do the source analysis and research yourself as to what content was favored by either camp.  The Plurality people hit up Wikipedia 16 times and cited a lot more references.  The Majority people were a bit more laid back.   Other than Wikipedia, Bloomberg, Reuters, NY Times and The Guardian were popular sources:

Majority URLS

  1. Twice: http://news.yahoo.com/polands-eurosceptic-opposition-tops-polls-possible-majority-104935918.html
  2. http://www.bbc.co.uk/news/world-europe-34631826
  3. http://www.bloomberg.com/news/articles/2015-10-23/poland-election-how-sunday-s-vote-could-shake-up-europe
  4. http://www.dailymail.co.uk/wires/reuters/article-3261511/Poland–Factors-Watch-Oct-6.html#ixzz3np2hpIv4
  5. http://www.nasdaq.com/article/polands-ruling-party-suffers-setback-in-referendum-20150907-00178#ixzz3lM02fFL6
  6. 5 times: http://www.politico.eu/article/polands-government-defeated-in-parliamentary-elections-2/
  7. http://www.rte.ie/news/2015/1025/737452-poland-election/
  8. http://www.theguardian.com/world/2015/oct/25/polish-parliamentary-election-vote-2015
  9. http://www.thenews.pl/1/10/Artykul/222758,Russian-Ambassador-summoned-to-Polish-Foreign-Ministry#sthash.P6HqnDkS.dpuf
  10. http://www.tvn24.pl/prawo-i-sprawiedliwosc-wygralo-wybory-parlamentarne,589033,s.html
  11. http://www.usatoday.com/story/news/world/2015/10/25/exit-poll-right-wing-party-wins-polands-parliamentary-vote/74598082/
  12. http://www.wsj.com/articles/poland-s-ruling-party-suffers-setback-in-referendum-1441650085
  13. https://en.wikipedia.org/wiki/Polish_parliamentary_election,_2015#/media/File:Model_sonda%C5%BCy.png
  14. https://en.wikipedia.org/wiki/Sejm#Most_recent_election
  15. https://euobserver.com/political/130821
  16. https://twitter.com/carlbildt/status/656724656532013056/photo/1

Plurality URLS

  1. http://ewybory.eu/sondaz-cbos-24-09-2015/
  2. http://ewybory.eu/sondaz-ibris-dla-onetu-14-10-2015/
  3. http://news.yahoo.com/polands-eurosceptic-opposition-tops-polls-possible-majority-104935918.html
  4. http://news.yahoo.com/polls-predict-solid-conservative-victory-poland-204029744.html
  5. http://static.presspublica.pl/red/rp/img/kraj/FRONTwybory.jpg
  6. http://wiadomosci.onet.pl/wroclaw/byly-wspolpracownik-o-kukizie-pawel-stal-sie-zaprzeczeniem-samego-siebie-z-kampanii/6ns03x.
  7. Twice: http://www.bloomberg.com/news/articles/2015-10-06/polish-opposition-s-sliding-lead-points-to-messy-election-result
  8. http://www.bloomberg.com/news/articles/2015-10-15/poland-s-anti-migrant-party-sets-sight-on-parliamentary-majority
  9. http://www.ecfr.eu/article/commentary_polands_parliamentary_election_the_ultimate_explainer4082
  10. Twice: http://www.electograph.com/2015/10/poland-october-2015-ibris-poll.html
  11. http://www.firstpost.com/world/polands-main-parties-present-election-promises-to-lure-voters-reuters-2431760.html
  12. http://www.janes.com/article/55470/conservative-agrarian-coalition-in-poland-will-be-stable-but-likely-to-lean-towards-state-interventionist-policies
  13. http://www.nasdaq.com/article/polands-ruling-party-suffers-setback-in-referendum-20150907-00178
  14. http://www.nsd.uib.no/european_election_database/country/poland/parliamentary_elections.html;
  15. http://www.nytimes.com/2015/09/13/world/europe/eastern-europe-migrant-refugee-crisis.html?_r=0
  16. http://www.reuters.com/article/2015/09/22/us-poland-climatechange-idUSKCN0RM22820150922
  17. http://www.reuters.com/article/2015/10/01/poland-coal-idUSL5N11V39X20151001
  18. http://www.reuters.com/article/2015/10/02/poland-factors-idUSL5N1213PN20151002
  19. http://www.reuters.com/article/2015/10/16/us-poland-election-absentees-idUSKCN0SA17C20151016
  20. http://www.reuters.com/article/2015/10/22/us-poland-election-coalition-idUSKCN0SG24Z20151022
  21. http://www.sejm.gov.pl/english/sejm/pos.htm
  22. http://www.spiegel.de/international/europe/beata-szydlo-revamps-polish-law-and-justice-party-ahead-of-elections-a-1058506.html
  23. Twice: http://www.theguardian.com/world/2015/oct/22/polish-elections-2015-a-guide-to-the-parties-polls-and-electoral-system
  24. http://www.thenews.pl/1/9/Artykul/222111,Polish-parliament-has-no-time-to-take-up-president%E2%80%99s-draft-retirement-bill
  25. http://www.thenews.pl/1/9/Artykul/222453,Gap-closes-in-Polands-latest-electoral-opinion-poll
  26. http://www.thenews.pl/1/9/Artykul/224149,Poll-PiS-well-ahead-of-PO-before-general-election
  27. http://www.tvp.info/21608768/ipsos-dla-tvp-info-pis-moglby-rzadzic-samodzielnie-w-sejmie-cztery-ugrupowania
  28. http://www.tvp.info/21772754/zwyciestwo-pis-i-piec-partii-w-sejmie-najnowszy-sondaz-cbos
  29. http://www.tvp.info/21783027/sondaz-dla-wiadomosci-tvp1-pis-na-czele-i-samodzielnie-rzadzi-lewica-poza-sejmem
  30. http://wybory2007.pkw.gov.pl/SJM/EN/WYN/M/1.htm
  31. https://en.wikipedia.org/wiki/D’Hondt_method
  32. 16 times: https://en.wikipedia.org/wiki/Polish_parliamentary_election,_2015
  33. Twice: https://polishpoliticsblog.wordpress.com/
  34. https://polishpoliticsblog.wordpress.com/2015/09/11/who-won-polands-referendum-war-and-how-will-it-affect-the-october-election/

 

Advertisements

2 thoughts on “Retrospective on crowd forecasting in the Sejm question

  1. There is an isomorphism to the Turkish election other than the poor forecasting result.

    If it’s hard to be good at forecasting then it must be equally hard to be bad at forecasting. You can always invert bad to get to good.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s