Let’s talk about categories

thought-categories

GJOpen has 15 categories for scored questions:

  1. Business
  2. Economics
  3. Entertainment
  4. Environment
  5. Finance
  6. Geopolitics
  7. Health
  8. International
  9. Open
  10. Politics
  11. Security
  12. Society
  13. Sports
  14. Technology
  15. US

nftags

One question can go into multiple categories.  Here are the categories of all of the questions (note that the Superforecasting book category was never assigned a category; we have assigned it to “US” and “Business”.

ID Question Finished Categories
1 What will the end-of-day closing value for the dollar against the renminbi be on 1 January 2016? Open International, Economics
2 How many seats will the Justice and Development Party (AKP) win in Turkey’s snap elections? Closed International, Politics
3 Will Russia deliver an S-300 or S-400 missile system to Iran before 1 January 2016? Open International, Geopolitics, Security
4 Will there be an initial public offering on either the Shanghai Stock Exchange or the Shenzhen Stock Exchange before 1 January 2016? Open International, Finance
5 Will the Export-Import Bank of the United States be re-authorized before 1 January 2016? Open US, Politics, Economics
6 Will a trilateral meeting take place between Chinese President Xi Jinping, Japanese Prime Minister Shinzo Abe, and South Korean President Park Geun-hye before 1 January 2016? Open International, Geopolitics
7 Will Iran release Jason Rezaian before 31 October 2016? Open International, Geopolitics
8 Will North Korea launch a land based missile with the capacity to reach Alaska, Hawaii, or the continental United States before 1 January 2017? Open International, Geopolitics, Security
9 Will the government of Venezuela or Petroleos de Venezuela S. A. (PDVSA) default on their foreign-currency debt before 1 March 2016? Open International, Economics
10 Will Russia officially lift any part of its food embargo on the United States, any or all of the European Union member states, Canada, Australia, Norway, Iceland, Albania, Montenegro or Liechtenstein before 6 August 2016? Closed International, Geopolitics, Economics
11 Will Congress pass a resolution disapproving the Joint Comprehensive Plan of Action? Closed US, International, Politics, Geopolitics
12 Will OPEC announce any changes to its production quota before 1 January 2016? Open International, Geopolitics, Economics
13 Will the city of Venice lift its ban on “Jean A Deux Mamans” or “Piccolo Uovo” before 1 September 2016? Open International, Society
14 Will Africa remain free of the wild poliovirus through 31 December 2016? Open International, Health
15 Will Bashar al-Assad cease to be President of Syria before 1 March 2017? Open International, Geopolitics, Security
16 Superforecasting Closed  US, Business (patch)
18 Will negotiations on the Transatlantic Trade and Investment Partnership (TTIP) be completed before 1 January 2017? Open International, Geopolitics, Economics
19 Will Iran host a head of state or head of government from one of the G7 countries on an official visit before 1 July 2016? Open International, Geopolitics, Security
20 Who will win the presidential election in Argentina? Closed International, Politics
24 Will the US federal-funds rate be increased before the end of the year [2015]? Open US, Economics
25 Which main character will be the first to die in Season 6 of the Walking Dead? Open US, Entertainment
26 Who will win the World Series? Closed US, Sports
27 Will the Iranian Parliament approve the Joint Comprehensive Plan of Action before 1 November 2015? Closed International, Politics, Geopolitics, Security
28 Will the IAEA conduct an on-site inspection of the Parchin military complex in Iran before the end of the year? Closed International, Security
30 Before 1 January 2017, will it be officially announced that Greece is leaving the eurozone? Open International, Geopolitics, Economics
31 What will be the lowest closing spot price for West Texas Intermediate crude oil, according to the US Energy Information Agency, between 14 September 2015 and 30 December 2015? Open International, Economics, Finance
33 Will both houses of Congress pass legislation lifting or relaxing the oil-export ban before 1 April 2016? Open US, Politics, Economics
34 Will China officially declare an Air-Defense Identification Zone (ADIZ) over any part of the South China Sea before 1 October 2016? Open International, Security
35 Will Jurassic World (2015) pass Titanic (1997) to become the #2 all-time highest-grossing movie domestically by the end of the year? Open US, Entertainment
36 Will the conviction of Adnan Syed, featured on the “Serial” podcast, be overturned before the end of 2016? Open US, Entertainment
37 Will Joe Biden announce that he is running for President of the United States before Thanksgiving? Closed US, Politics
40 Will there be a dissolution or suspension of the Northern Irish Assembly before the end of the year? Open International, Politics
41 Which party will the next Canadian Prime Minister come from? Closed International, Politics
42 Will Russians conduct airstrikes in Syria before 1 May 2016? Closed International, Security
43 Who will Twitter name as their permanent CEO before the end of the year? Closed US, Business
44 Will Roger Goodell be commissioner of the NFL at the start of the 2016 regular season? Open US, Sports
45 As of 31 December 2015, how many refugees and migrants will UNHCR report as having arrived in Europe by sea? Open International
46 What will be the average number of Bitcoin transactions per day in the first week of June 2016? Open International, Finance
47 Will Apple sell more than 75 million iPhones in its December quarter? Open US, Business
49 Will there be a partial or full government shutdown in the United States before the end of the year? Open US, Politics
50 Before the end of the year, will the Bank of Japan announce an increase in asset purchases under its quantitative easing program? Open International, Economics
51 Will any disputes over IAEA access to Iranian sites be referred to the Joint Commission before 1 July 2016? Open International, Geopolitics
54 Will NATO invite Montenegro to join the alliance before the end of the year? Open International, Geopolitics
55 Will Malaysia impose capital controls before 1 October 2016? Open International, Economics
56 Will Winds of Winter be released before Season 6 of Game of Thrones airs? Open US, Entertainment
57 How many countries will Russia remove from its food embargo list before 6 August 2016? Open International, Geopolitics, Economics
61 Will the US or its coalition partners declare a no-fly zone in Syria before the end of the year? Open International, Security
62 Will there be a new King of Saudi Arabia before 1 June 2016? Open International, Politics
63 Will the US Justice Department file criminal charges against Volkswagen or any of its employees before 30 March 2016? Open US, Business, Environment
64 Will President Putin meet with Prime Minister Abe in Japan before 1 July 2016? Open International, Geopolitics
66 Will negotiations on the Regional Comprehensive Economic Partnership (RCEP) be completed before 1 January 2016? Open International, Geopolitics, Economics
70 Will any NATO member invoke Article 4 in response to actions taken by Russia before 1 January 2017? Open International, Security
71 When will Iran next launch a ballistic missile? Open International, Security
74 Will either the Palestinian Authority or Israel announce a halt in security cooperation before 1 July 2016? Open International, Security
76 Will North Korea conduct a nuclear test before 1 January 2017? Open International, Security
77 Before 1 August 2016, will any NATO members invoke Article 4 or Article 5 in response to actions taken by ISIS? Open International, Security
78 Will the expanded Panama Canal open to shipping before 15 May 2016? Open International, Economics
79 Will Vladimir Putin attend the Nuclear Security Summit in Washington, DC, in March 2016? Open International, Geopolitics
80 Before 1 May 2016, will Britain set a date for a referendum on EU membership? Open International, Politics
81 Who will win the Republican Party nomination for the US presidential election? Open US, Politics
82 Will the National Retail Federation report that online and in-store spending for Thanksgiving weekend exceeded $50.9 billion? Closed US, Business
84 Will Turkey impose capital controls before 1 December 2016 ? Open International, Economics
86 Will China conduct naval exercises with any ASEAN member before 1 June 2016? Open International, Geopolitics
87 Who will be the next managing director of the IMF? Open International, Geopolitics
88 Which team will win the NFLs Super Bowl in 2016? Open US, Sports

There are 13 finished questions not of type “Open”, which span the following 8 categories:

  1. Business
  2. Economics
  3. Geopolitics
  4. International
  5. Politics
  6. Security
  7. Sports
  8. US

We can get a good idea of who among our forecasters is good at these 8 categories.  We can’t get a good idea at this point of who is good at these 7 categories. Superforecaster status for these categories is still up for grabs:

  1. Entertainment
  2. Environment
  3. Finance
  4. Health
  5. Society
  6. Technology

Thus the open questions for which we have no or partial (in case the question has multiple categories) information about the quality of the forecaster pool are these 10 questions:

  • 4. China IPO. International, Finance
  • 13. Venice Mommies.  International, Society
  • 14. Africa Polio.  International, Health
  • 25. Zombies. US, Entertainment
  • 31. WTI. International, Economics, Finance
  • 35. Jurassic World. US, Entertainment
  • 36. Serial. US, Entertainment
  • 46. Bitcoin.  International, Finance
  • 56. Winds of Winter. US, Entertainment
  • 63. Volkswagen. US, Business, Environment

Now let’s think about picking forecasters.  All of these questions intersect with categories where we have some winners.  We can just pick out of those pools and give those forecasters higher weight based on accuracy.  Remaining forecasters in the crowd by definition will be assumed to have an accuracy of 0.  When someone is perfectly right and the crowd is perfectly wrong they can have an accuracy of -2.  Let’s give unknown people a weight of 1.  Let’s give superforecasters additional weight up to 21 people’s worth by adding -100 sum(A) units to the weight of the superforecaster, where sum(A) is the sum accuracy over all questions this person has scored in.  I’m summing rather than averaging to give proportionally larger weight to people who participate in more questions successfully.   Sound fair?  I’m pulling that out my drainpipe, but it’s a place to start.   This is the sort of thing that could be calibrated scientifically with different kinds of weighting functions, with more data and more time and patience.

Advertisements

One thought on “Let’s talk about categories

  1. I really appreciate the effort you’ve put into analysis of the project to date. A lot of the methodology is beyond my ken, but nevertheless it’s very interesting

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s