Some numbers on the GJ Open population

GJ Open tells us, on each user’s profile, the following information:

  1. In what order they signed up (the integer user ID in the URL)
  2. How many questions answered
  3. How many predictions made
  4. How many upvotes
  5. The overall Brier score
  6. The median Brier score for the equally-participating crowd
  7. The accuracy score (equals overall – median)

There are now about 9828 forecasters.  Of these, 6882 have never put in a forecast, and 2946 have answered at least one question.  So 30% of the population is active and 70% is inactive.

Questions answered.  61 questions have been posed (of which 3 are question-questions, i.e. will never be scored).   The following chart shows how many users have answered how many questions.  I am one of two users who have answered all questions.  The other forecaster who has answered all questions is ravel.  He has managed to do so while retaining a decent accuracy score; I have not.  Bravo, ravel!

0: 6882, 1: 1067, 2: 551, 3: 322, 4: 191, 5: 167, 6: 103, 7: 77, 8: 57, 9: 48, 10: 43, 11: 39, 12: 31, 13: 20, 14: 22, 15: 13, 16: 12, 17: 15, 18: 14, 19: 12, 20: 13, 21: 13, 22: 10, 23: 11, 24: 4, 25: 9, 26: 7, 27: 7, 28: 5, 29: 3, 30: 4, 31: 3, 32: 3, 33: 2, 34: 1, 35: 3, 36: 1, 37: 3, 38: 4, 39: 1, 40: 1, 41: 4, 42: 3, 43: 3, 44: 5, 45: 0, 46: 1, 47: 2, 48: 0, 49: 1, 50: 1, 51: 1, 52: 1, 53: 1, 54: 0, 55: 2, 56: 0, 57: 0, 58: 3, 59: 2, 60: 2, 61: 2

As a picture:
questions_answered

This is kind of a boring picture, other than telling us that the number of people who want to work for free follows an inverse exponential law. So let’s limit ourselves to the population which has these qualities, which we will call the accurate busy forecasters (ABFs).   I will not use the term Superforecasters™ which has been taken, and also in my mind connotes people who pick and choose their questions and pay special attention to timing and scoring methodology, rather than simply applying Hobson’s Choice to pick questions. So, people who are:

  1. Willing to do a lot of work for free.  So, people who have answered 20 or more questions.
  2. Good at their pretend job.  So, people who beat the crowd (have a negative accuracy score).

Of our population of 9828 people, there are 58 ABFs, the leader among these being user Rote.  So 0.5% of the total population.  Here is the list, based on user ID (what you see in the URL of the profile of a user), along with accuracy score:

-0.368:783,-0.366:1448,-0.363:29,-0.328:4042,-0.282:241, -0.280:1682,-0.259:68,-0.259:83,-0.228:110,-0.222:1591,-0.219:3828, -0.203:212,-0.191:465,-0.184:136,-0.184:657,-0.183:36,-0.177:231, -0.173:43,-0.171:568,-0.165:1337,-0.164:24,-0.162:586,-0.157:2, -0.148:60,-0.145:38,-0.141:1575,-0.117:271,-0.112:1529,-0.106:20, -0.105:961,-0.104:3617,-0.098:1646,-0.085:1296,-0.078:182, -0.076:493,-0.073:42,-0.073:1078,-0.073:1114,-0.069:1300, -0.067:584,-0.067:689,-0.063:506,-0.051:131,-0.051:306, -0.047:40,-0.032:74,-0.027:73,-0.019:532,-0.018:53, -0.015:57,-0.014:3328,-0.014:740,-0.009:1304,-0.008:1278, -0.008:1280,-0.007:680,-0.005:1500,-0.003:1127

The big takeaway from this is that the order of entry is completely uncorrelated with accuracy score.  Superforecasters and people with prior GJP experience were let into the club first.  Then came the randoms.  Randoms 783, 1448 and 4042 are the lead ABFs.  Moral of the story:  Natural ability seems to trump prior experience at this game or special Tetlock training.  Correct me if I’m wrong!

To get a better sense of correlation, here is a scatterplot of order of entry versus accuracy score for the ABFs:

abo_corr_entry_accuracy

Predictions made.  The maximum number of forecasts was 1728  made by cdob63.  Here are the number of users per forecast.  I have made 368 forecasts which puts me in the top 6:

0: 6883, 1: 960, 2: 554, 3: 309, 4: 211, 5: 145, 6: 114, 7: 88, 8: 78, 9: 51, 10: 38, 11: 36, 12: 32, 13: 22, 14: 22, 15: 26, 16: 12, 17: 11, 18: 15, 19: 16, 20: 9, 21: 20, 22: 13, 23: 8, 24: 7, 25: 5, 26: 5, 27: 8, 28: 2, 29: 2, 30: 6, 31: 7, 32: 4, 33: 5, 34: 2, 35: 4, 36: 2, 37: 4, 38: 4, 39: 4, 40: 1, 41: 3, 42: 3, 43: 2, 44: 1, 45: 4, 46: 1, 47: 3, 48: 2, 50: 2, 54: 1, 55: 1, 57: 1, 59: 3, 62: 1, 63: 1, 65: 1, 68: 1, 69: 3, 70: 1, 71: 3, 72: 3, 75: 1, 77: 1, 78: 2, 79: 1, 80: 1, 81: 1, 82: 1, 86: 2, 90: 1, 93: 1, 95: 1, 98: 1, 102: 2, 104: 1, 110: 1, 111: 1, 113: 1, 117: 1, 144: 1, 147: 1, 150: 1, 151: 1, 153: 1, 161: 1, 175: 1, 184: 1, 208: 1, 255: 1, 278: 1, 291: 1, 324: 1, 328: 1, 329: 1, 368: 1, 403: 1, 405: 1, 474: 1, 865: 1, 1728: 1

We may wonder how forecasts correlate with accuracy. Here is a scatterplot of # forecasts versus accuracy for the ABFs.  Those with fewer predictions seem to have less accuracy, but the correlation is not obvious at all:

abo_corr_forecasts_accuracy

Upvotes.  The most upvotes received is 758 by Anneinak and the least upvotes is -19 for user 5032.  However note I’ve checked that user’s comments and none of them show negative upvotes, so this is probably a bug.  There are some people with authentic negative upvotes, but it is hard to tell given the likely presence of bugs, so we’ll leave that distinction aside.  In terms of #users with a given upvote count, here are the tallies.  I have 366 upvotes (365 now, somebody bumped me down one), putting me in the top 5 for upvotes:

-19: 1, -14: 2, -13: 2, -12: 2, -11: 1, -10: 1, -9: 4, -8: 3, -7: 11, -6: 11, -5: 11, -4: 47, -3: 64, -2: 139, -1: 250, 0: 7844, 1: 534, 2: 292, 3: 162, 4: 108, 5: 69, 6: 35, 7: 42, 8: 21, 9: 19, 10: 11, 11: 12, 12: 10, 13: 8, 14: 11, 15: 3, 16: 2, 17: 2, 18: 2, 19: 3, 20: 3, 21: 9, 22: 2, 23: 2, 24: 3, 25: 2, 26: 1, 27: 1, 28: 3, 29: 5, 31: 2, 33: 1, 34: 1, 35: 3, 36: 1, 37: 3, 38: 1, 39: 2, 40: 1, 41: 1, 42: 1, 44: 1, 45: 2, 47: 1, 54: 1, 55: 1, 56: 1, 58: 1, 61: 1, 65: 1, 66: 1, 71: 1, 75: 1, 79: 1, 80: 1, 83: 1, 86: 1, 90: 1, 94: 1, 101: 1, 103: 1, 104: 1, 108: 1, 120: 1, 131: 1, 135: 1, 152: 1, 157: 1, 175: 1, 188: 1, 201: 1, 213: 1, 231: 2, 260: 1, 354: 1, 366: 1, 388: 1, 404: 1, 460: 1, 758: 1

For the ABFs let’s make a picture of #upvotes versus accuracy.  This is another inconclusive picture but I’d be tempted to say that upvotes are somewhat negatively correlated with accuracy:

abo_corr_upvotes_accuracy

Moral of the story: The ABFs are the half of 1 percenters.  They get a lot of upvotes but that is no indication of relative standing.  They do a lot of forecasts but the absolute number are not clearly correlated with accuracy.  By definition my ABFs answered at least 20 questions out of 61 available.  If I looked at the population of forecasters answering all questions, I’d be dealing with a sample of 2, so I’ll stick with 20 questions.  You gotta play 20 questions to get on my list.

Advertisements

6 thoughts on “Some numbers on the GJ Open population

  1. downvotes could be on replies too, which wouldn’t show up under forecasts.

    Thanks for posting all of this info. I’ve been curious about some of these things.

    Like

  2. I’ve been visiting your blog most days – it’s become a “must read” for me, along with @morrell – but somehow missed this entry. I’m a low tech sort of person who doesn’t look “under the hood” for hardware or software but would like to know how did you access the information that you have? I haven’t been able to find any listing on the site. At times I’ve laboriously checked some individuals’ pages, but that still didn’t give me their “accuracy” score, only the difference between their score and the median, with no indication of the numbers of questions scored or the duration of forecast on them.

    Is the “accuracy score” given by you the same as the GJOpen system i.e. difference between one’s own score and median for each question, multiplied by the “participation rate”, and then summed up? Again, if so how does one access this data?

    Like

  3. @Khalid, if I click on my profile I get a URL in the browser address bar of https://www.gjopen.com/memberships/102/scores. Note the 102. That is my user ID. This page gives me my overall Brier score against the overall median. What I am calling Accuracy is Brier – Median. To get the Accuracy of say forecaster 103 (the next one to register after me), just paste in https://www.gjopen.com/memberships/103/scores. To get the entire population, just keep incrementing the user ID from 1 to 12,000. There will be some missing IDs. The last available ID is currently between 11,000 and 12,000.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s