Good Judgment Open

The Good Judgment Project (GJP) is an organization dedicated to "harnessing the wisdom of the crowd to forecast world events". It was co-created by Philip E. Tetlock (author of ''Superforecasting'' and ''Expert Political Judgment''), decision scientist Barbara Mellers, and Don Moore, all professors at the University of Pennsylvania.

The project began as a participant in the Aggregative Contingent Estimation (ACE) program of the Intelligence Advanced Research Projects Activity (IARPA). It then extended its crowd wisdom to commercial activities, recruiting forecasters and aggregating the predictions of the most historically accurate among them to forecast future events. Predictions are scored using Brier scores. The top forecasters in GJP are "reportedly 30% better than intelligence officers with access to actual classified information."

History

The Good Judgment Project began in July 2011 in collaboration with the Aggregative Contingent Estimation (ACE) Program at IARPA (IARPA-ACE). The first contest began in September 2011. GJP was one of many entrants in the IARPA-ACE tournament, which posed around 100 to 150 questions each year on geopolitical events. The GJP research team gathered a large number of talented amateurs (rather than geopolitical subject matter experts), gave them basic tutorials on forecasting best practice and overcoming cognitive biases, and created an aggregation algorithm to combine the individual predictions of the forecasters. GJP won both seasons of the contest, and were 35% to 72% more accurate than any other research team. Starting with the summer of 2013, GJP were the only research team IARPA-ACE was stil funding, and GJP participants had access to the Integrated Conflict Early Warning System.

People

The co-leaders of the GJP include Philip Tetlock, Barbara Mellers and Don Moore. The website lists a total of about 30 team members, including the co-leaders as well as David Budescu, Lyle Ungar, Jonathan Baron, and prediction-markets entrepreneur Emile Servan-Schreiber. The advisory board included Daniel Kahneman, Robert Jervis, J. Scott Armstrong, Michael Mauboussin, Carl Spetzler and Justin Wolfers. The study employed several thousand people as volunteer forecasters. Using personality-trait tests, training methods and strategies the researchers at GJP were able to select forecasting participants with less cognitive bias than the average person; as the forecasting contest continued the researchers were able to further down select these individuals in groups of so-called ''superforecasters''. The last season of the GJP enlisted a total of 260 superforecasters.

Research

A significant amount of research has been conducted based on the Good Judgment Project by the people involved with it. The results show that harnessing a blend of statistics, psychology, training and various levels of interaction between individual forecasters, consistently produced the best forecast for several years in a row.

Good Judgment Inc.

A commercial spin-off of the Good Judgment Project started to operate on the web in July 2015 under the name Good Judgment Inc. Their services include forecasts on questions of general interest, custom forecasts, and training in Good Judgment's forecasting techniques. Starting in September 2015, Good Judgment Inc has been running a public forecasting tournament at the Good Judgment Open site. Like the Good Judgment Project, Good Judgment Open has questions about geopolitical and financial events, although it also has questions about US politics, entertainment, and sports.

Media coverage

GJP has repeatedly been discussed in ''The Economist''. GJP has also been covered in ''The New York Times'', ''The Washington Post'', and Co.Exist. NPR aired a segment on The Good Judgment Project by the title "So You Think You're Smarter Than a CIA Agent", on April 2, 2014. The ''Financial Times'' published an article on the GJP on September 5, 2014. '' Washingtonian'' published an article that mentioned the GJP on January 8, 2015. The BBC
...