# Science or Fiction Prediction: Getting a Statistical Edge

People sometimes say that if you're not sure what the answer is on a multiple choice question, you should guess c.  I've always wondered if such a system could be applied to the Science or Fiction section on The Skeptic's guide to the Universe (SGU) podcast.

What a multiple choice test may look like

Quick background for non-listeners:  The Skeptics Guide to the Universe is a super great science podcast that you should listen to.  Each episode, they play a game called Science or Fiction, where one host (usually Steve) reads science news items or facts, one of which is completely made up.  The others then try their best to determine which one is the fiction.

While it isn't practical to examine all of the multiple choice tests that have ever existed to determine if c is more likely to be correct, we can actually take a look at each round of Science or Fiction.  It turns out that they keep good show notes on the SGU's website, including each science or fiction item and whether or not it's true.

As of this post, there are 480 episodes, so it's not practical to get the data by hand, but since each episode's page is neatly organized on the website it only took a couple minutes to whip up a little scraping script with python and Beautiful Soup to get the data. (Interestingly enough, scraping through all of the pages I found a tiny mistake: Item #1 of episode 247 is missing a "1".  This broke my scraper the first time through.)

I only collected information about episodes where there were three science or fiction items (which is most of them), so that we can make a meaningful comparison:

Item 1 Item 2 Item 3
Frequency 128 119 133
Probability of Fiction 33.7% 31.3% 35.0%

So it appears that item 2 is fiction less often than items 1 and 3.  The question is, is it a "real" difference, or is it just part of the expected statistical background noise?  Basically, we're trying to empirically determine if Steve uses some sort of random number generator to determine which item will be the fiction each week. Doing a chi squared test tells us that there's a 67% chance of observing such a difference.

In other words: the frequencies are consistent with a uniform distribution, and you can't get a significant edge based on the item ordering.  Steve outsmarts us again!

I did the data collection and analysis with ipython, and you can check out the code here.

# More Fun with OCN Server Data

This is a follow up to the previous post about tracking Overcast Network's (OCN) server activity.

A couple things:

1. At the time of that posting, there were only a few days of data in the database.  Since then, the script has been churning away for the past few weeks, giving us a much larger sample.
2. The original scripts spend most of the time juggling dictionaries and reshaping the data to plot.  This isn't particularly elegant.  This time around I'm using Pandas for the data preprocessing after restructuring the database.

In retrospect, it would have probably made more sense to store the information in an SQL database. I used MongoDB only because I had never used it before (my favorite reason), and the prospect of being able to dump python dictionaries right in seemed fun.  And I pretty much did just that - dumping dictionaries of data - which seemed simple enough at the time but ultimately led to processing complications later (see above).

With all of this in mind, I played with the data a bit in an ipython notebook, and so it only makes sense to display the code and results using the very cool browser notebook viewer.  Check them out here!  (If you aren't using the ipython notebook daily, you're blowing it. It's a lot of fun.)

As you can see from the plots, the player count varies quite a bit throughout the day, even with a very large spread of players across the globe.  This can cause some issues since many of the servers are designed with a certain number of players in mind.  OCN recently implemented dynamic servers, which turn on and off depending on the number and distribution of players online and will hopefully solve this issue.

Code and more graphs