Some podcasts you should listen to if you're involved in A/B testing

Statistics has been in the news recently which has made for some really thoughtful content being made on the topic. I started compiling a list of people who I thought would enjoy listening to these podcasts on the topic and that list got pretty long so I'll use this blog to broadcast instead. I'll resist giving commentary or critiques on the actual conclusions of the speakers except to say they are interesting.

First was Frakt on Medicaid and the Oregon Medicaid Study on EconTalk which is a great discussion of the statistical power of studies.

Second is Paul Bloom and Joseph Simmons on Bloggingheads.tv which really illustrates how getting fake results from bad statistical practices isn't just a theoretical problem and how you can demonstrate this with simulations.


And finally, back on EconTalk, is Jim Manzi on the Oregon Medicaid Study, Experimental Evidence, and Causality which gets into some more subtle analysis flaws that can destroy the value of A/B testing and really drives home the point that it is a failing endeavour to try to harvest a lot of confidence out of any single experiment. That confidence is gained through an iterative process that comes out of a lot of simple experiments that are constantly updating your priors.

I'll break my no commentary promise a little here. One thing I find quite interesting is how Simmons and Manzi essentially come to the same conclusion on the problem of gaining knowledge from a single experiment while using modern data mining techniques; but they offer different cures. Simmons recommends not allowing yourself to search over your data over lots of dimensions as that will surely lead to false positives. Where as Manzi seems to say you should never be too positive about the results of any single experiment. So iterate over a series of small experiments instead; each one informing the next. Perhaps this is a reflection of their industries (academic vs business) but then this too may be overfit. They both agree that we have to accept that we can't gain truths as quickly as we currently think we can.