Excited about that new randomized controlled trial result showing microcredit works for men between the ages of 30 and 40.5 who have between 3 and 4 siblings? Or how about yet another study showing a positive effect of aid on growth when you restrict it to countries with good governance and names that begin with the letters A to F? XKCD hits the nail on the head as to why you should approach these results with extreme caution:
2 Comments
Matt (or others), do you know any references off the top of your head to papers that show if/how you can adjust standard errors when dealing with things like this (e.g. when you're estimating over different subsamples)?
I just posted about this cartoon on my blog as well, including a quick reference to Bonferroni (which responds in part to Nicolas's question): rnhttp://tunesforbears.com/2011/04/07/quasi-random-ouch-thoughts-on-statistics/