Posted by: Peter | July 28, 2009

Fooled by randomness?

In the comments to one of my recent posts,  a commenter was confused about my post on Joe DiMaggio’s hitting streak:

Are you saying that there is a 0.003% chance that a .350 hitter would have this streak on any random 56 game stretch? Or in their career? I’d like to know what the chances that someone would have done it in the history of baseball.

This is a good question, and something I should have been more clear about. It was the former probability I was talking about: the chance of sustaining a hitting streak over any particular stretch of 56 games. But the commenter is right, the second question is the more interesting.

And it’s an answerable question, it just requires figuring out how many chances–how many runs of 56 games–there have been in baseball history. Luckily, before I started to contemplate figuring this out myself, a friend reminded me that someone had actually addressed this question recently (thanks, Andy G!)

Unfortunately, the results of that research kind of undermine the point I was making before. When the mathematician Steven Strogatz ran an extensive series of simulations of the history of baseball, he found that a 56-game streak actually wasn’t so unusual after all. You can see the popular version here, the detailed and geeky version here, and listen to an interesting podcast  discussion of all this from the show Radiolab here.

Strogatz’s conclusion was certainly a little disappointing to me, even though he tried to sugar coat the bad news by noting that even if the existence of a 56-game streak wasn’t that surprising, the existence of such a streak in 1941 was, given the levels of offense that year. But more interesting than that was something else Strogatz pointed out, a little article in the Baseball Research Journal by one Trent McCotter, who did another study of the frequency of hitting streaks.

You can read that article here, and a summary here. It’s an attempt to assess one of the assumptions in all of these hitting streak discussions, namely that a player’s probability of getting a hit in a game does not depend on whether they are on a hitting streak. In other words, there is really no such thing as being “hot” or “cold”, and these apparent streaks are really just arising due to random chance, like a coin coming up heads ten times in a row.

McCotter’s method of testing this assumption is pretty clever.  He observes that if streaks really are totally random occurrences, like the flipped coin, than we should get the same frequency of hitting streaks if we take all the at-bats and rearrange them in a different order. So McCotter takes the at-bats of all the hitters since 1957, and for each one he tries rearranging them into a random order 10,000 times. And what he finds is that in the randomly permuted at-bats, there are fewer streaks than in the actual at-bats. This is strong evidence that at-bats actually aren’t independent, that a player’s performance one day depends partly on how he did the day before.
McCotter pretty much leaves the story there–he doesn’t actually have an explanation for the pattern he’s discovered. But he suggests that at least part of it is that hitters really are doing something different to extend these streaks–as evidenced by the fact that there are an especially large number of streaks at round numbers like 25 and 30. Much more needs to be done to understand all this–but it seems that we may have to unlearn something we thought we’d learned about the non-existence of the “hot hand”.
Hopefully this will be my last post on extremely improbable events for a while. Although as I write this Mark Buehrle is four innings into his second straight perfect game, so…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: