The Folly of the "Eye Test"

I had never really heard of sabermetrics until I started following this blog. A lot of the latest and greatest in statistical analysis of players - FIP, ISO, WAR, wOBA - were things I learned while I was trawling through the posts and comments here. Overall the community here seems to be pretty "with it" when it comes to the newer trends in understanding the game of baseball, which is cool. It's been a great learning experience.

Lately, though, I've noticed an awful lot of comments along the following lines:

"I don't need any statistics to judge a player, I can rely on my own eyes."

"I know that's what the numbers say, but they don't match up with what I've seen, so clearly the numbers are wrong."

"I don't understand advanced statistics, so I'm going to consider them worthless."

...OK, for that last one I read between the lines a little, but the rest are paraphrases of comments I've seen on this site quite frequently as of late.

The Good, The Bad, and The Optometrist: Why The Eye Test Isn't Always Reliable

There's a very interesting blog out in the wilds of the internet called You Are Not So Smart. I strongly recommend it to anyone who's interested in this kind of thing; the blog doesn't update often, but when it does the posts are always well-researched treatises on why and how we trick ourselves. A few of those posts focus exclusively on sight, and I'm going to borrow one of their examples to open this post:



The above animated gif is actually switching between two different pictures. If you're anything like me, you're going to sit there and watch it for... well, a long time before you finally either give up or actually notice the difference. Once you see it, you see it every time - but until you finally spot the change, it's going to drive you crazy. In fact, you'll probably even stare right at what's changing and not realize it. This is an example of what's called "Change Blindness:" our brain tricks us into not noticing the difference, and it's one of many reasons that we shouldn't trust what we see. There are lots of these little "tricks" we play on ourselves, and they all lead to situations where the eye test isn't as reliable as we'd like to think it is.

"But duwanis," you yell, exasperated after you finally notice the difference above, "that was such a small change! I'd have noticed if it had been something bigger!" Well, not so much. Consider the following experiment:

Experimental Psychology - Change Blindness (via Nonconceptualcontent)

"But they weren't trying to pay attention! I am!" OK. Let's try something a little more sports related. See if you can come up with the correct number of basketball passes from this video:

Test Your Awareness: Do The Test (via dothetest)

Now, you may have done well on any one of these tests, or even more than one. This is a highly studied area of psychology, though, and the fact that one (or a few) tests don't work for you doesn't mean you're immune to these effects; it just means that the circumstances were right. You can be - and are - fooled by this sort of thing on a daily basis. And this is just one of the things that plays against our perceptions: just from that one blog, there's hindsight biasconfirmation biasthe "Texas Sharpshooter" fallacy, and the backfire effect - all ways in which our brains conspire against us to convince us of things that aren't true.

Statistics: Our Last Line of Defense Against Ourselves

OK, that's probably being a bit melodramatic, but it sounds good. Sue me.

You've often heard it said that "numbers don't lie," I'm sure, and there's truth to that. In most cases, what we perceive can be measured to be confirmed or denied. Think Alex Gonzalez walks a lot? Quick, check his stats. Think Uggla produces more runs for the Braves than Chipper? It's a bit more complicated, but you can crunch the numbers. Think Heyward has a hole in his swing against Lefties? Check the pitch data. It's all there.

Of course, with "numbers don't lie" comes another oft-quoted aphorism: "There's lies, damned lies, and statistics." This catchy little quip is easy to misunderstand, though - the problem isn't in statistics, it's in how people use them.

As an example, I could suggest that J.C. Boscan is an amazing major league hitter. After all, he had an OBP of 1.000 in 2010! Other guys we consider to be legendary hitters are lucky to get to half that! Quick, sign him to an extension!

But you are all familiar with this trick. I used an incredibly small sample size (not only did I limit it to one year, but I limited it to a year in which he only had one plate appearance). I used unreliable numbers to back up my argument. Just because it's a number doesn't make it a useful number. Statistics are only better than the eye test if they provide reasonable data. Otherwise, they can be just as misleading.

Ne quid nimis

The truth of the matter is that there isn't really folly in statistics (no matter how advanced) or in the "eye test." Both are useful tools. The thing you have to be aware of is knowing what the pitfalls are - it's too easy to trust in numbers without considering the circumstances they came out of, and it's too easy to trust in what you see or remember without trying to confirm that recollection objectively. If you reach a point where the two are in disagreement, consider the possible reasons. Are you tricking yourself into believing something unrealistic? Or is there a reason the numbers might be misleading? If you lean too much one way or the other, you're likely to miss something.

This FanPost does not express the views or opinions of Talking Chop.

Log In Sign Up

Log In Sign Up

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

By becoming a registered user, you are also agreeing to our Terms and confirming that you have read our Privacy Policy.

Join Talking Chop

You must be a member of Talking Chop to participate.

We have our own Community Guidelines at Talking Chop. You should read them.

Join Talking Chop

You must be a member of Talking Chop to participate.

We have our own Community Guidelines at Talking Chop. You should read them.




Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.