This post, as you can probably tell by the title, is a companion piece to the projection review of position players completed earlier this week. As it turns out, based on the data I examined, there’s somewhat less to compile for members of the 2016 Atlanta Braves pitching staff, for a few reasons. First, Steamer, ZiPS, and myself varied the specific endpoints estimated. Steamer was the most comprehensive, with ERA, FIP, fWAR, and rWAR*, ZiPS did three of those but not rWAR, and in my narrowly-focused worldview, I chose only to focus on FIP and fWAR. Second, projections for relievers are a fairly foolhardy task, and I did not collect those last year, though I may do so this coming season. The corps of relievers varies a fair bit from week to week each season, and given the small number of outings a reliever can make, as well as how a good or bad outing can skew a reliever’s overall results, it didn’t feel like a great use of time to put forth projections for those. But, if you are interested in seeing them this season, I can consider producing them - let me know in the comments which relievers you’re particularly interested in forecasting.
* Note: rWAR refers essentially to “Baseball Reference WAR,” which is a way of assessing pitcher value based on actual runs allowed, after accounting for innings pitched, defense, park, etc. It assumes that the pitcher, with some adjustments, is responsible for every run he allows. On the other hand, fWAR assumes that pitchers are only responsible for the strikeouts, walks, home runs, and infield pops they cause/allow, and is a more isolating measure of pitcher production.
Once again, the table below provides an overview of results. The previous projection writeups focused on 11 potential starters for the 2016 Braves (ZiPS did not cover Jhoulys Chacin, however). These 11 pitchers ended up constituting about 85 percent of the total starting pitching innings for the team over the course of the season; the remainder was largely covered by Joel de la Cruz, Roberto Hernandez, and Lucas Harrell.
The categories are hopefully self-explanatory. ERA and FIP are on their straight-up, “runs per nine” measure, while the WAR values are pro-rated to 200 innings pitched, though the right-hand table includes the actual innings pitched if you’d like to do some mental math about actual production.
The following is a quick rundown of the player-by-player projected-versus-actual results for FIP and fWAR.
FIP
The issue with comparing projected and actual FIPs, of course, is that FIP is not a counting stat, and therefore ignores the number of innings a given pitcher completed. Therefore, errors for pitchers with relatively few innings are weighted equally as errors for pitchers that played for most of the season, which skews the results a bit. With that said, it’s still interesting to array all the pitchers together in a way that’s distinct from relying on their total innings pitched.
The pitching for the 2016 Braves, was, in two words, not good. But, the relationship between the projections and performance is fairly interesting. The most glaring items in the above chart are Aaron Blair and Tyrell Jenkins, each of whom struggled well beyond what any projection system thought, based on their minor league numbers and other characteristics. To a smaller extent, Williams Perez, Ryan Weber, and Matt Wisler also fell short of expectations. On the flip side, Jhoulys Chacin, Mike Foltynewicz, Casey Kelly, and Julio Teheran all came in with lower FIPs than expected. Teheran, was, of course, the real jewel of both the 2016 Atlanta rotation and this exercise: his 3.69 FIP clambered under the 3.87 projected by ZiPS and the higher values (3.94, 4.06) projected by IWAG and Steamer, respectively.
John Gant, Casey Kelly, Bud Norris, and Williams Perez are all fairly interesting cases as well. Each of them ran ERAs below replacement level, but livable FIPs (after adjusting for park effects/etc.). Kelly was fairly surprising in his 22-inning sample, but that’s mostly just because he didn’t allow a homer: his 6.28 xFIP from last season is gnarly.
Another thing that I found interesting is that my own calculations in IWAG tended to diverge more for pitchers than for hitters. If you look at the summary table of the hitter post, you’ll see that generally IWAG agreed with either Steamer or ZiPS, if not both. There were a few places where the deviation was a little bigger (i.e., Kelly Johnson, Nick Markakis, Mallex Smith), but even there, it was mostly a matter of degrees and the general expectations were not far off. From the table above, though, you can see this wasn’t quite the case for pitchers:
- Based on his minor league numbers, IWAG expected Aaron Blair to be far worse than Steamer/ZiPS, who had him pegged as an average-ish pitcher.
- Based on shaky major league performances and minor league track record, IWAG expected Mike Foltynewicz to have a terrible 2016. Luckily, he didn’t, and outperformed the modest Steamer (#4 starter-ish) and ZiPS (#5 starter-ish) projections.
- IWAG expected Bud Norris to be substantially below average (4.50 FIP, 0.8 fWAR/200), which is notable only because Steamer had him pegged for a 3.87 FIP and 2.3 fWAR/200, while ZiPS was at 4.13 and 1.6 fWAR/200. Norris ended up at 4.33 and 1.2, respectively, which makes him one of two players (along with John Gant) who ended up in the middle of the projections, rather than above or below all of them.
fWAR
More saliently for the team’s actual performance, though, was how the FIP was spread across innings actually pitched; that is, the actual WAR put up by the team’s starting pitchers. The chart below summarizes the projections (spread across actual innings completed) and actual outcomes for the 11 pitchers.
The data displayed above are a lot of things, but one of them is depressing, given how few of the bars climb above the 2.0 mark (or even the 1.5 mark). No one could have predicted the negative value beating that Aaron Blair and Tyrell Jenkins took, and Ryan Weber didn’t help much either. On the flip side, Julio Teheran went on and above and made everyone look silly for their conservative projections, while IWAG got some pie in its face with regard to Mike Foltynewicz. Jhoulys Chacin did much better than expected (mostly for the Angels, though), and IWAG being the low projection for Matt Wisler is also notable.
Summary statistics ensue:
Average projection difference (positive numbers mean the projection system overestimated, negative numbers mean it underestimated):
- Steamer — +0.54 points of FIP, +0.2 fWAR
- ZiPS — -0.54 points of FIP; +0.1 fWAR
- IWAG — -0.25 points of FIP, -0.1 fWAR
On average, after accounting for the number of IP for each pitcher, the projection systems were pretty much right there. The spread in FIP is bigger, but again, a lot of that spread is in Tyrell Jenkins and Aaron Blair, who only completed 72 innings total at the major league level.
Root Mean Square Error (this is just another measure of how well the projections worked, which is generally considered better than using a straight average because of how it penalizes large projection errors):
- Steamer — 0.96 points of FIP, 0.8 fWAR
- ZiPS — 0.96 points of FIP; 0.7 fWAR
- IWAG — 0.86 points of FIP, 0.7 fWAR
Closest Projections (ties count as a “point” for both systems, using fWAR):
- Steamer — 3 players
- ZiPS — 5 players
- IWAG — 5 players
Best Projections:
- Steamer — Perfectly nailing the FIP of Mike Foltynewicz
- Special shoutout to Williams Perez, who was projected for a lower FIP, but essentially the same value-rate basis by Steamer and IWAG. I suppose that means he must have pitched in some pretty high run environments across his 54 innings.
- John Gant was also collectively predicted fairly well. his FIP of 4.39 was fairly close to the projections (4.21, 4.45, 4.13). Once again, even a lack of major league track record is not always a problem for the projections, though it does increase the variance quite a bit (again, see Aaron Blair and Tyrell Jenkins).
Worst Projections:
- ZiPS — Aaron Blair was projected at a 3.94 FIP and 2.2 fWAR/200, and ended up with a 6.15 FIP and -1.7 fWAR/200. Yikes.
- Steamer — Julio Teheran was projected with a 4.06 FIP and 1.9 fWAR/200, and ended up at 3.69 and 3.4. Steamer has continually underrated Julio Teheran’s productivity (this isn’t the first year this has happened), and this time, it proved costly.
- Collectively, the three projection systems did the worst in terms of Tyrell Jenkins’ FIP (because IWAG was not sanguine on Blair, and Jenkins’ actual FIP ended up being unimaginably bad). Similarly, the systems all erred as far as Julio Teheran’s production by being overly conservative, given that he blew even the ZiPS 2.3 fWAR/200 projection away.
So, that’s about it for the projection review. Any thoughts or takeaways that I didn’t touch on above? I’ll try to do better in 2017, though much in the vein of 2016 Erick Aybar, I’m sure strange happenings will occur anyway.