Pratfalls in Primitive Prognostication

Earlier this year a tweet brought back some long-forgotten fun with projections:

Rubén Sierra was signed by Texas as a 17-year-old out of Puerto Rico in 1982, back before P.R. was added to the MLB draft (in 1990, along with Canada). He came up with the Rangers in 1986 at 20, and had a solid first three seasons in right field — OPS+’s in the low 100s, including a 30-HR season at 21 in 1987 (a big hitter’s year). Then in 1989, at 23, he hit .306/.347/.543, leading the AL in Slugging, good for a 146 OPS+. He finished 2nd to Robin Yount in the MVP voting that year in a close vote.

He had a down year in 1990, and it was after that ‘90 season that a young schmenkman was finishing his career projections program*, and used it to generate projections for every major league hitter with more than a couple years’ experience. Below is a shot of an old printout of Sierra’s projection, and it turns out it wasn’t quite as laughable as 500 HRs would have been, but still pretty bad — 449 HRs (and 3,444 hits):

Sierra went on to play parts of 20 seasons in MLB, in a career that included 100+ RBIs four times, some World Series heroics, and a Comeback Player of the Year award. Players with the most similar career stats start with Raul Ibanez and Joe Carter, and include Paul O’Neill, Don Baylor, Del Ennis, and Dale Murphy. His SABR bio is worth a read.

He finished with 306 HRs and 2,152 hits — very impressive numbers, but well short of the 449 and 3,444 in that projection.

Despite the big miss, there were still some kernels of good stuff there, including these advanced stats:

- No OPS, but instead OBA*SA (On base average times Slugging Average, “OTS” if you will), because there was some research that multiplying the two together provided a more accurate gauge than adding them (and OPS was just as obscure as OBA*SA at that point). On the other hand, multiplying is obviously harder to do in your head, which makes it much less useful. It was proposed by Bill James, for one, and in his 1982 Abstract (the first of his annual Abstracts to be distributed widely), he described offensive value as a rectangle where the width is a hitter’s on-base ability, and the height is their slugging (or “advancement”) ability, and the area of the rectangle (width x height) is a good approximation of their overall value.
- Runs Created, developed by Bill James (as well as annual, 3-year, and 5-year views of Runs Created per “game”, i.e. per 27 outs made)
- LW are Runs Above Average based on Pete Palmer’s linear weights (from books coauthored with John Thorn), such as The Hidden Game of Baseball and the annual Total Baseball encyclopedias.
- TA is Tom Boswell’s Total Average, which was intended to measure a hitter’s overall offensive effectiveness on the basis that “all bases are created equal”.

In addition, the concept of replacement level was used, based on the difficulty of the player’s defensive position. When the aging curve dropped a player’s production (in RC/G) below that replacement threshold of overall hitting, his career ended.

It’s a sort of sabermetrics time capsule, from a period when principles and ideas around baseball analysis were still being developed and refined. They were also being tested using ever-better computing resources, but there was no widely used internet yet to make these ideas broadly available to statistically-inclined baseball fans. A longer timeline is here, but these are some selected mile posts:

1982: The first of Bill James’ annual Abstracts to be widely distributed
1984: The Hidden Game of Baseball is published by Pete Palmer, John Thorn, and David Reuther, introducing run expectation tables, among other concepts
1989: The first Total Baseball encyclopedia is published by Palmer and Thorn, intended to correct and replace the long-standing bible The Baseball Encyclopedia by Macmillan, while also adding sabermetric stats like Runs Created and Total Average
1990s: A’s GM Sandy Alderson begins using sabermetric principles to identify undervalued players
1995: Sean Lahman makes the Lahman Baseball Database freely available for download
1996: Baseball Prospectus is founded, and the site launches the following year
2000: Sean Foreman builds a web interface to the Lahman Baseball Database, which eventually becomes baseball-reference.com
2003: Moneyball is published
2005: FanGraphs launches

(also, after reviewing this history I have a new-found appreciation for Davey Johnson**)

Looking back, we can see how this “system” did, using that term loosely. We could compare Runs Created, which as they’re calculated here are in the same ballpark as the stat of the same name tracked at Fangraphs, for example. For Sierra’s first 5 years in the league...
- in the projection..... 53, 90, 81, 121, 88
- and at Fangraphs.... 52, 86, 76, 115, 81

The LW (linear weights) numbers are also mostly in the same range as the Runs Above Average (RAA) stat at Fangraphs:
- in the projection..... 6, 13, 8, 42, 16
- and at Fangraphs.... 5, 1, 3, 40, 6

However Hits are also fine for this comparison, and are a handy measure of both durability and effectiveness.

Any projection system, if it’s not biased, should be about as likely to overestimate, as underestimate. That was not the case here. I looked at the first 36 players with projections (last names beginning with A through H), plus Sierra. Of the 37, only four (11%) exceeded their projections:

- Carlton Fisk, who was already 42 in 1990, and lasted even a bit longer than the model projected (164 more hits, vs. 100 predicted)
- Andres Galarraga, thanks to a big career resurgence in Colorado (1,585 more hits, vs. only 613 predicted)
- Billy Hatcher, continuing to get jobs long after he was good (438 vs. 216)
- Rickey Henderson, the big stopped-clock-is-right-twice-a-day success story here, thanks to playing forever and playing well at that (1,293 vs. 1,288, only 5 off).

By the way, at least for Sierra, and also for someone like Hatcher, their careers lasted much longer than they probably would today. Sierra finished with 16.8 bWAR — 20.2 with Texas (his first 10 years), and -3.4 in the 10 years he spent playing for 8 other teams. Modern front offices would have moved on to cheaper options much sooner.

Below are the 37 players, with their projected hits for the rest of their careers (i.e. for 1991-on) along the x-axis, and their actual hits along the vertical axis. Those below the solid diagonal line had projections that were too high:

There are two main reasons for the vast majority of the estimates being too high: 1) injuries, which is a fine and understandable reason, and 2) survivor bias, which is just an error in the model.

And in Sierra’s case (and really, most cases), survivor bias was the big culprit: the rudimentary aging curves were based on hitters’ major league production by age, and if a player was no longer good enough to play in the majors from say age 32 on, that player was simply not included in the analysis for those ages. The player’s age 32 production (which would have been below replacement level) was not included when calculating average age 32 production. Only those players who were still good enough to play in the majors at 32 were included. That’s an issue that even young schmenkman recognized, but didn’t get around to fixing before real life pushed it aside. And to be honest, I’m not sure he would have known how to go about fixing it anyway.

Thankfully projection systems are quite a bit smarter today.