Stolen from "Jose Can You Seabiscuit" from this thread at Baseball Think Factory, while discussing how small sample sizes are manipulated when making arguments:
For example, Jack Morris was a great postseason pitcher because he threw a 10-inning shutout in the (1991) World Series. What about the 1992 World Series? Doesn't count.
As for what to do with Joba Chamberlain (the focus of the thread, if you didn't click through) - starters are simply more valuable than relievers. That's why the best starter on the market gets five to seven-year deals at over $20 million a year, and the best reliever on the market* is lucky to get a three-year deal at half the AAV. If you have a young player who has the potential to be an ace starter or a shut-down closer, you try to make him into a starter first.
* Mariano Rivera doesn't count, because he is in a class by himself. Perhaps the single most misguided piece of conventional wisdom of the last 15 years is the notion that any other reliever on earth could provide the same results for their team as Mariano Rivera has done for the Yankees. The man is inhuman, and mere mortals cannot replicate his results.
The Yankees should give Joba a chance to be a starter for a full season, with no restrictions and no manipulation of his pitch count or his spot in the rotation and see what happens. If he flops after 30 starts, then you consider making him a set-up man again.
Putting Chamberlain in the bullpen now is a quick fix, one that will certainly yield a measure of success, but will prevent the Yankees from ever finding out his full potential. There may be a #1 starter inside Joba that will never emerge if he spends two or three years bridging the gap to Rivera and another 10 years protecting three-run leads in the ninth inning.