Baseball Hack 22: translate.pl and rosters.pl

I'm reviving the blog when it applies to my 35 for 35. Yes, I'd love to blog here more, but life has brought along many more important things over these past few years. That being said, here's some Windows-compatible code for hack 22 in the book Baseball Hacks. First, translate.pl. Run this from the same directory your zipped Retrosheet event files are stored, and it will unzip them and concatenate all the play-by-play data to pbp.csv. This code requires the Perl extension Archive::Extract, and also takes advantage of readdir functionality only available in Perl 5.2 or later.

#!/usr/bin/perl
use Archive::Extract;

$outfile = '"C:\Users\John\Desktop\Baseball Hacks\retrosheet\pbp.csv"';
print `type all_hdr.txt > $outfile`;

opendir RSDIR, "." or die "can't open directory .: $!\n";
while (readdir RSDIR) {
 if ( $_ =~ /(\d\d\d\deve)\.zip$/ ) {
   print "Unzipping $_\n";
   my $ae = Archive::Extract->new( archive => $_ );
   my $ok = $ae->extract( to => '.\\' . substr($_, 0, -4) );
   opendir YRDIR, substr($_, 0, -4) or die "can't open directory .: $!\n";
   chdir(substr($_, 0, -4)) or die "can't change to directory .: $!\n";
   while (readdir YRDIR) {
    if ( $_ =~ /(\d\d)(\d\d)(\w\w\w)\.EV[AN]$/ ) {
     $century = $1; $year = $2; $team = 3;
     print `..\\BEVENT.EXE -y $century$year -f 0-96 $_ >> $outfile`;
    }
   }
   chdir("..") or die "can't change to directory .: $!\n";
   close YRDIR;
 }
}
close RSDIR;
print "done\n";

Similarly, here is rosters.pl, which loops through the unzipped event directories and concatenates all roster files for all years into a single file. You must specify this file on the command line, e.g. ./rosters.pl > rosters.csv

#!/usr/bin/perl

print "retroID,lastName,firstName,bats,throws,team,pos\n";

opendir RSDIR, "." or die "can't open directory .: $!\n";
while (readdir RSDIR) {
 if ( $_ =~ /(\d\d\d\d)eve$/ ) {
   opendir YRDIR, $_ or die "can't open directory .: $!\n";
   chdir($_) or die "can't change to directory .: $!\n";
   while (readdir YRDIR) {
    if ( $_ =~ /(\w{3})(\d{4})\.ROS$/ ) {
     $team = $1;
     $year = $2;
     open FILE, "<$_";
     while () {
      s/\n//;
      s/\cM//;
      s/\"//g;
      if (/[a-z]{5}\d{3}/) {
       print "$year,$_\n";
      }
     }
     close FILE;
    }
   }
   chdir("..") or die "can't change to directory .: $!\n";
   close YRDIR;
 }
}
close RSDIR;
print "done\n";

Once you've run this Perl code to create pbp.csv and rosters.csv, you can add them to your SQL database using the instructions in the book.

Way Too Early Must Watch Games of the Week: Weeks 1-3

by F-Bomb

Week 1 (September 1st)

9a, ND vs. Navy (Ireland) – NBC

College football on the international stage. Historic rivals that finally have a rivalry as Navy has
won 2 of the last 3 but ND really shut down the triple option last year in a rout.

3:30p, Miami @ BC – ABC

In this ACC / former Big East battle who will get a jump on the season? These teams have a
pretty solid past including BC giving Miami all they could handle during Miami’s prime in the
early 2000s. This is also a preview of two ND opponents. It wouldn’t hurt to do a little scouting.

8p, Michigan vs. Alabama (Dallas, TX) – ABC

What can Denard do this year? He has shown the ability make amazing plays, especially early
in the year before he gets banged up. The Wolverines will be a preseason top 10 and facing an
Alabama defense that lost 3 guys to the first round of the NFL draft, 1 to the second round, and
another 3 in the rest of draft. Did Saban reload? We will see. Also, can Michigan’s shut down an
offense?

*Bonus*: August 31st 8p, Boise State @ Michigan St. – ESPN

Will Michigan State capture the spark they have had the past two seasons? They lost offensive leader, Kirk Cousins, but return beastly RB Le’Veon Bell. Also, William Gholston comes back to lead a dirty and stingy Spartan defense. Will Boise State continue their string of early season success against BCS teams in their last season in the Moutain West?

Week 2 (September 8th)

TBA, Nebraska @ UCLA – TBA

Only one name needs to be mentioned with regards to Nebraska turning the corner this year
and that name is Taylor Martinez. At times he has showed brilliance but his inconsistency
especially in regards to passing. His TD to INT ratio isn’t great nor is it bad but his poor play
really comes at inopportune times, see the Ohio St. game. Yet he had the ability to help rally
them back. UCLA finds themselves with a new coach in Jim L. Mora / Jim Mora Jr. / Does it really matter it is UCLA. UCLA hasn’t won more than 7 games since 2005. It will be interesting to see if Jim L. Mora can get it done. This out of conference test will be a good measuring stick.

TBA, Wisconsin @ Oregon St. – TBA

Big 10 vs Pac 12. Montee Ball returns to lead the always powerful Wisconsin ground game. How
will Maryland transfer Danny O’Brien be at the helm for QB? What will life be like without the
Rodgers brothers at Oregon State? They always seem to be a team on the cusp who can at least
play spoiler in the Pac 12. How will they handle run strong team in Wisconsin?

TBA, Florida @ A&M – TBA

This is A&M’s welcome to the SEC. Can Will Muschamp’s Gators make the jump from a
disappointing 7-6 season? Will A&M be able to move on without Ryan Tanehill or his super
attractive wife? A&M had a disappointing season last year. They should have won games against
Oklahoma State and Arkansas but those slipped away due to a lack of defense. They also lost to
Kansas St. and formal rival Texas.

*Bonus*: 3:30p, Purdue @ ND – NBC

Purdue has been disappointing over the past few years but they return several defensive stars who are on the Bednarik watch list. Will this translate into the wins that have escaped this once consistent Big 10 team? ND looks to continue their streak over Purdue but with the QB debate still left to be settled they could be in for a long game with Purdue’s projected stingy defense.

Week 3 (September 13th)

TBA, USC @ Stanford – TBA

USC is projected to be at the top of college football once again. They return Heisman front
runner in Matt Barkley. Their top WRs, Freshman All American Marqise Lee and Robert Lee,
are both back for another season. The Trojans almost topped Stanford last year but ended up
losing in triple over-time. Not bad for a team that was on suspension. Finally Stanford looks to
move on without their star QB, Andrew Luck, who was selected #1 overall in the NFL draft. Can
they build themselves into a consistent top 25 program or will they fall back to the middle of the
pack? This game should be full of big hits as USC tries to reclaim dominance in the series after
losing 4 of the last 5.

TBA, Alabama @ Arkansas – TBA

Arkansas brings back a QB, WR, RB trio slated to be at the top of the SEC. This offensive
firepower could turn the tide in a defense heavy league. This early season matchup with
Alabama should be a test for both teams. Alabama, after the Michigan game, should know
better where they stand and Arkansas is one of those SEC matchups that may be a problem
game if Alabama loses sight of the goal of another National Championship.

TBA, Notre Dame @ Michigan State – ABC

This matchup has been full of intense games for the past 12 years. There have been last minute
wins by both teams including two years ago the MSU fake field goal / Mark Dantonio heart
attack game in 2010. Last year’s game was in doubt even though ND had a two possession lead
into the 4th quarter. With MSU returning a slate of defensive stars will they play above ND or will
it be another grudge match.

*Bonus*: 12p, Cal @ Ohio State – ABC

Ohio State returns their prized athletic QB, Braxton Miller. Will they be able to carry over the
momentum they had towards the end of last year which included a victory over Big 10 champion
Wisconsin? That said, they also suffered their first loss to Michigan since 2003. As for Cal, Jeff Tedford’s team looks to build off a 7-6 season in which they were blown out by the better team every time except a for a surprising 31-28 loss to Stanford. Cal will certainly be tested by traveling to Columbus for yet another Pac 12-Big 10 matchup.

*Bonus #2*: TBA, Florida @ Tennessee

The world’s largest cocktail party features SEC powers of yesteryear. Derek Dooley continues to try
and bring back the prestige that Tennessee had under Phil Fulmer. Will Muschamp is trying to translate recruiting success into wins on the field.

The Class of 2012

It was a small electorate this year - only 3 voters plus the carryover. As such, the threshhold for election was 2, and there will be no carryover votes next year.

Yeah, yeah, they're aren't any pictures. Consider it a weak (and belated) SOPA protest or something like that. Without any further adieu, here's the first all-female class of the KankaNation Hall of Fame

Mistelle
A million times stronger than any of us losers. Plus she puts up with Pete.

Muffett McGraw
Two halls of fame in two years, plus the best all-around team she's had in a decade.

Rose Kanka
Not even three months old, and already totally awesome.

Ramblings (or: Possible Future Column Ideas)

• Former Indians GM John Hart said that the new MLB CBA was a good thing because "it gets the best players on the best teams, and that's what fans want." No, fans want their own team to be competitive.
• The new CBA limits spending on the draft and international amateurs, but does nothing to limit spending on free agents. Why? Well, look at the parties involved. One one side, the owners want to save money (and some/most care about saving money more than winning). On the other side is the Players Association. Draftees and international amateurs aren't in the union yet (neither are signed minor leaguers until they're added to a 40 man roster), so cutting their money keeps the owners happy without taking money away from those in the Players Association.
• There are some that say the Moneyball approach is stagnating, some 10 years after the book was published. Well, now the arms race begins again as smart teams who had been spending their resources on the draft and international amateurs will now spend their time and effort trying to get ahead in the climate of the new CBA. It sucks as a fan of a small team, but it will be fun to watch nonetheless.
• No, the Albert Pujolses and Chris Pauls of the world shouldn't be "well-paid slaves" - they should be able to play wherever they'd like. That doesn't make it any easier for the clubs they're leaving though.
• That being said, the NBA needs to watch that they don't become a league consisting only of the Lakers, Heat, Celtics, Knicks, and 26 feeder teams. Of course, the only way for the "feeder teams" to counter their unattractiveness to superstars, they will need to develop their own Moneyball approach to roster construction, whether through underrated veterans, four-year college players, or even a return to a team-first college approach that simply out-hustles the superstar teams.
• Robert Blanton is expected to be a mid-to-late round draft choice. But I wouldn't be surprised if his career outshines his draft position. Blanton has always been a hard-hitting ballhawk, and could easily play both corner and safety. That versatility can keep him in the NFL for a long time, much as it has for Cleveland's Mike Adams.
• Those few people still complaining about Brian Kelly running the new-fangled spread offense at Notre Dame remind me of the people who complained about Frank Leahy going with the new-fangled spread over Knute Rockne's traditional box formation. And for what it's worth, Kelly's "spread" looks a lot like Charlie Weis's "pro-style," which to me means there's a lot more to an offense than just the name.
• Moneyball is a recurring theme in this post, and there's a point where Billy Beane asks why his [stuff] doesn't work in the playoffs. Arguably, it's because his hitters' top ability is taking walks, which works over the course of the season against a variety of pitchers, but fails against the best teams and their shortened rotations. But I digress. A similar question is, "Why doesn't Mike Brey's [stuff] work in the tournament?" Is there a specific component of Brey's gameplan that works well enough during the long season against a variety of opponents, but fails against the best team? Definitely worth further research.
• Speaking of Brey, should he be praised for his great ability to bring in transfer students like Ryan Humphrey and Ben Hansbrough? Or does it just seem like the basketball team takes in a lot of transfers since the football team takes in almost none? How many transfers do other schools take in, in basketball and in other sports? Yet another component of Brey's legacy worth looking into.
• I can't understand the (mostly national) media calling for Colt McCoy's replacement as Browns quarterback. He's a second year starter who had about a month to learn a new system before the start of this season. Plus, as other clubs have shown, it doesn't matter how good the QB is if he's surrounded by a good run game and good defense. (Sorry Dave, it's not the best Obligatory Tebow Reference, but it's the best I could do.) The Browns have too many other holes right now to waste a high draft pick on another quarterback who may or may not work out.
• Speaking of the Browns defense, they have some good pieces in their front seven, but have been pretty bad against the run. Is it because the unit hasn't played together enough yet? Or because the young guys like Phil Taylor and Jabaal Sheard focusing too much on the pass rush because it's their specialty? Or is this unit just not as good as the sum of its parts?

It's Ballot Time Again!

Once again, it's time to vote for the KankaNation Hall of Fame.

List up to 10 people you think should be in the KankaNation Hall of Fame, and email to kanka@kankasports.zzn.com by 5 pm Eastern on December 31.
Those who receive a certain percentage of the vote (depends on how many ballots are received; usually 66-75%) will join the Classes of 2004-2010. Those who received multiple votes last year will receive one carryover vote this year.

To view past inductees and voting results, click one of the links below:
Class of 2004
Class of 2005
Class of 2006
Class of 2007
Class of 2008
Class of 2009
Class of 2010
Class of 2011

Notre Dame Football 2011: BEAT SC Game Preview

USC Offense vs Notre Dame Defense

USC's offensive playcalling has been more-or-less balanced, only slightly favoring the pass. The Trojans may want to favor the pass a little more this Saturday, both because of Notre Dame's defensive reputation, and because leading rusher 26 Marc Tyler is expected to miss the game after dislocating his shoulder against Cal. Tyler's primary backup, 22 Curtis McNeal, is averaging an impressive 7 yards per carry, but that's mostly as a 5-7, 180 pound change of pace back. Assuming McNeal can't take a full game's worth of snaps, the Irish will also see a healthy dose of 30 DJ Morgan.

Quarterback 7 Matt Barkley wouldn't be considered a mobile quarterback by most standards, but he has gained more yards on the ground than he's lost (thanks partly to only 4 sacks on the year). Fullback 40 Rhett Ellison has only run the ball once this year (for -5 yards), but he has been an effective outlet in the passing game, averaging two catches per contest.

USC's passing attack centers around wideout 2 Robert Brooks, who is averaging 10 catches per game. Fellow receivers 9 Marqise Lee and 80 Brandon Carswell are averaging less than 3 catches per game, and no other Trojan receiver (outside of fullback Ellison) is averaging more than one per game.

Regardless of who he's throwing it to, Barkley has been a very accurate passer this year, completing 68 percent of his throws, with 16 touchdowns to only 4 interceptions.

Not surprisingly, Manti Te'o is the key component of Notre Dame's defense. Te'o is leading the team in tackles (59), tackles for loss (8.5), and sacks (4). Cornerbacks Robert Blanton and Gary Gray have two interceptions apiece to lead the team, while Harrison Smith has 7 pass breakups - many of them near interceptions.

Notre Dame Offense vs USC Defense

The Irish have put up spectacular numbers on both sides of the ball. The run game is averaging close to 200 yards per contest, with Cierre Wood at 108 yards per game and Jonas Gray at 66.

Tommy Rees is completing 66 percent of his passes, with 14 touchdowns and 6 interceptions. Not surprisingly, Michael Floyd has been his main target; Floyd is averaging 9 catches per game. Tyler Eifert has come on of late as Rees's #2 option, and is averaging 5 catches per game. Wideouts not named Floyd (Theo Riddick and TJ Jones) are averaging 4 catches per game, and Cierre Wood is averaging 2. Riddick got a boost of confidence with playcalling that centered around him in the Air Force game - now let's see if he can carry that forward into this week.

As usual, USC's defensive strength is in its linebackers. 18 Dion Bailey is a converted safety, and he still plays like one in many ways. While Bailey leads the team in tackles with 48, he's also first on the team in interceptions with 2. Defensive end 8 Nick Perry leads the Trojans with 4 sacks. Both are impressive numbers, but this is a USC defensive unit that isn't quite as good as it was in its glory days last decade.

USC also has 5-8, 165 21 Nickell Robey starting at one corner. Even if Robey isn't matched up on Michael Floyd, it's likely that he'll have to guard the 6-2 TJ Jones, and that's a matchup Tommy Rees will have to look to exploit.

Special Teams

Southern Cal's 48 Andre Heidari has been near perfect on the year, converting 10-11 field goals, including 3-4 from 40+. David Ruffer, by comparison, is 4-8 with a long of 39.

USC punter 35 Kyle Negrete is averaging just 39 yards per punt, but he has put 10 of 18 inside the 20. However, opponents are averaging 16 yards per punt return. The Irish punt team has similar mediocre numbers, with Ben Turk averaging 37 yards per punt and opponents are averaging 22 yards per return.

USC's primary punt returner is Nickell Robey. He's averaging 10 yards per return with a long of 44. For the Irish, John Goodman is averaging just 0.3 yards per return, but rumors are flying about Michael Floyd dropping deep to return punts in this game.

Robert Woods returns kicks for the Trojans, and he's averaging 26 yards per return. For the Irish, George Atkinson is averaging 29 per return, including an 82-yard touchdown.

Andre Heidari also kicks off for the Trojans, and he's averaging 63 yards per kick, while his teammates are giving up 20 yards per return for an average start at the 26. Five of Heidari's 36 kicks have gone for touchbacks. Notre Dame's Kyle Brindza is averaging 67 yards per kick, with 7 of his 35 kicks going for touchbacks. Notre Dame is giving up 22 yards per return, meaning opponents start at the 24 yard line on average.

Prediction

Notre Dame 33, USC 25

Evacuating Notre Dame Stadium

In the past 13 years of attending games at Notre Dame Stadium, I've sat in a pile of ice and snow, and I've baked on cloudless 95-degree days.  I've seen the crowd cheer for a squirrel running into the end zone when the Notre Dame offense could't, and I've seen sprinklers try to stop Pittsburgh when the Notre Dame defense couldn't.  But I'd never had to evacuate the stadium until Saturday - and neither had anyone else.

Saturday started as one of those cloudless 95-degree days.  When PA announcer Mike Collins did his usual pregame weather report, he got wild cheers when he mentioned they were tracking storms headed to the stadium, since rain sounded like quite a relief at that point.  I had also seen a weather report that called for possible showers, but they didn't look possible at that point.

Towards the end of the first half, there was a brief but refreshing shower.  They sky still didn't have any storm clouds though - at least until the end of the half.  As the band was running on the field for their halftime show, Collins asked them to stay on the sideline.  Immediately after that, he asked them to seek shelter in the tunnel.  Then, the now infamous announcement came:

"Would Usher 800 please report to the press box?"

While still in school, I knew a student who had gotten a job as an usher after graduation.  This was briefly after 9/11, and he had told us that a message calling usher 900-something to the press box was code for a terrorist attack.  So I knew this message was a similar emergency code.  Sure enough, minutes later we were being asked to seek shelter.

Fans were given the option of simply staying on the concourse, or moving to one of the nearby buildings.  One of the buildings mentioned was Fitzpatrick, the engineering building.  I figured that would be our best bet.  One, our group wouldn't have to stand on a packed concourse for an undetermined amount of time.  Two, we'd have easy access to restrooms and vending machines in a building that probably wouldn't be ask crowded as the other options mentioned (among them DeBartolo hall and the Joyce Center).  Plus, Fitz has not one but two basements in case of a tornado - not that one had been mentioned in the weather reports.

The evacuation itself was quite orderly - I don't think people were taking the threat of lighting seriously, especially when it wasn't even raining yet.  In fact, if anything people were slow in leaving the stadium - my wife and I waited close to 10 minutes for my dad and sister to catch up with us from their seats 11 rows above.

My wife, dad, sister, and I set up shop on some couches on Fitzpatrick's second floor, right outside the dean's office.  The wife and I then set out for some vending machines.  When we came back, my dad and sister were nowhere to be found.  It was a bit eerie - their coats and newspapers had been abandoned as if they were kidnapped or needed to flee to the basement.  As it turns out, they had just been invited into the dean's office - by Dr. Brockman, one of my former professors - for freeze pops.

After a two hour delay that included Dr. Brockman and his son stopping outside to play Frisbee between the storms, and a text from my brother-in-law about the press box being hit by lightning, we finally got a message - piped from the stadium press box into the designated evacuation buildings - letting us know the game would be starting again soon.

We made it back to our seats after missing only the kickoff and USF's brief opening drive.  The crowd was electric at that point.  Maybe it was the fact that most of them had been standing around on the concourse for over two hours.  Maybe it was that others had used that as additional drinking time.  Or maybe it was just the announcement that Tommy Rees had taken over for a jumpy Dayne Crist.  But that crowd was ready to go, and the team responded.

Halfway through the fourth quarter, we could see more lightning off to the south of the stadium. On the field, it seemed like NBC was making up for lost commercial time, and I groaned that that was going to cause us to get stuck in another weather delay.  Sure enough, with just under five minutes left in the game, Usher 800 was again called to the press box.

This evacuation was a little less orderly, as most people just wanted to get out and go home at that point.  But there were still no incidents, at least in my section.  My pregnant wife refused to climb the ramp a third time that day, so our group made the decision to pack up for the hotel, and just lie and say that we had stayed if the Irish did in fact come back to win.

We started to make the trek back to White Field, stopping for about 15 minutes under the overhang of the library when it started pouring.  Believe it or not, after 11 hours on campus, that was the first time we had actually gotten rained on.  After the rain let up, we continued to White Field, and heard the announcement about the game resuming as we were passing Stepan Center.  However, we decided to continue on.

We made it to our car shortly after the game resumed, and were amused by a very cranky Don Criqui's take on the events of that day.  We were able to listen to the final touchdown as we were pulling into the hotel parking lot, and see the onside kick from the hotel lobby.  The Irish weren't able to recover that kick, but at that point the group was too concerned about drying off and getting to bed to care.

So, it was a very eventful home opener this year.  And an eventful day for Jim Smith, who in his first game as Head Usher got to call the shots on the first two evacuations in Notre Dame Stadium's 82-year history.