Ad Hoc Data Analysis From The Unix Command Line

Once upon a time, I was working with a colleague who needed to do some quick data analysis to get a handle on the scope of a problem. He was considering importing the data into a database or writing a program to parse and summarize that data. Either of these options would have taken hours at least, and possibly days. I wrote this on his whiteboard:


 * Your friends: cat, find, grep, wc, cut, sort, uniq

These simple commands can be combined to quickly answer the kinds of questions for which most people would turn to a database, if only the data were already in a database. You can quickly (often in seconds) form and test hypotheses about virtually any record oriented data source.

Intended audience
You've logged into a Unix box of some flavor and run some basic commands like ls and cd and cat. If you don't know what the ls command does, you need a more basic introduction to Unix than I'm going to give here.

Table of Contents

 * 1) Preliminaries
 * 2) Standard Input, Standard Output, Redirection and Pipes
 * 3) Counting Part 1 - grep and wc
 * 4) Picking The Data Apart With cut
 * 5) Joining The Data Together With join
 * 6) Counting Part 2 - sort and uniq
 * 7) Rewriting The Data With Inline perl
 * 8) Quick Plotting With gnuplot
 * 9) Appendices