Stories/PietdeJong

Piet de Jong, Statistician

As a statistician my preferred platforms are J and Mac. For standard stuff J on the Mac is fine. But when you get into heavy number crunching, J on Windows is better. In particular I am referring to the LAPACK interface which is available on Windows. Having LAPACK is a great time saver for numerically intensive computations. I use LAPACK mainly for QR transforms which are embedded in J code to do Kalman filtering and smoothing. It works fine.

Fortunately the J platforms on Windows and a Mac work pretty much the same. It is trivial to move from one platform to the other while using J.

When I really want to get into heavy duty stuff I use a Win/J machine. For example at the moment I am analyzing just under 1/2 a million cases on 40 variables. Having J is a dream come true and super fast (compared to, for example the ponderous, constrained workings of SAS). With J it easy to explore and do things on the fly with conciseness and intellectual elegance.

The SVD operation on very large matrices is a dream with J and LAPACK. For many matrices it is ok throwing things inside an explicit J loop as I do with Kalman filtering.

Learning J is another matter. Great effort is required (or at least great effort was required by me). But it is intellectually satisfying effort, unlike other statistical "languages" I'm familiar with which (relative to J) appear to be a hodge podge collection of half baked kludges. One note, when I worked out what a J "item" was, and the relationship between items, rank, verb rank etc I felt that I had learned something new about arrays and how to think about data, operators across arrays and much more. J is a great intellectual achievement. That is one reason I stuck with it.

Programming in "tacit" J is dream come true. With J you can approach almost any data set without being boxed in by artificial constraints and the usual conventional ideas of how the data should be structured. Being able to effortlessly transcend the usual straitjackets frees up thinking and opens doors.

Anyway that is my 2 cents worth.