User:Devon McCormick/JTalk20140121

J: the Hard Parts

On 1/21 2014, I spoke to the "7 languages in 7 months" meetup at The Ladders office in New York City. Here are my slides from that talk in which I attempted to introduce the J programming language to a group interested in programming languages per se. Relying on the presumed sophistication of such a group, I attempt to introduce a few of the key concepts in the language that I think are perhaps the most difficult to grasp.

Introduction

This graph represents a subjective view of how various programming languages are related to each other. The data are all invented: the X and Y axes represent my feelings, where higher is better. The only significance to this graph is that languages are rated on a scale of one to ten, but J goes to eleven.

J Language Hierarchy

Here I introduce J's functional hierarchy and briefly explain what "nouns" and "verbs" are. I explain that most of the talk will be about these two classes of things in J, that I'll only briefly touch on the higher-levels of adverbs and conjunctions but will go into some depth on nouns and verbs.

Dictionary of J Language

This slide and the next two present the J language in its entirety. This dictionary is from the "NuVoc" page on the J wiki. The colored columns contain all the J symbols; the color-coding distinguishes different types of symbol as indicated by the key at the top. There are usually two names, separated by a dot, for each symbol: the first name denotes the monadic use - where it is applied to a single argument on its right - whereas the second name is its dyadic use where it is applied to two arguments, one on each side of it.

Notice how most of the symbols on this page are verbs - this is an important part of the language.

On this page we see more non-verbs than on the other page, as well as a couple of three-character symbols. Notice how the first column generally has a single-character symbol, the second version of the name is usually this character followed by a dot, and the third version is the character followed by a colon.

On this page we see that we've run out of symbols and have resorted to letters. Notice how many advanced concepts these J symbols address - things like polynomials and Taylor coefficients.

The dictionary page we see, in sections, above is a new version of this original one. However, the following pages for details about specific J "words" are taken from the links referred to by the original one. This represents efforts by the J community to present a variety of learning materials based on our own experiences with learning the language. The community continues to welcome feed-back and helpful criticism from anyone interested in finding out more about J.

Example Dictionary Entry

Here we see some detailed definitions of two of these symbols. Note how sometimes the monadic and dyadic cases are closely related, as shown by negate and minus at the top, but are sometimes conceptually distinct, as shown by the lower section. Here, we see that the less-than symbol is used dyadically as a comparison verb; by looking at the example of dyadic use to the far right, we see an extract of a J session where the user input is indented three spaces and the system output following it is not indented. Note how the result of a true comparison, like "1<2", is a one for "true" whereas the result of a false comparison is zero for "false".

The lower example shows how the monadic use of "<" is not closely related to the dyadic use as we see in the upper example. The monadic use of "<" is "box", which encloses a noun, like the numeric vector "1 2 3", in a box, turning it into a boxed scalar. Here the meaning of the monadic case relates more to the visual aspect of the symbol, where we see the larger, open end on the side of its argument, suggesting a wrapping-up of it. An inverse of the monadic "box" is the monadic use of the ">" symbol which suggests the reverse operation of "unboxing".

Digression on Cognition

We'll now step back from a direct discussion of the J language to cover a few more general concepts about cognition: how we think. This is to explain an aspect of J that most people find immediately peculiar compared to most other programming languages: its terseness which is, in part, due to its nature as a symbolic notation. This compactness of the language, which compresses a lot of meaning into a very small space, is deliberately chosen for a number of important reasons relating to the way our minds are able to deal with complex concepts.

For instance, when we learn chess, we start by learning the names of the pieces and how they move - much as we are now learning the special names J has for its components. Once we've learned this most basic thing, we advance to slightly more abstract concepts, things like "protect your king", "control the center", and that the pieces are seen as having numeric values of their worth.

At a level above this most basic knowledge, we start to look beyond these details and start seeing larger patterns. The more advanced knowledge layers on top of the more basic knowledge, allowing us to think in terms of larger concepts like standard openings and patterns of pieces on a board.

Continuing this theme of moving from basic to advanced concepts, we can say that we encapsulate many pieces of basic information into some sort of compressed idea, perhaps represented by a symbol or word. Since our conscious thought appears to be limited by how much we can hold in our working memory, this allows us to work with more complex ideas by encapsulating the more basic ones as tokens, then combining these to build more complex notions.

We accomplish this - sometimes known as "chunking" - by organizing and grouping our information in some orderly fashion. Once we have a sufficient set of higher-level concepts, we can use them like we would use building blocks, not having to think about their underlying details. This allows us to work at a more advanced level.

A language like J gives us a very useful set of "blocks" for building computational concepts and talking about algorithms at a high level. This allows us to focus on a problem rather than being distracted by unessential aspects of a computer.

In the blog entry on the lower right, we see how this sort of focus can be eye-opening to someone perhaps more used to the traditional view of a programming language as an obstacle to be overcome rather than as a tool of thought.

Benefits of Terse Notation

Here I'm emphasizing the benefits of a notation - especially a compact one. The more we wrap high-level concepts into symbols couched in a simple, consistent syntax, the better able we are to manage thinking about complex algorithms.

Technical Aspects of J

With that in mind, let's focus on one of the most important basics about J "nouns" - the simple, consistent notion of "array" that underlies them.

So let's talk about arrays which, in J, are homogeneous with respect to type, and orthogonal ("non-ragged").

Simply Arrays

Most programmers seem to find the idea of higher-dimensional arrays fairly easy to grasp even if most languages don't provide very good support for them. However, a key to understanding the simple logic behind arrays in J is at the other end of the dimensionality spectrum: empty arrays. This seems to pose the most difficulty for people but it follows logically from the key concept that "the shape of an array is a vector of non-negative integers" (and a vector is a one-dimensional array). Once you understand this, the rest of it follows logically.

What I'm attempting to illustrate in the slide above is that the shape of a scalar is the empty vector, usually denoted by "i.0" or " '' " (a pair of single-quotes representing the empty character vector). This doesn't mean there's nothing in a scalar - it has a value as shown above where the triple summation of a three-dimensional array gives us the scalar "276" - just that its number of dimensions is zero.

This is one of many instances where zero is important and useful.

Verbs on Array Dimensions: Rank

Now that we know about the array properties intrinsic to any noun in J, let's proceed up a level in J's grammatical hierarchy to verbs. First, I'll note that new programming languages seem to pop up like mushrooms. And, like mushrooms, most of them have an internal structure that is undistinguished mush, especially when they try to deal with arrays. Also, like mushrooms, not all of them are good to consume.

As an example of this, I'll pick on R even though it's a language I like and it has a better array-orientation than most programming languages. We can create one- and two-dimensional arrays in R but, for some reason, we need to use different words to ask about the shape of each of these: for a vector, we ask for its "length". We can also ask for the length of a matrix, but this tells us how many things are in it. To find its shape, we have to use the word "dim" but, when we apply this to a vector, it tells us nothing useful. However, when we ask for the "dim" of a matrix, we get back what looks like a vector of non-negative integers, just like in J. Why this does not extend to vectors, I can't say.

So, we have a few different things to remember when dealing with the shape of an array instead of one simple, consistent idea as in J.

This array-orientation affects verbs in J as they have an implicit "rank" at which they apply. On the bottom half of this slide, we see the dictionary entry for the J shape primitive "$". As annotated here, this definition shows the symbol $, at the top of the page, followed by three numbers denoting its rank in the monadic case and for the left and right side of the dyadic case.

The monadic case for shape is infinite - denoted by _ - so it applies to the entire argument, at its highest dimension. In the dyadic case - where we reshape the noun on the left by the vector on the left (e.g. 2 6 $ 'Hello,world!') - we see that this left argument is of rank one - a vector - and the right argument rank is infinite. As in the monadic case, reshape works on the array to its right at its highest dimension.

Explicit Rank

This is one of the more difficult-to-grasp concepts in J, so I'll attempt to illustrate its implications by showing how we can override a verb's implicit rank when we use it. Here, we start with our usual three-dimensional table and show how summation - using the derived verb +/ - works along the first axis by default. So we "lose" the first dimension, going from a 2x3x4 argument to a 3x4 result.

However, on the lower right, we see how we can specify the rank at which to apply the summation to work along different axes. In the +/"1 example, we might think of the three-dimensional array (of scalars) as a two-dimensional array of 4-element vectors and sum across these as indicated by the "1 modifying the summation.

In the last example, we think of the three-dimensional array as a two-element vector of two-dimensional arrays when we sum the rank two cells of it with +/"2. This effectively adds up the columns in each of the two planes to give us our 2x4 result.

Verb Composition: Hook

Here we look at the second of the three "hard" things to understand about J. Hook is a verb construct - formed by any pair of isolated verbs (i.e. in parens) - that combines the pair of verbs in a potentially useful fashion.

The continued fraction example at the end hints at some of the interesting implications of the hook construction.

Verb Composition: Fork

Here we have the third and last part of J syntax that I consider part of the "tough nut" at the core of the language. If you understand this, hook, and rank, you've grasped some of the most difficult aspects of the language. In fact, if you understand these well, you probably know the language better than I do even though I've been using it for ten years.

The other centrally-important part of J we've covered - arrays - I don't think is terribly difficult but I've been used to them for a long time now, so YMMV.

Code Examples: Project Euler

For most of the rest of this talk, we'll look at specific examples of J code and I'll try to impress upon you how its clear, simple rules and concise symbols give us extremely useful building blocks for thinking about algorithms.

To come up with some random problems to solve in J, I turned to those of Project Euler, shown above. Project Euler is a set of basically simple problems meant to be solved in one's favorite programming language. This can help one learn a new language and learn about algorithms.

Back in about 2007, the J community discovered Project Euler and shot to the top of the rankings, for reasons that will become clear when we look at some examples. Since then the "APL/J/K" group has drifted down as the small J community has moved on to other things and newer, more popular languages have engaged people in the project.

One point I'd like to make about the Project Euler solutions is based on one of the FAQs shown on the right here: solutions should have reasonably fast run-times of less than a minute. However, to my mind, coders often obsess over run-time performance at the expense of the more important thing I call "total time to solution." I know that, in my own work, the main bottleneck to coding is the time it takes me to write the code, not how long it runs.

Here we see a summary of the first six Project Euler problems and my own J solutions to them. Since I've long lost my original ID for this site, and it's been years since I looked at it, I created a new account and came at the problems anew. You can see from the "Last 10 Problems Solved" table on the lower right that I solved the first ten or so in less than an hour.

This is due largely to having J as a tool with which to solve these problems. You can see from some of the solutions I've listed, especially the second-to-last one, using such a powerful tool as J is akin to cheating on problems like these.

Code Example: Table Manipulation

Here I show the sort of thing for which I use J almost every day: grabbing a table of data from somewhere and doing something with it. Some of what I do could be done almost as easily in something like Excel but, even for these, having the J code to do it makes repetitions even quicker and lends itself to easier automation for anything I end up doing many times.

Resources

Here is a list of resources for more information about the J language. There are several books available, both on-line and in print. My own meetup has been going on longer than meetup.com has - we have notes on many of our past meetings posted on-line.

Various Examples

We'll finish up with various examples of J code. We'll show how "array-thinking" helps simplify coding a solution and we'll touch on some of the higher-levels of the J functional hierarchy: adverbs and conjunctions.

This example illustrates how "array-thinking" can simplify code. Instead of many lines of deeply-nested code as shown at the top, we have a few simple array operations to accomplish the same thing. Whereas understanding the verbose code requires us to keep many things in mind at each nested block - like, what flags have to be updated? Which logic branch am I on? - the array code lays out a simple, static representation of the state of the problem solution at each step.

This slide shows how sometimes J code lends itself very nicely to a direct mapping of code to meaningful words. Here we see examples of applying a correlation verb to two different argument sets to illustrate that it delivers expected results. If we assign the pieces of code to words indicating what each piece does in succession, we can actually use these words directly as shown in the second pair of examples.

Thanks

I'd like to thank the many members of the J community who contributed examples, advice, and feedback on this talk, particularly Roger Hui, Dan Bron, John Randall, David Lambert, Ian Clark, Raul Miller, Pepe Quintana, Henry Rich, Brian Schott, Joe Bogner, the members of NYCJUG, and others.