NYCJUG/2014-01-14
introduction to the hard parts of J, J as a tool of thought, code simplification, code clarity, different directions for implementing user interface in J, coding for children
Meeting Agenda for NYCJUG 20140114
1. Beginner's regatta: progress on talk for "7 Languages in 7 Months" meetup: suggestions on how to proceed? See '"7 Languages in 7 Months" Meetup - Talk on J'. 2. Show-and-tell: J as a tool of thought: see "Example of Algebraic-like Simplification of J Code". 3. Advanced topics: See "J as a Tool of Thought - Code Clarity". Whither J I/O - "wd" or "HTML"? See "Guides/Window Driver/Layout" and "Paul Jackson's NGN". 4. Learning and teaching J: See "Coding Kids".
Proceedings
We reviewed a recent presentation on J that intended to address some of the more difficult aspects of the language. We also studied some examples of the strength of J in allowing for simplification of code by a process reminiscent of an algebraic proof, as well as the utility of J for clearly describing an algorithm.
We explored some different avenues for writing GUIs in J: either a desktop-based method using the new "wd" that runs under Qt, or an example of a browser-based interface as shown by an example in online APL.
Finally, we looked at some recent efforts to introduce programming to young people and some of the controversy surrounding this idea.
Beginner's Regatta
The talk for the "7 Languages in 7 Months" Meetup (now named "n Languages in n Months") is presented, with some explanation here.
Show-and-tell
An example of simplifying code by replacing certains forms of expressions with known equivalents of them.
Example of Algebraic-like Simplification of J Code
Subject: simplify mario sonnino <mario.sonnino@gmail.com> Jan 05 10:51PM +0100
I was reading in the essays the 235 problem , which I think is quite simple to program. Something like:
p=. 4 :'x ^ i. y' NB. build the powers 'e2 e3 e5'=. int 2 3 5 log n=. 2^64 NB. find max exponents c=.(2 p e2) */(3 p e3) */ (5 p e5) NB. find all combinations l =. sort ~. , (n>c) * c NB. find all numbers
My question is: is there a better way to write
c=.(2 p e2) */(3 p e3) */ (5 p e5) NB. find all combinations
in an array-like way, t.i 2 3 5 (something to find) p2 p3 p5
Thanks Mario ---
Subject: Re: simplify James Wood <jammyatjammy@gmail.com> Jan 07 02:44PM -0800
So, you want to have 1 verb applying across the lists 2,3,5 and e2,e3,e5?
Should be possible. I'll step through my derivation, since that's the only way this'll be helpful.
The first thing to do is replace the expression of the form a f b f c with f/a,b,c. So:
(2 p e2) */ (3 p e3) */ (5 p e5)
becomes:
*//(2 p e2),(3 p e3),(5 p e5)
Next, we see an expression of the form (x0 f y0),(x1 f y1),(x2 f y2), and turn it into
(x0,x1,x2) f (y0,y1,y2).
So, our next step is:
*//2 3 5 p e2,e3,e5 NB. e2,e3,e5 needs to be made using the , dyad NB. because e2, e3 & e5 aren't literals
Finally, we swap f x g y out for x f@:g y:
2 3 5 *//@:p e2,e3,e5
In that form, it would be easier to assign to some e array, rather than the individual e2 e3 & e5 items. Also, why explicit p, rather than p=.^i.?
Advanced topics
A fairly extensive exchange of e-mails illustrated the power of J for expressing algorithms very clearly.
J as a Tool of Thought – Code Clarity
from: Dan Bron <j@bron.us> to: programming@jsoftware.com date: Mon, Jan 13, 2014 at 1:45 PM subject: [Jprogramming] Code clarity
We often say the APL family of languages allow us to use language as a tool of thought. How does this play out in practice? Do we approach reading J programs differently from those written in other languages? If so, how?
These questions occurred to me today while I was knocking together an implementation of a RosettaCode task on reading configuration files. The task is to parse file formatted like the following:
# This is the fullname parameter FULLNAME Foo Barber # This is a favourite fruit FAVOURITEFRUIT banana # This is a boolean that should be set NEEDSPEELING # This boolean is commented out ; SEEDSREMOVED
Fuller example at RosettaCode task to read a configuration file
After reading the intro, I copy/pasted the example into a J noun and proceeded to write this:
deb L:0@:(({.~ ; [: < [: ;^:(1=#) ',' cut (}.~>:)) i.&1@:e.&' =')&>@(#~a:&~: > ';#'e.~{.&>)@:(dlb&.>)@:(LF&cut)
Which is a verb which takes the configuration text as input and produces a table of name-value pairs as output. My first thought was "wow, I was able to knock that together in literally less than a minute, through simple incremental iterations in the REPL: J is AWESOME".
But then, thinking about posting it, I realized "this is awful, no one's going to be able to read it like this, and it's going to take more work to make it readable than it took to make it actually work".
So that got me thinking about what exactly we mean by J as a notation. And I wondered: how could we use the language to express our thoughts more clearly, and how does that differ from how we write J when we just want to get something done? And is this a different or more difficult problem for J than other languages?
So, how would you write an configuration file parser in J, if clarity were an important concern? I'm interested in not only the actual program, but the reasoning behind the decisions you make.
-Dan ---
from: [[User:Raul Miller|Raul Miller]] <rauldmiller@gmail.com> date: Mon, Jan 13, 2014 at 2:10 PM
I find that my experiences and insights from working with J transport to other languages (often with some frustrations, but life is full of frustrations) in a useful fashion.
It would take some time, but I imagine you could translate that code you wrote into another language (perhaps paraphrasing parts, depending on your thinking and the target language).
In a much smaller amount of time, I imagine you could take that code and rephrase it to be more suggestive and "digestible". As a start, I'd try thinking up names for every parenthesized phrase (possibly adjusting the boundaries of the parenthesis or the behavior of parts of the code, if that makes it easier to name or describe).
Put differently, it's often worthwhile to think of code as being a "draft document". J is a tool of thought, but that should not prevent you from using other tools nor from having other thoughts.
J can be incredibly productive but it's sometimes worth putting in a little extra effort with an eye towards helping other people understand your thoughts.
That said, you do not need to be immediately critical of your work. Sometimes it's better to just write it down and get onto other issues and trust that you will probably have time to come back later. I've often noticed myself being impatient with things I had written before, and I like to think that my subsequent edits have sometimes helped other people understand my code (especially at rosettacode).
Finally, I think that one of our big limitations, as technical people, is that we often lack audience for our work as individuals. But since everyone faces analogous difficulties (and team efforts get some emphasis for a variety of reasons), I'm not sure if that's so much a characteristic of the technicalities or just something inherent in people.
Thanks,
Raul ---
from: km <km@math.uh.edu> date: Mon, Jan 13, 2014 at 2:31 PM
An important consideration is, "clarity for who"? I want the code to be clear to me when I come back to it later. For that reason I usually code explicitly, calling explicit helper verbs, although many of the lines are "tacit in spirit". Suggestive names help.
--Kip Murray ---
from: William Tanksley, Jr <wtanksleyjr@gmail.com> date: Mon, Jan 13, 2014 at 4:08 PM
Dan Bron <j@bron.us> wrote:
> We often say the APL family of languages allow us to use language as a tool > of thought. How does this play out in practice? Do we approach reading J > programs differently from those written in other languages? If so, how?
I think this is a fantastic question.
I completely agree that it's very easy to write unreadable programs in J. Some people have pushed back that it's easy to write unreadable programs in any language; but I would actually counter that it's easier in APL derivatives.
But I would contend that this is actually a consequence of APL derivatives being developed as a tool of thought rather than a tool of communication (or a tool of command).
Thinking is hard. Communicating is also challenging, but it's not the same as thinking.
I would like to point to the general teaching that APL people give for reading APL code -- what they say (I don't have a link) is that you can best understand APL code by rewriting it, and then comparing what you write with what was originally written. In other words, you learn to THINK about what the author was thinking about, and then you try to understand the WAY the author was thinking.
This reminds me of Chuck Moore's approach to program design, which he called "Thoughtful Programming". He advocated using many prototypes to cull out decent and unacceptable designs.
My brain is refusing to give me the name of the guy who's developing a thinking system using an APL derivative.
-Wm ---
from: [[User:Raul Miller|Raul Miller]] <rauldmiller@gmail.com> date: Mon, Jan 13, 2014 at 5:06 PM
I'll counter your suggestion that it's easier to write unreadable code in APL derivatives with an observation that looks to me like a social issue rather than anything intrinsic in the language.
I say this because with minimal training (one class in APL at a community college, and occasional use in other classes, like biology), I was able to debug and improve other people's APL code in a large codebase.
From my point of view the thing that has been holding back APL is that it is too valuable and too productive. Consider, for example, the impact of Arthur Whitney on wall street. Things like that tend to drive up the price of the implementations which makes business people nervous. And when a business tries to switch away, and fails? That makes them even more nervous.
Meanwhile, most schools do not teach APL. If there were a large supply of programmers, the above issues would not be such a problem. (Instead, we'd have the problem of lots of code much of which would not address most people's needs, sometimes colloquially called "bad code" - popular languages suffer this issue and people mistakenly attribute that kind of problem to the language, also.)
The problem with APL is mostly a lack of source material, for people who might be interested in using the language. This leads to people being intimidated and also leads to a lack of implementations.
So that probably means that admirable books and examples would help.
Thanks,
Raul ---
from: Joe Bogner <joebogner@gmail.com> date: Mon, Jan 13, 2014 at 8:32 PM
Great question!
> So, how would you write an configuration file parser in J, if clarity were > an important concern?
I find it helpful to identify the audience when writing - code or non-code. I then try to write for the audience. If there won't be an audience other than a computer and I will never read it again, then I'm not so worried about clarity. Otherwise, it's a real concern. My audience typically includes a lot of imperative/procedural programmers. I don't know if that's completely due to experience or how brains are actually wired for some people. I just know that's what I learned growing up. It's stuck with me for 20 years. I think of myself as a logical, "step by step" thinker.
As such, I would probably write it in a style that scans the lines and acts upon the lines. I think the PicoLisp example is close to how I would write it. As an aside, I have a few years of experience with PicoLisp
(de rdConf (File)
(pipe (in File (while (echo "#" ";") (till "^J")))
(while (read)
(skip)
(set @ (or (line T) T)) ) ) )
Or I would get clever and translate the config file into something that can be evaled in the native language
If I compare that to the your J implementation
> deb L:0@:(({.~ ; [: < [: ;^:(1=#) ',' cut (}.~>:)) i.&1@:e.&' =')&>@(#~
> a:&~: > ';#'e.~{.&>)@:(dlb&.>)@:(LF&cut)
This J implementation feels more like code golf or a compressed string. How many tokens/operations are included in it? I won't count, but I am fairly sure it's more than the (pipe, in, while, echo, till, read, skip, set, or, line) 9 in the PicoLisp example.
When reading a long J string or an entry in the obfuscated C code contest, I try to recognize patterns or operations. Having used J for about 6 months, I can recognize probably about half the operations in that string without having to look them up. That's progress. It still feels like a "run on sentence" which is harder to read than short sentences.
> and it's going to take more work to > make it readable than it took to make it actually work".
That's normally true for any type of writing. The difference between a first draft and final version is a fair amount of work. Taking a stream of consciousness and turning it into something other people understand takes some effort.
I think there's a fine balance between tacit expressions and clarity. It may be my level of inexperience with the language. However, I wonder if I've put as much time on it as any intro-level APL programmer. Are there any conventions in the language for # of tokens, trains, etc for a readable sentence? There might be some relation to phone numbers and the Magical Number Seven, Plus or Minus Two [1] of the number of objects a brain can hold in working memory. That J expression exceeds it for me as my brain tries to parse it
[1] - http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two ---
[[User:Raul Miller|Raul Miller]] via forums.jsoftware.com 10:28 PM
On Mon, Jan 13, 2014 at 8:32 PM, Joe Bogner <joebogner@gmail.com> wrote:
> PicoLisp > > (de rdConf (File) > (pipe (in File (while (echo "#" ";") (till "^J"))) > (while (read) > (skip) > (set @ (or (line T) T)) ) ) )
Is that complete?
I learned lisp back in highschool, and I've used drracket and emacs and other such lisp environments, but never learned picolisp. I tried to install picolisp but it would not build for me and I do not feel like debugging the source for picolisp just for this message.
My impression, though, is that a J implementation like what you have written would look something like this:
conf=: a:-.~(#~ 1 -.&e. '#;'&e.S:0)<;._2 fread file
In other words, read the file as lines, removing blank lines and comment lines.
If all you are doing is saving the unparsed lines then we should expect simpler code. But maybe I have missed a subtlety of picolisp?
I get that @ is a wild card, but I do not understand the mechanism well enough to say whether your implementation is correct, nor do I know whether (while (read) .. is stashing the read result somewhere or what. Nor do I know if your skip is assuming a rigid file structure or is allowing free-form comments in the config file.
> If I compare that to the your J implementation
>
>> deb L:0@:(({.~ ; [: < [: ;^:(1=#) ',' cut (}.~>:)) i.&1@:e.&' =')&>@(#~
>> a:&~: > ';#'e.~{.&>)@:(dlb&.>)@:(LF&cut)
>
> This J implementation feels more like code golf or a compressed
> string. How many tokens/operations are included in it? I won't count,
> but I am fairly sure it's more than the (pipe, in, while, echo, till,
> read, skip, set, or, line) 9 in the PicoLisp example.
I count 64 tokens in the J implementation and 54 tokens in your PicoLisp example. I'm not sure why you have implied that parentheses are not tokens but I do not think they qualify as whitespace?
We could get further into parsing and punctuation issues, but I'm not sure whether that would be relevant.
> When reading a long J string or an entry in the obfuscated C code > contest, I try to recognize patterns or operations. Having used J for > about 6 months, I can recognize probably about half the operations in > that string without having to look them up. That's progress. It still > feels like a "run on sentence" which is harder to read than short > sentences.
I also usually prefer shorter sentences. Not always, but I'd probably try to split Dan's code into two or three lines. Posting a fair bit of code to email has been an influence there.
I imagine I would also favor a shorter implementation than what Dan has done here. For example, in his code I see the phrase e.&' =' but I see no equals signs in the config file nor in the specification on the companion task (whose J entry should perhaps be simplified?).
> I think there's a fine balance between tacit expressions and clarity. > It may be my level of inexperience with the language. However, I > wonder if I've put as much time on it as any intro-level APL > programmer. Are there any conventions in the language for # of tokens, > trains, etc for a readable sentence?
Personal taste?
Thanks,
Raul ---
Joe Bogner joebogner@gmail.com via forums.jsoftware.com 11:18 PM
Yes, it is complete. I didn't write it or test it as it was already posted to rosettacode. I will explain how it works assuming there is interest
It uses some uncommon tricks. It leverages the read function which is the same function used in the repl to read input characters. So the goal is to take the file and skip any comments and then pass it on to set the variable with the key and value.
(pipe (in File (while (echo "#" ";") (till "^J")))
Reads the file and echos until it encounters a comment character and then reads til EOL
while (read) > (skip) > (set @ (or (line T) T)) ) ) )
Then read those echoed characters. read gets the first symbol, the key. Skip moves the input stream ahead by the space or nothing. Set assigns the variable @ which is the result from the last operation (read - which is the key) with the value from the rest of the line or T if it is blank (for boolean example in the config)
My brain has been trained to think of parens as whitespace. It didn't start that way. I can see why you may consider them tokens. I was also counting unique function/operation tokens, not characters. The idea being if I only have 4 english words with 3 characters each on a line, that is easier for my brain to parse than 5 operations using 2 ascii symbols that I don't recognize the meaning.
However as my J vocabulary improves it becomes less of an issue. I can parse i. or e. As fast as a function called idx or el?
Line length is still important I think. Also a functional style with splitting up the train may help reusability, comprehension, and may help identify small areas to refactor. Those small topics like "filter out lines starting with a comment character" can get lost to me in a long line of compound operations. Again, some balance and personal perference and familiarity
I became interested in picolisp for its speed, conciseness and expressiveness. Many of the same attributes as J. It is almost always in the shortest solutions on rosettacode too. Happy to help resolve your build issue off the list if you are interested. ---
[[User:Raul Miller|Raul Miller]] via forums.jsoftware.com 12:25 AM }}}
Ok, I think I understand.
The basic issue, here, seems to be that PicoLisp is stream oriented and this is a stream oriented task. No one in the J community has cared enough to build a stream oriented library for J. J has enough to do stream oriented design for academic purposes, but ... Consider xml/sax as an example of how one might approach streams in J – call out to some standardized implementation and instead focus on what is unique to the application.
Meanwhile, for a J programmer, words like [| & @ [ and ]] occupy a role not too different from parenthesis for a Lisp programmer. Parenthesis might seem simple, but in fact there are a fair number of contextual rules that one must learn before really understanding their significance. Do the parenthesis delimit a lambda definition? An argument list? Do they denote a function call? Some other special form? That, I think, is the issue you were focusing on when counting tokens - how many special rules does a person have to understand to parse the code. J has 9 parsing rules, each roughly at the same complexity level as a lisp-like lambda. Explicit contexts add a few more, though that's mostly syntactic sugar.
Meanwhile, J is and is not fast. It can be fast, but only if you design your code using big, homogeneous data structures to represent large data sets.
I'm not sure if I am making sense, so I suppose I should write some code instead.
Here's an implementation of a config file reader which should perform reasonably well:
ChrCls=: '#;';(' ',TAB);LF;a.-.'#; ',TAB,LF
NB. comment, space, line, other
tokens=: (0;(0 10#:10*".;._2]0 :0);<ChrCls)&;:
1.0 0.0 0.0 2.1 NB. 0: skip whitespace (start here)
1.0 1.0 0.0 1.0 NB. 1: comment
3.3 4.3 5.2 2.0 NB. 2: word
3.0 3.0 5.1 3.0 NB. 3: comment after word
3.0 4.0 5.1 2.1 NB. 4: space after word
1.3 0.3 0.3 2.1 NB. 5: line end after word
)
readConf=: ({.,<@(;:inv)@}.);._2@tokens@fread
This uses a state machine to deal with all the low level character munging and then forms the result into a two column table where the first column is the name and the remainder is whatever followed that (with redundant whitespace removed).
Is it readable? Not if you are not familiar with the language. In fact, this took me about half an hour to write. And I would not bother doing something like this, normally, unless performance really mattered (which implies large file size). But, if performance does matter, this approach should behave reasonably well and (at least for J) should have much better performance than an implementation which loops or otherwise uses separate primitives for separate states.
That said, I should also note that the idea of using decimal fractions for state machine instructions was Ken Iverson's. I'll not go into why sequential machine was not implemented that way, because I feel guilty about it.
Thanks,
Raul ---
Don Kelly dhky@shaw.ca via forums.jsoftware.com 1:39 AM
I have had considerable experience with APL
What I have found is that it is important, more than with many other languages, to document the code. With J it is even more important. This is a consequence of the power involved in a statement combined with what the hell did I mean when I wrote it..
Other languages are more concerned with the"computer science structure" than the problem that is being approached (hence APL and J are overall considered as "lesser",.
When I think of a problem, I look at what is wanted and what is known and the tools to get to the former from the latter. I don't give a damn in defining something as integer vs floating point etc. If a number is close to integer- treat it as such, APL and J do this. and the array processing capabilities help with dealing with the problem of interest. An example. is +/ (somelist_ of_ numbers)_
The downside is that one can become overly focused on tight tacit programs and it might be better to have a documented explicit program filed somewhere, even in the same script file, that spells out the reasoning behind the tacit version.
Reading J is different from other languages because it removes much of the overhead from the tiddly details that can be handled more efficiently in the background by the idiot box.
Don ---
Roger Hui via forums.jsoftware.com 2:29 AM
I find that it helps to describe the J or APL code as if you are writing a paper about it for the expert. For example, see the essays in the J wiki. If carried to the extreme, it becomes the Literate Programming<https://en.wikipedia.org/wiki/Literate_programming>of Knuth.
In such writing it is unnecessary to describe the working of primitives because they can be looked up in the dictionary or reference manual. ---
Joe Bogner joebogner@gmail.com via forums.jsoftware.com 7:02 AM
I was curious to do some basic benchmarking. Posting here if others are interested. Not sure if this should be moved to another thread. If someone replies please do what is appropriate (same thread or new one)
https://gist.github.com/joebo/61e0841fbf511e1aab4d
… Speed probably isn't important in a typical implementation of this task. I'm not posting the PicoLisp code or timings as a competition but just as another reference point. ---
Ian Clark via forums.jsoftware.com 9:40 AM
The problem of human vs computer readability resurfaced for me recently when planning a J paper for MagPi, the how-to journal for the Raspberry Pi.
It took no imagination to predict the response of the average (computer-literate) reader on seeing J code for the first time. I hoped to forestall it with a ref to:
"The alleged unreadability of J - and what to do about it" http://www.jsoftware.com/jwiki/Essays/unreadability
which essentially covers the ground of this topic for the "educated layman".
Problem not solved, however. I belatedly realise that the article's reading age needs to be (techie) 14-18 for MagPi, whereas the aforementioned essay has a reading age of 50+ (and maybe 70+ :-\ ) ---
Don Guinn via forums.jsoftware.com 9:47 AM
It's always been a mystery to me why it is OK to spend several hours (or sometimes days) analyzing several pages of FORTRAN or C but when reading a few lines of APL or J which do the same thing I must grasp it in a few minutes or I start feeling overwhelmed. But I have written similar "run-ons". Why? Because I can set up test data and add a little at a time to a line or a few lines, executing it and looking at the results as I go.
I have to force myself to break that monster up into more readable chunks. I can't do that in other languages as I have to compile or whatever, So I tend to write all the code then start debugging.
Then comes documenting. I put a brief description of what it's for and expected arguments. Then add references and why the code does what it does. I try not to repeat describing what the code does. But then I end out with comments many time larger than the code. That just seems weird! ---
[[User:Raul Miller|Raul Miller]] via forums.jsoftware.com 9:48 AM
It might be interesting to try it on a large file.
Here's another state machine implementation that might perform better:
StateMachine=: 2 :0
(m;(0 10#:10*".;._2]0 :0);<n)&;:
)
CleanChrs=: '#;';(' ',TAB);LF;a.-.'#; ',TAB,LF
NB. comment, space, line, other
clean=: 1 StateMachine CleanChrs
1.0 0.0 0.0 2.1 NB. 0: skip whitespace (start here)
1.0 1.0 0.0 1.0 NB. 1: comment
3.3 4.0 6.0 2.0 NB. 2: word
3.0 3.0 6.1 3.0 NB. 3: comment after word
3.3 5.3 6.0 2.0 NB. 4: first space after word
3.0 5.0 6.1 2.1 NB. 5: extra space after word
1.3 0.3 0.3 2.0 NB. 6: line end after word
)
NB. .0 continue, .1 start, .3 end
SplitChrs=: (' ',TAB);a.-.' ',TAB
NB. space, other
split=: 0 StateMachine SplitChrs
0.6 1.1 NB. start here
2.3 1.0 NB. other (first word)
0.6 3.1 NB. first space
3.0 3.0 NB. rest
)
NB. .6 error
readConf=: split;._2@clean@fread
I think the performance problem you observed is because the first version started boxing too early. Here, I save boxing till the end, and create fewer boxes, both of which should reduce overhead.
Thanks,
Raul ---
[[User:Raul Miller|Raul Miller]] via forums.jsoftware.com 10:01 AM
I have also struggled with documentation, and not only in the context of apl and/or j.
I sometimes wonder, though, how important it really is. So much of the skill of computer programming comes through seeing the code through experimentation and seeing both the code and its variants in action.
Quite often, I find that the code makes a lot more sense when I see what it is doing. (And, all too often, that winds up being "nothing useful" so then I wonder if there are other cases where it would be useful.)
Anyways, writing documentation is a mentally and socially intensive task, and I have the utmost respect for people that can do it well. And good documentation gives valuable perspectives and insight into the underlying code. But... it's a struggle for me.
Thanks,
Raul ---
jph.butler@mailoo.org via forums.jsoftware.com 10:33 AM
The difficult part in my experience is describing the structure and contents of my inputs and outputs.
Usually, I provide sample datasets, and show what verbs to run on them. But I don't have too many readers of my code so I am not sure how practical that really is.
I was wondering whether naming the main data structures encountered would be useful?
Philip
Learning, teaching and promoting J
Materials
- File:7 Languages in 7 Months-Talk on J.pdf
- File:Coding Kids.pdf
- File:Example of Algebraic-like Simplification of J Code.pdf
- File:J as a Tool of Thought-Code Clarity.pdf
- File:JQTLayout.pdf
- File:Paul Jacksons NGN.pdf
-- Devon McCormick <<DateTime(2014-04-15T02:01:43-0200)>>