NYCJUG/2011-05-10

NuVoc, dictionary, test cases, OLE, Excel, Android, XML, old wine in new bottles, Glade, GTK

Location:: the Heartland, downstairs

Agenda

             Meeting Agenda for NYCJUG 20110510
             ----------------------------------

1. Beginner's Regatta: basic introductory material - see "AccessibleDictionary.pdf".


2. Show-and-tell: adventures with J driving Excel via OLE - see "OLEToExcelFromJ-SomeLessons.pdf".

Experimenting with algorithms - see "PermutationsInJ.pdf"; compare and contrast
to "PermutationGeneratorInJavascript.pdf" and "JavaPermutationGenerator.pdf".


3. Advanced topics: re-visiting old code to find simpler ways to do something
in a more standard way: see "ExerciseInFindingStandardWaysToDoThingsInJ.pdf".

Beginning to use GTK: see "GTKwithGladeIntro.pdf" and "ExamplesOfUIEncodedInXML.pdf"


4. Learning, teaching and promoting J, et al.: John's new pedagogic material.

Some musings on why J is so hard to learn - see "ProblemsWithUsingAPLandJ.pdf".

Examples of the sorts of introductions we could use in J, especially with
GTK - see "GraphicExamplesToEmulate.pdf".

Proceedings

We discussed efforts to present J in more accessible ways and the difficulty novices have with its terseness. We looked at some code for accessing Excel from J using OLE and some of the problems this presents. The examples in our discussion on permutation algorithms highlighted the utility of designing algorithms in J; we also looked at how to integrate test cases with code development.

The more advanced topics started off with how a re-visit to old code whereby we uncovered existing J facilities that addressed the problem much more nicely. We then looked at how Glade is used to produce layout for GTK by means of XML files. This is an increasingly common way to handle the expression of layouts - we saw an example from the world of Android development as well.

Beginner's Regatta

We perused discussion in the J GoogleGroup on Template:NYCJUG/2011-05-10/AccessibleDictionary, under the heading "making programs the easy way", in which Gosi opines the following

On the webpage http://www.jsoftware.com/jwiki/ControlNuVoc there is a start of alternative vocabulary. It is basically open for anyone to contribute.

The best thing is if non-technical people will also send in texts. They can send it here or to someone who passes the text on.

It is very hard for someone who has been around J for a long time to remember what the original most difficult barriers were and what is difficult. Some of us tend to send in so advanced stuff that it scares off those are thinking about simple initial attempts.

Ian Clark responds with a proposal for an alternate vocabulary page called NuVoc.

On the webpage http://www.jsoftware.com/jwiki/ControlNuVoc there is a start of alternative vocabulary. It is basically open for anyone to contribute. The best thing is if non-technical people will also send in texts. They can send it here or to someone who passes the text on. It is very hard for someone who has been around J for a long time to remember what the original most difficult barriers were and what is difficult. Some of us tend to send in so advanced stuff that it scares off those are thinking about simple initial attempts. I am sure all they want to do is show the power of J and how brilliantly it is possible to solve and capture in a line or to what the beginner is using many pages of code to accomplish. I guess we all love the challenge to solve complex issues but it is also a challenge to explain the initial steps to a beginner. I have sometimes thought about taking one item at a time from the vocabulary and explain it but I more or less always find it so trivial that I stop. Finding instances that might help others solve an issue is what we need.

Ian Clark, who has been driving this effort, seconds this

Ian Clark < earthspotty@gmail.com > Apr 27 06:16PM +0100 ^

. Thanks for reminding people about this, Björn. I thought I was going to have to write the rest of it myself :-) "By popular request" -- it's a lowbrow version of Voc: http://www.jsoftware.com/help/dictionary/vocabul.htm . And it's got a name: "The Accessible Dictionary" (aka NuVoc). The proper entry-portal for end-users will be: http://www.jsoftware.com/jwiki/NuVoc ...but as-yet most of the pages are stubs. They are laced-up with all the basic links, and just need content. Actually it's not all bad: many of the stubs need only slight variations on existing pages. Some of the trickiest primitives have been done. It's grunt-work, mainly... for all those primitives like Negate / Minus (-) which are rarely going to be accessed anyway. So it's worth a look right now. Yes, you're right, anyone can contribute. Provided you ask Chris Burke for permission to edit the wiki. Then study some finished pages to get an idea of the format. See the "control page": http://www.jsoftware.com/jwiki/ControlNuVoc -which is where you can take charge of a page, and release it when you've done. Or you can email the example to me, and I'll stick it in. Actually there are no "finished" pages: simple examples will always be welcome in the section(s) "Common Uses". You can supply a pet use without having to do the whole page. Each page will have only the barest description of what the primitive does, with very simple examples. For a precise, if highly-compressed, definition there's still the original Vocabulary, which has a link on each page under the heading "See Also". . Thus, in: http://www.jsoftware.com/jwiki/Vocabulary/gtco you'll see: Entry in the J Dictionary for >: People regularly appeal on the forums for something like this, not aware that it's underway.

We also looked at the little bit of NuVoc that I've worked on, those based on the "minus" sign (as befits my naturally contrarian attitude): . I chose these because I wanted to address - a little bit - why J uses underscore to indicate negative numbers. I was a bit surprised that Zach, who has more experience with non-J languages, was very enthusiastic about this difference of J from other languages. Of course, he appreciates the deep structure and consistency of J.

I also like this set of J verbs because of the common experience I've had in which I'll compare numeric arrays for equivalence, e.g.

   A -: B
0

And, on finding that they differ, my next step often starts with something like this

   A - B

to see where they differ and by how much. Similarly, when comparing boxed vectors of text strings, say lists of filenames, I'll often use this expression when I'm expecting them to be the same:

   fls0 -: fls1
0

Again, when they differ, I'll then proceed to see where they differ by using another verb based on the minus sign, the one I call "without":

   fls0 -. fls1
+-----------+---------------+
|NewFile.txt|AnotherNew1.xyz|
+-----------+---------------+

Show-and-tell

We looked at some adventures in using OLE with Excel and then at developing a permutation generator algorithm in J with the idea of implementing it in Java. Here is File:PermuteExperiments.ijs

OLE to Excel: Warts and All

I recounted my recent experiences calling macros in an Excel spreadsheet from J using OLE. I was having some problems so, doing what I normally do, I first searched the J wiki to see what others had done with OLE and Excel. Unfortunately, the most extensive piece I uncovered was one I had written myself a few years ago, so I guess that makes me the reigning expert on the subject.

We looked at some of what I had written then (and am updating currently based on my recent experience) and I pointed out a few problems I'd had with what was there. One frustrating waste of my time was spending time figuring out how to adapt "writeLine2WB" to write multiple lines to a workbook, only to eventually figure out that the existing function already did want I wanted - it was just named misleadingly and was insufficiently commented. It's now named "writeLines2WB" - a difference of one small character that could have saved me an hour or so.

The major problem I had, and still have to some extent, is that I'm calling a macro that might run for several minutes. Unfortunately, in an example of the excruciating, mis-placed over-helpfulness of some software, OLE pops up a dialog box to tell me that the last call I made is taking a long time and I should look at the application and do something about this. This wrecks any chance of fully automating my process, which was the whole point of what I'm attempting to do.

The annoying "Switch To" or "Retry" box is shown here:

There seems to be a way to disable this interruption in COM-based environments but not in plain OLE. However, one trick I found to minimize this interruption is to spin off the J process with no session window. This does lead to another complaint from Windows but it seems to be much less frequent if not absent many times.

The current OLE/Excel essay has a section on the "wartiness" of Excel and recent experience indicates that there will be more examples for this section. A recent kluge is shown in the following lines from "writeLines2WB": the simple line which formerly was like this:

   xlid 'ws' [ xlget 'temp worksheets'  NB. Set worksheets name->"ws"

is now accomplished by these two lines:

    try. xlid 'ws' [ xlget 'temp worksheets'          NB. Set and name worksheet "ws"
    catch. xlid 'ws' [ xlget 'base worksheets' end.   NB. Fix mystery fail.

This is because the former line would fail mysteriously and sporadically, requiring this kluge to get it to work more reliably.

Developing a Permutation Algorithm in J

Recently, I set about to learn how to program in the Android environment. For those of you who have spent the last few years on a desert island, "Android" is "a software stack for mobile devices that includes an operating system, middleware and key applications" - that is, it's an immensely popular platform for the "smart phones" now taking the world by storm. Its programming environment is mainly Java. In spite of this limitation, the associated IDE and emulator environment can provide us with some good lessons.

The most advanced application I've written so far is from the "Hack-a-day" website. This app allows us to enter quotations into the SQLite DB provided with Android, then to press a button to show us a random quotation. Once I'd gotten this to work and entered about a dozen quotations, I noticed that the "randomness" of the retrieval was not as well-behaved as one might want. Typical of other cheap implementations of this random shuffle idea, one would see the same quote twice in a row or within a few iterations.

It occurred to me that a good modification to this app would be to avoid repetition as long as possible by generating a permutation vector of the quotation keys and simply access them in order. I did a little research on what people have written about permutation in Java and discovered any amount of elaborate code. Also, most of these exercises aimed to produce a full set of permutations, not a single random one which is what I wanted. The Java example here seems like reasonably good Java code. Note that it warns against using it for very large numbers (i.e. much greater than, say, ten) as the number of permutations will be intractably large.

Actually, this fundamental limitation should be a good hint not to do it this way. We may be thankful that Roger provides us with the handy anagram verb A. to generate specific permutations, as well as the verb "permute" { to generate all permutations for reasonably small sets. However, neither of these help me implement the special case in Java - as powerful as they are, they are J primitives.

So, I experimented with a simple algorithm - to permute an index vector - that I could easily translate into Java. This means I will loop unnecessarily and use the random number generator with an argument of zero to simulate the more primitive ones available in other languages. There were a couple of interesting lessons that came out of this exercise, the most useful of which was the development of my test cases alongside and in advance of the nascent algorithm.

The details of these experiments can be found here.

Advanced topics

We looked at the value of re-vamping old code. On the topic of graphics and GTK, we looked at using XML to specify graphic layout formats and how the Glade tool provides a GUI for window layout and produces such an output.

Finding a Standard Way to Do Something in J

I had this routine, with a couple of the associated sub-routines, written years ago:

commaFmtNum=: 3 : 0
NB.* commaFmtNum: put commas in a long number: '12345678.91'->'12,345,678.91'.
   0.01 commaFmtNum y
:
   if. x=0 do. x=. 1 end.               NB. Make suitable larg to roundNums
   ndpd=. >.10^.%x                      NB. Num Digits Past Decimal
   fmtstr=. j. ndpd                     NB. Format to num digits past decimal.
   if. isNum y do. y=. ":y end.         NB. Start with char vec.
   y=. <;._1 ' ',dsp y                  NB. Handle multiple numbers in string.
   nn=. fmtstr&":&.>(x&roundNums@:".)&.>y    NB. Rounded char vec
   dc=. 2{.|:><;._1&.>'.',&.>nn         NB. Separate before from after decimal
   dc=. (inscomma&.>0{dc) 0}dc
   dc=. (('.'#~&.>*#&>1{dc),&.>1{dc)1}dc     NB. take care of any decimal
   }:;' ',~&.>,&.>/dc                        NB. Space-separated numbers
NB.EG    commaFmtNum 1234 123456.78 12.1 111222333
NB. 1,234.00 123,456.78 12.10 111,222,330.00
NB.EG    0 commaFmtNum 1234 123456.78 12.1 111222333
NB. 1,234 123,457 12 111,222,330
)

NB.* inscomma: insert commas every 3 digits from end.
inscomma=: 3 : 0
   3 inscomma y
:
   }:;([:|.','&,)&.>(-x)<\&.|."_ y
)

NB.* isNum: 1 if arg is some kind of numeric array.
isNum=: 3 : '1 4 8 16 64 128 1024 4096 8192 16384 e.~ 3!:0 y'

It’s been so long since I used this, I searched my code and the J site for “format number”, and stumbled on something I knew about but had not used in a while: the format verb 8!:. This allowed me to shorten the three routines above to this single one (shown here sans documentation and example comments):

commaFmtNum=: 3 : 0
   0.01 commaFmtNum y
:
   }.;(' ',' '-.~])&.>('c0.',":>.10^.%x) 8!:0 y
)

Glade, GTK, and XML to Represent Graphical Layouts

We looked at File:GTKwithGladeIntro.pdf, mostly just because it's well done. I especially liked the clear diagram explaining each part of the sample widget constructed.

Glade provides a GUI interface for designing graphical elements. It produces an XML specification embodying the element designed.

Learning, Teaching, Promoting J

John is working on introductory material for J. One of his motivations is to overcome the notion that learning J is simply worthwhile for its own sake. We need to provide better motivation than this.