NYCJUG/2007-02-13
NYCJUG meeting, 64-bit J, installing applications, parallelism, name shadowing, maths challenge, invoking J
NYC JUG Meeting of 20070213
In this meeting, we covered topics from basic beginner resources, and what we can do to improve them, to J on 64-bit and parallel architectures, as well as a little about the suitability of J for math challenges.
Meeting Agenda for NYC JUG 20070213
1. Beginner's regatta: some resources for beginners: the JWiki FAQ (see
"Guides_Language FAQ - J Wiki"), introductory parts of the dictionary
(see "I. Alphabet and Words", "II. Grammar", and "III. Definitions").
2. Show-and-tell: 64-bit J on Linux installation?
Running scripts from desktop: problems with invocation (see
"invokingCodeFromDesktop").
Debugging a name shadow (see "Name-ShadowDebugging").
3. Advanced topics: parallelism, multi-core CPUs (see "Teraflop chip");
current use of multi-core parallelism.
4. Learning and teaching J: Maths Challenge (see "APL-J-K are Top of the
League", and "Project Euler-APL-J-K Standings").
+.--------------------+.
Ramen noodles have earned Mr. [Momofuku] Ando
an eternal place in the pantheon of human progress.
Teach a man to fish, and you feed him for a lifetime.
Give him ramen noodles, and you don’t have to teach him anything.
- Lawrence Downes, obituary of Momofuku Ando, ramen inventor
Proceedings
In order of the agenda, not necessarily the order of our discussion:
Beginner's regatta: Existing Introductory Materials
To follow up on some points raised at the previous month's meeting, we had a look at some of the standard J introductory material included in the system. This material, accessible by backing up from the vocabulary, establishes the conventions used in subsequent documentation.
Starting with the most basic definitions of the alphabet and words (see [1]), we proceed to grammar ([2]), which details the parts of speech: nouns, verbs, adverbs, conjunctions, punctuation, and copula. Subsequent sub-sections of grammar also provide an overview of comparatives, parsing, verb trains, scripts, locatives, and errors. More to the point of last month's questions, the Definitions section ([3]) explains the conventions of the dictionary entries.
We agreed that, while the main vocabulary page is a good starting point for an experienced user, a beginner needs to work up to using this page and the links to which it points. The afore-mentioned explanatory material perhaps needs more emphasis to neophytes.
One other page we looked at briefly is the FAQ on the wiki. This is something that the group feels we can contribute to as this sort of thing has been part of our "Beginner's Regatta" discussion for some time now. One suggestion was to take a look at the FAQs of other languages to see what beginners are apt to ask. Another idea is to review the newbie questions on the forum to see what should be addressed to best help the initiate.
Show-and-tell: 64-bit J
John described his difficulties getting 64-bit J to run on a Linux installation. He had a machine with a 64-bit Intel chip but not a 64-bit version of a Microsoft OS, so he installed Red Hat's Fedora - a 64-bit Linux. He said one of his major stumbiling blocks was that the 64-bit browser did not work with all 32-bit plug-ins but that he had no problems with 32-bit applications in general until he got to J. He posted some messages to the Jgeneral Forum and got enough help to get things running as he says in JForum:general/2007-February/029018. Part of the help involved fixes to J which will be released in version 6.02.
Installing J Applications
I sought help with problems I've been having running some J applications I invoke from icons on my (MS XP) desktop. They had worked fine until I made some changes coincident with J 6.01 involving running my own, customized profile script. See "Problems Invoking Code from Desktop" for more details on these problems.
The more interesting discussion this led to was about difficulties people seem to be having installing their J applications in a standard fashion. Dan made a good argument that re-inforced the J Software party line: just install J first. This is really no different from a user's point of view than installing some package that goes into "Program Files" on Windows. Though J has historically wanted to install itself at the root, I've been successfully running it from this standard Windows installation area for many months now. Some of the problems I mentioned in the preceding paragraphs may be related to customizations I've done for my own version of this environment but there's no reason to think this is any longer a problem with the J installation.
John averred that the Linux installation for J still has some rough edges, that it needs to be moved from the root directory. In general, none of us are using the J package manager which may address some of our problems but that's our own fault. It may be time to embrace that part of the package.
Another issue related to this is the desideratum of installing an application written in J so that it looks like a single executable to a user. Again, Dan argued that most other applications require multiple files as well but, at the end, the user is presented with something like a single icon on which to click or a simple, top-level script to run. So, from their point of view, this should not really matter.
In fact, the J install is no worse than any other application which requires a DLL or two. Even today, I find it annoying when uninstalling an application that the uninstall procedure sometimes tells you that removing a particular DLL may break other apps, but can't tell you if something else definitely uses the DLL in question. At least with J, you know nothing else will be using its DLLs.
One of these issues came up again recently after the meeting:
from Geoff Canyon <geoffc@inspiredlogic.com> Mar 24 to Programming forum <programming@jsoftware.com> date Mar 24, 2007 2:30 PM subject [Jprogramming] Application Delivery? Is there a way to bundle a J application into a single-file executable? If not, how close is it possible to come? regards, Geoff
An initial reply seemed to indicate that this is possible:
from Tracy Harms <t_b_harms@yahoo.com> Mar 24 to programming@jsoftware.com date Mar 24, 2007 6:20 PM subject [Jprogramming] re: Application Delivery? ... Yes. Details are available at the following wiki page: http://www.jsoftware.com/jwiki/Guides/Standalone_Applications
However, on examination of the reference, it is aimed more at the basics of setting up a "stand-alone" J application as the title of the guide suggests.
The ensuing discussion did raise some concerns and desires that people have about installing a J application so that the installation appears the same as for any other software. The full discussion can be found in the mail archives of the J Programming Forum.
In brief, it looks as though Lorin Lund has had success using an installation package called the Inno Setup Compiler. He gives the example of a sample of the configuration file for this tool he uses to pack up a J script with necessary files. See his post on the Programming Forum ( JForum:programming/2007-March/005742 ) for March 25, 2007 at 12:29 pm. You can look at the thread under its subject Application Delivery? to see the initial message ( JForum:programming/2007-March/005724 ) and replies to it.
Also, Lorin gives a link to a list of other "setup creators" that may be of interest.
Name Shadowing
One problem the forum recently helped me solve looked initially like some kind of installation oddity. I posted the following message on the Jprogramming Forum:
from Devon McCormick <devonmcc@gmail.com> Feb 7 reply-to devon@acm.org to J-programming forum <programming@jsoftware.com> date Feb 7, 2007 1:24 PM subject Using regexes in name list searches Did simple regular expressions used to work better with "names" in previous versions of J? I think that if I entered something like 'is*' names 3 I used to get the names starting with "is". Now I get a much larger list that looks like all names starting with either "i" or "s". Has this changed or am I just misremembering it? In any case, how do I get a list of the verbs starting with "is" now?
Dan Bron helped me debug this by tracking down where I had inadvertently shadowed the name of a standard function with my own definition. Here's the "Eureka" response:
from Devon McCormick <devonmcc@gmail.com> Feb 7
reply-to devon@acm.org
to Programming forum <programming@jsoftware.com>
date Feb 7, 2007 6:39 PM
subject Re: [Jprogramming] Using regexes in name list searches
OK - I found it. It's in the WS.ijs code I put up on the Wiki a
couple of months ago.
Thanks for your help. Now I just have to make sure my WS code
works with that line commented out.
On 2/7/07, Dan Bron <jf@dbron.net> wrote:
> Devon,
>
> My definition came from the standard library, ~system\main\stdlib.ijs .
> Run this line in your J sesion to figure out where your definition came from:
>
> (4!:4{.;:'nl') {:: 4!:3''
>
> This is ringing a vague bell. I suspect the fact that you use jconsole
> (via Emacs) instead of the J IDE is relevant. Or maybe there's some J
> script that inadvertantly redefines nl , like load'trace' used to
> redefine noun verb adverb conjuction etc. My grepping didn't turn
> up any candidates, though.
...
The key was the expression (4!:4{.;:'nl') {:: 4!:3'' to search for the definition of nl. This uncovered where I shadowed the standard definition. This is a handy thing to know about.
Advanced topics: Taking Advantage of Parallel Hardware
We talked for a while about taking advantage of the inherent parallelism of J in making use of the current trend toward multi-core CPUs (See "Teraflop Chip" below). This was in response to a thread on the J Forum by Skip Cave about how J might help overcome the programming difficulties posed by this new architecture.
The consensus at our meeting was that parallelism at a multi-core level was probably too coarsely-grained to mesh well with J's natural fine-grained, array parallelism. However, this is not to say that the interpreter implementers might find a way to take advantage of this. However, even now, an application programmer in J or any other language can use multiple processing cores to some advantage.
I mentioned how my work on the Netflix challenge has accidently allowed me to take advantage of my dual-core machine. This is because the large amount of data has forced me to break it up and the way in which I did this, using an adverb to apply verbs to functions saved to file using my WS facilities, naturally lends itself to breaking up a problem in sufficiently large pieces that I can use my two cores in parallel.
As alternatives to this sort of parallelism, we talked about some recent Forum threads about using the graphic-processing chips' finer parallelism to potentially find a better fit to J's array parallelism. John also mentioned how even thread-based parallelism is too heavy-duty to support fine parallelism but that a socket-based mechanism can currently work well. He told how a socket-based dispatcher that apportions processing can be a lighter, finer method because a single thread can monitor many sockets. However, this still requires application-based work to break up the work and apportion it across different processes.
John pointed out how the success of fast computation with LINPAK is in part due to the way higher-level algorithms are separated from a hardware-oriented computation layer. This allows the hardware-layer (BLAS) to be tuned to a particular processor architecture without disturbing the application code. This approach has a lot going for it, especially in light of some traffic we saw on the forum recently where there was some debate about fine-tuning an algorithm for performance. It turned out that some of the putative improvements only improved throughput for one individual on a particular machine: others were not able to replicate his speed-up on their machines. This points up the dangers of over-optimizing.
Another point we covered regarding fine- versus coarse-parallelism is that memory-allocation times are often the bottleneck in processing large arrays, not computation. There is some hope that the inherent parallelism of a language like J could give an advantage here by allowing us to pre-compute memory movement but this would still fall upon the implementer.
Learning and Teaching J: Maths Challenge
We re-visited a topic we'd talked about at an earlier meeting: the use of J in the "Maths Challenge", also known as Project Euler. An article in the most recent Vector magazine points out that the APL/J/K "language" is the one of choice for top-scoring contestants; see APL/J/K are Top of the League by Graham Parkhouse.
Mr. Parkhouse gives some strategic tips for keeping us at the top of the league for people who choose to enter this. One thing he mentions is that you don't have to note the language in which you are working right away. Waiting until you have solved some problems before declaring your allegiance will avoid bringing down the average for the group as you solve your first few problems.
However, as John pointed out, there may be more danger of the number of APL/J/K'ers dropping below 1%, thus losing out to some lower-scoring but better-represented language. In fact, I recently (April, 2007) checked my rating and found that I'm now in a higher percentile than I was previously. I moved up by about 2% not because I've solved any problems in the past couple of months but the denominator has grown larger - more people have come in under me.
I was inspired by John and also by Mike Day to enter the competition a couple of months ago. I was able to solve about half the problems in about two weeks, not because of any particular brilliance on my part but largely because of the power of J. Since this initial spurt, however, my problem-solving has lagged because of other interests.
Still it's an interesting way to keep your mental faculties tuned up and I urge everyone to take a look at competing.
Also, this sort of challenge is a good selling point for our favorite array languages - the problems in this competition are like the ones many of us encounter every day and are able to quickly solve. The importance of this "total-time-to-solution" cannot by over-emphasized! We need to re-frame the traditional discussion of the relative merits of computer languages to get away from outdated notions of considering only CPU time as this measure grows increasingly irrelevant for most problems every day. See our discussion at another NYCJUG meeting - on 20070410 - on this idea of re-framing the efficiency discussion: [4].
Teraflop Chip
An article on BBC News tells about Intel's new prototype 80-core chip and makes the point that the challenge is to find a way to program the many cores simultaneously.
Problems Invoking Code from Desktop
I have a couple of different desktop icons to invoke J code. Unfortunately, they have been behaving differently recently, probably because of changes I've made to run with a customized profile script.
Initially, I had trouble running both the -jprofile option together with my application script on the command line invoked by the icons. Chris Burke of J Software gave me the following fix.
from Chris Burke <cburke@jsoftware.com> 10/12/06
reply-to Programming forum <programming@jsoftware.com>
to Programming forum <programming@jsoftware.com>
date Oct 12, 2006 3:39 AM
subject Re: [Jprogramming] Cannot invoke my own profile and an application script from same command line
Devon McCormick wrote:
> Is there any simple way to run both a profile and an application script?
Normally, the application script run on start up will be ''startup.ijs''. To
run another script in addition, with a custom profile, add the
following to the end of your custom profile:
load >{.3}.ARGV
The J command line should be something like:
c:\j6\j.exe -jprofile c:\work\myprofile.ijs c:\work\myapp.ijs
I elaborated on his suggestion by putting this at the end of my general utilities script invoked by my startup:
NB. per Chris Burke's suggestion of 20061012 (load the specified script):
3 : 0 ''
NB. if. 3<#ARGV_j_ do. 0!:0 ] {.3}.ARGV_j_ end.
NB. Load 1st element of ARGV (after 3rd) w/".ijs" in it.
if. 3<#ARGV_j_ do. 0!:0 ] {.(]#~[:+./&>(<'.ijs')E.~&.>~]),&.>3}.ARGV_j_ end.
)
The Problem with Two Icons
However, I now have a problem with very similar-looking command lines working differently. Of the two icons, the one labeled "Show Quotes" works properly; however, the one labeled "Get Cross-Street" initially failed to run at all.
Their respective command lines are:
Show Quotes:
"C:\Program Files\j601\j.exe" -jprofile C:\amisc\Jsys\profile.ijs -jijx C:\amisc\txt\showQuotes.ijs
Get Cross-Street:
"C:\Program Files\j601\j.exe" -jprofile C:\amisc\Jsys\profile.ijs -jijx C:\amisc\Jsys\user\code\addressArithmetic.ijs
Additions to my customized profile that differ from the standard one:
Change from standard profile in my "profile.ijs": NB. Use my own temp area.... NB. tmp=. prf,sep,'temp' tmp=. 'C:',sep,'amisc',sep,'JSys',sep,'temp' SYSTEMFOLDERS_j_=: ids ,. prf;sys;add;tmp;usr;cfg;hlp;hom 9!:49]1 NB. Allow "y." and "x." 9!:7 '+++++++++|-' NB. Box-drawing characters
Code for "Show Quotes"
Contents of showQuotes.ijs:
NB.* showQuotes.ijs: show quotes repeatedly.
load 'cmdtool semaphore'
load '~User/code/logger.ijs'
onlyRuntime=: 3 : 0
NB.* onlyRuntime: only invoke if not loaded via interactive session.
if. (5{.&.>'-jijx';<'-rt') +./ . e. 5&{.&.>tolower&.>ARGV_z_ do.
'Y' initFlagFile setFlagFileName 'c:\amisc\txt\quote.flag'
cmd=. '"C:\Program Files\j601\j.exe" -jijx C:\amisc\txt\manipQuotes.ijs'
shell cmd
while. greenLight '' do. ctr=. 100+?100 NB. mean 149.5
while. (greenLight '')*.0<ctr=. <:ctr do.
wait >:?11 end. NB. 15 +or- 3 mins btw
if. greenLight '' do. shell cmd end. NB. quote display
end.
end.
)
3 : 0 ''
try. onlyRuntime ''
catch. DISPLAY_logger_=: 1
logMsg_logger_ ' ---- showQuotes.ijs ----'
'ern erm'=. (13!:11 '');<13!:12 '' NB. Error number, message
logMsg_logger_ '*Error: error number, message: ',(":ern),', ',erm
logMsg_logger_ deepDisclose_logger_ 13!:18 '' NB. Show stack
end.
)
(Partial) Code for "Get Cross-Street"
The "Get Cross-Street" icon invokes the "addressArithmetic.ijs" code which ends like this:
coclass 'base'
onlyRuntimeRun=: 3 : 0
NB.* onlyRuntimeRun: only invoke if not loaded via interactive session.
if. (5{.&.>'-jijx';<'-rt') +./ . e. 5&{.&.>tolower&.>ARGV_z_ do.
sink=. getadd_run_Manhattan_ ''
else. smoutput 'ARGV_z_: ',;ARGV_z_,&.>' ' [ wait 1
end.
)
3 : 0 ''
try. onlyRuntimeRun ''
catch. DISPLAY_logger_=: 1
logMsg_logger_ ' ---- addressArithmetic.ijs ----'
'ern erm'=. (13!:11 '');<13!:12 '' NB. Error number, message
logMsg_logger '*Error: error number, message: ',(":ern),', ',erm
logMsg_logger_ deepDisclose_logger_ 13!:18 '' NB. Show stack
end.
)
I found that changing the beginning of this code from
coinsert 'base' [ coclass 'Manhattan'
to this:
coclass 'Manhattan' coinsert 'base'
invokes the desired window though it also brings up the session (.ijx) window.
Dan says that this funniness with locales, having coinsert on the same line as coclass is a known problem.
In any case, there is now a new problem: the -jijx command-line option which is supposed to suppress the session window works in one case (for "Show Quotes") but fails in the other.
A patch I've come up with is to insert the code
sys_timer_z_=: 3 : 'wd ''timer 0''[wd ''pclose''[wd ''psel '',":qsmout_jijs_ ''''' wd 'timer 1'
in the startup for "Get Cross-Street" immediately prior to the invocation of the main window. This finds the parent session window and kills it. There is a very brief "flash" of the short-lived session window but this works well enough for now. The question remains why two very similar invocations give such different behavior.
Scan of Meeting Notes
