NYCJUG/2007-03-13

From J Wiki
Jump to navigation Jump to search

Overview of J Wiki introductory material, notation FAQ, linear Diophantine equations


NYC JUG Meeting of 20070313

In this meeting, we talked about how we think about how the introductory material on the J Wiki might be better organized and what could be added to provide more motivation for learning J, the idea of a FAQ to address recurring questions on issues of "notation", solving linear Diophantine equations, and improving the performance of a long-running adverb.

Meeting Agenda for NYC JUG 20070313

             Meeting Agenda for NYC JUG 20070313
             -----------------------------------

                    Working with the J Wiki
                    -----------------------
1. Beginner's regatta: overview of intro material on J Wiki -
how might it be better organized?  Where is the motivational material?
See section 4 below.


2. Show-and-tell: in progress on J Wiki: notationFAQ,
linearDiophantineEquations.


3. Advanced topics: improving the performance of an adverb:
from "getVarInfo" to "gavInfo": from 150 hours to 12 hours.

Possible enhancements: how to divide more evenly in 2 dimensions?
Alternately, how to represent subdivisions within existing divisions?


4. Learning and teaching J: what motivational examples can we put
on the J Wiki?  How can we make FAQs more transparent, more geared
toward actual beginner questions?

            +.--------------------+.

Like waves of the sea,
events have two sides:
either you ride them out
or they ride you down.
  - Arabic proverb

Proceedings

We debated the layout of the J Wiki in the "Beginner's regatta", discussed what should be in a J language FAQ specifically in relation to notational "oddities", looked at some work on solving linear Diophantine equations, briefly discussed performance improvement of an adverb used in working on the Netflix Challenge, and discussed what examples we could provide to motivate people to learn J.

Beginner's regatta

The consensus on the J Wiki was that it was bit crowded and not sufficiently visually interesting. The initial page does not make one think "Really? Tell me more!"

We debated the idea of a splash page, common on many sites, but decided this was more annoying than compelling. One example of compelling J some of us like is the cover of Howard Peelle's J book: it's festooned with short J definitions having familiar names like "pascal" and "pythagoras".

Show and Tell

See [[[NYCJUG/notationFAQ]]] for the answer to a commonly asked question about one of J's prominent notational conventions: why is the default order of operation right-to-left?

See [[[NYCJUG/linearDiophantineEquations]]] for the discussion on these equations.

Advanced Topics

I developed the adverb "getVarInfo" to apply an arbitrary function (verb) to a list of variables on file specified by the 99 names in the vector "UVN" in the directory specified by "VDIR". This is for handling a very large array by breaking it into a hundred or so pieces. Instead of applying a function to the array directly, we do so indirectly using this adverb which applies the supplied left function "u" to each of the pieces on file.

So, an expression like dts=: (3&{) getVarInfo&.>(<VDIR);&.>UVN applies 3&{ to get the date row from each matrix on file.

Original Adverb

NB.* getVarInfo: apply arbitrary function to each (filed) var named.
getVarInfo=: 1 : 0
   'dd varnm'=. y.                 NB. Vars dir, var names.
   rc=. dd unfileVar_WS_ varnm     NB. Get var from file
   if. >{.rc do. rc=. 1;u. ".varnm NB. Do something to it
       [4!:55 <varnm               NB. Erase when done to conserve space
   end.
   rc
NB.EG ({."1,.{:"1) getVarInfo &.>(<'C:\data\');&.>'var1';'var2';'var3'
NB.EG dts=: (3&{) getVarInfo&.>(<VDIR);&.>MVN
)

The function "unfileVar_WS_" above is from "WS.ijs" found at [[[Scripts/File J Variables]]]. This function allows us to read and write a J variable from and to a file.

Modified Adverb

The newer version of this adverb takes two verbs instead of one. This allows us to apply the function of interest (see "accumCMRatings" below) to a group of arrays at a time by specifying an appropriate concatenation, ",." in this case, to several variables before applying the function of interest. For some functions, it's much faster to work on larger pieces.

NB.* gavInfo: apply arbitrary fnc "u" to all file-vars joined by fnc "v".
gavInfo=: 2 : 0
   for_dv. y do. 'dd varnm'=. >dv  NB. Vars dir, var names.
       rc=. dd unfileVar_WS_ varnm
       if. >{.rc do.
           if. -.nameExists 'cumvals' do. cumvals=. ".varnm
           else. cumvals=. cumvals v ".varnm end.
           4!:55 <varnm
       end.
   end.
   1;u cumvals
NB.EG (_2 ({."1,.{:"1) gavInfo ,.)\(<'C:\data\');&.>'var1';'var2';'var3';'var4'
)

Timings

The timings below, first for the original adverb "getVarInfo" then for the newer adverb "gavInfo" are mis-leading as presented. This is because the first version was taking so long - it had been running for days and was only about halfway through the files - I came up with the newer version while the older one was still running. I then moved the unprocessed files, 32 of the 99, to an alternate directory and ran the new version on those.

Once both versions finished, I combined the results.

Session Using Original Adverb getVarInfo

   CTMAT=: 0$~CMClassVars ''     NB. Initialize the global we'll be updating
   6!:2 'accumCMRatings getVarInfo&.>(<VDIR);&.>UVN'
409856.93
   0 60 60#:409856.93            NB. Number of seconds as hours, minutes, and seconds
113 50 56.93

Note that this timing was for the first 67 files whereas the following is for the remaining 32 files. A little forethought in the design of this code allowed it to fail gracefully when I pulled the rug out from under the first adverb by removing some of its files after it had started running. Note that this graceful failure, combined with good modularization, also helps for coarse-grained parallelism: I can run multiple instances of this code on distinct sets of files on separate machines or different cores of the same processor.

Now, we finish the job using the newer adverb "gavInfo" which we apply to blocks of 8 variables from file at a time using the scan adverb "\" with a negative number to specify non-overlapping windows. The value "8" happens to divide evenly into the 32 files I had remaining but this doesn't matter for the result. A short block at the end would have been processed just fine.

Separate Session Using Updated Adverb gavInfo

   CTMAT=: 0$~CMClassVars ''         NB. Initialize the global
   VD2=: 'C:\Data\Netflix\AltVDir\'  NB. Specify alternate file variables' directory
   6!:2 '_8 (accumCMRatings gavInfo ,.)\(<VD2);&.>UVN'
14266
   0 60 60#:14266
3 57 46

We see that we needed almost 114 hours for the first 67 files was whereas we processed the final 32 files in less than 4 hours. So, extrapolating to the full set, the first version would have taken about 168 hours versus about 12 for the newer adverb.

Verb and Sub-functions Used with Adverbs

NB.* accumCMRatings: group (cust,movie) ratings by averages partitions.
accumCMRatings=: 3 : 0
NB. VDIR unfileVar_WS_ 'umurd0'
   cn=. classifyCMRatings ptnVar y
   cr=. cn </. 2{y       NB. Customer-movie ratings by CM-class
   cn=. ~.cn             NB. Class number/partition
NB.   ctmat=. (NCC,NMC)$0   NB. Count # ratings/class
   nd=. >:<.10^.NCC*NMC  NB. Max # digits in total class number
   for_fnum. i.#cn do.   NB. Rating info into appropriate CM-class file-var
       vnm=. 'cmclass',(-nd){.(nd$'0'),":fnum{cn
       VDIR unfileVar_WS_ vnm
       (vnm)=: (".vnm),>fnum{cr
       VDIR fileVar_WS_ vnm
       4!:55 <vnm
       ix=. <(NCC,NMC)#:fnum{cn
       CTMAT=: ((#>fnum{cr)+ix{CTMAT) ix}CTMAT
   end.
NB. accumCMRatings getVarInfo&.>(<VDIR);&.>UVN[CTMAT=: 0$~CMClassVars ''
)

NB.* classifyCMRatings: find Cust-Movie class assuming "cbpv" and "mclass".
classifyCMRatings=: 3 : 0
   cc=. <:+/cbpv </ mean&>2{&.>y   NB. Customer class based on avg movie rating
   mc=. mclass{~(0{mvnums) i. ;0{&.>y
   cc=. cc#~(1{$)&>y               NB. Customer class/rating entry
   cn=. mc+NMC*cc                  NB. Class number: (cust, movie)
)

NB.* countBiClass: count # (cust,movie) per bidimensional equi-rating groups.
countBiClass=: 3 : 0
   if. 0=#y do. y=. 10 10 201 end.
   'cnb mnb bsz'=. y          NB. Cust # breakpoints, Movie # brkpts, block sz
   CMClassVars cnb,mnb        NB. Vars: NCC, NMC, cbpv, mbpv, cclass, mclass
   ctmat=. (cnb,mnb)$0
   for_cb. i.>.NCUST%bsz do.
       len=. bsz<.NCUST-cb
       'ct ix'=. <"1 |:frtab ,(mnb*cclass{~cb+i.len)+/mclass
       ix=. <"1](cnb,mnb)#:ix
       ctmat=. (ct+ix{ctmat) ix}ctmat
   end.
   ctmat
)

NB.* findEqualBinBreakpoints: find distinct values to partition vec equally.
findEqualBinBreakpoints=: 3 : 0
NB. avgbp=. /:~avgmr=. %/2{.MMR
   'nbp vals'=. y.
   vals=. /:~vals
   anpb=. nbp%~#vals          NB. Average number per bin
NB. Locate index of 1st instance of breakpoint value
   bpi=. vals i. vals{~<.0.5+anpb*i.>.anpb%~#vals
   bp=. bpi{vals
   bp=. bp#~whUnq bp
   bp;(<./,>./,mean,stddev) 2-~/\bp     NB. Stats on breakpoint differences
NB.EG 'bpv stats'=. findEqualBinBreakpoints 100;%/2{.MUR
)

Scan of Meeting Notes

NYCJUGMeetingNotes070313 0000 30.jpg NYCJUGMeetingNotes070313 0001 50.jpg NYCJUGMeetingNotes070313 0002 50.jpg NYCJUGMeetingNotes070313 0003 50.jpg NYCJUGMeetingNotes070313 0004 50.jpg NYCJUGMeetingNotes070313 0005 50.jpg