User:Devon McCormick/ParallelizedJCodeExamples
Jump to navigation
Jump to search
Examples of Parallel Code in J
This contains complete examples of code presented at the APL2010 conference in Berlin, Germany, in September, 2010. There are two examples of parallel code tasks as well as an implementation of mutex (mutual exclusion), all written in the J programming language (version 6.02). A copy of the J programming environment is freely available from jsoftware.com. When attempting to run this code, be aware that some longer lines may have “wrapped”.
Immediately below is the first task, the original “photo flipping” code. The customizations for my site should be contained in the first few lines of globals’ assignment near the top. There are a number of utility functions here for ancillary uses but the core function is regflipphotos.
Examples of the parallelizing extensions to this code are contained below as well. When this script is loaded, an anonymous noun is displayed by the section of code beginning “smoutput 0 : 0” and ending with the bare “)” at the end of that short paragraph. This displays examples of some of the most commonly used phrases in our J session when we load this script so that frequently-used invocations can simply be selected and run in the session.
So, as typically used, I would select the line
info,<6!:2 'info=: getFrom1Drive ''F:\'''
to get all the photos from directories on the camera memory chip available as the “F:” drive, assuming a standard set of directory names that may vary by camera model. The “info” and “6!:2” parts assign information about the photos to a variable for later use and time the transfer, respectively. Currently these names are set for a Fuji FinePix which creates numbered sub-directories with names starting with “DSCF” under a directory “DCIM”. A Nikon camera might create numbered directories starting with “DSCN”.
After this function moves the pictures from the camera chip to a sub-directory named by the date range of pictures obtained, I would run some variant of the line
4 parcelOutFlipping dd=. endSlash '"'-.~>0{info NB. 4 parallel flippers
to spin off four independent processes to flip the photos in the recently-created directory. Typically I have been using the left argument “2” instead of “4” because my current machine has two cores and this seems logical. However, I’ve been running with four parallel processes lately to gather realistic timings to evaluate the efficiency of running more processes than there are cores.
[More recently, I've reverted to invoking only two processes because the four-process version seemed to occasionally lock up my PC.]
Parallel Photo Flipping
NB.* savePhotoDirInfo.ijs: save dir info of photos on camera/chip, copy pix.
load 'csv task filefns images'
coinsert 'fldir'
PHOTOP=: 'c:\amisc\pix\Photos\' NB. Top dir where I keep photos.
jpeg_quality_ima3_=: 99 NB. Save .jpgs as 99% lossless.
smoutput 0 : 0 NB. Remind us of usual use.
info,<6!:2 'info=: getFrom1Drive ''F:\'''
info,<6!:2 'info=: getFrom1Drive ''C:\amisc\''' NB. This or preceding...
6!:2 'regflipphotos ''"''-.~>0{info' NB. Single-thread or
4 parcelOutFlipping dd=. endSlash '"'-.~>0{info NB. 4 parallel flippers
fsize dd,'time.out'
ratioKept >0{info
)
NB.* jpgSzs: give size and number of .jpg files in dir y.
jpgSzs=: 13 : '(+/,#);2{"1 dir ''*.jpg'',~endSlash y'
NB.* szRatio: give files sizes, ratio of photos discarded and kept.
szRatio=: (jpgSzs,ratioKept)
szRatio_eguse_=: 0 : 0
TD=. 'C:\amisc\pix\Photos\2010Q2\'
kr=. (3 : 'y(],.[:szRatio&.>],&.>~[:<[) jd dir ''*'',~y=. endSlash y') TD
((0{"1 kr),.":&.>&>1{"1 kr) writecsv 'KeptRatio2010Q2.csv'
)
NB.* evenlyPartition: evenly partition y into x pieces w/smaller at end.
evenlyPartition=: 4 : '(x(([:i.]) e. ([:i.[) * [:>.%~)#y)<;.1 y'
NB.* evenlyPartitionBy: evenly partition y into (>0{x) pieces by >1{x amts.
evenlyPartitionBy=: 4 : '(+/ (+/\>1{x) >/~ (}.i.>0{x)*(>0{x)%~+/>1{x) </. y'
NB.* delFlIfExist: delete file if it exists.
delFlIfExist=: 3 : 'if. fexist y do. ferase y end.'
NB.* parcelOutFlipping: run x separate processes to flip photos in dir y.
parcelOutFlipping=: 4 : 0
'fls szs'=. |:0 2{"1 dir '*.jpg',~y=. endSlash y
szs=. ;szs [ fls=. <"1 fls
NB. fls=. x evenlyPartition fls NB. Put filenames into evenly-divided lists.
fls=. (x;szs) evenlyPartitionBy fls NB. % filenames evenly by file size.
scrfls=. (<'.ijs'),~&.>(<'FlipScript'),&.>":&.>i.x
delFlIfExist&.>scrfls
1!:44 y [ svdir=. 1!:43 '' NB. Move to target dir to flip photos.
(<fread jpath '~Code/sampFlipPhotos.ijs') fwrite&.>scrfls
(<y) appendArgsToScript&.>scrfls;&.><"0 fls
exe=. (<'"',(jpath '~bin'),'\j.exe" -jijx "'),&.><endSlash 1!:43 ''
fork&.>exe,&.>scrfls,&.>'"'
1!:44 svdir
NB.EG parcelOutFlipping '"'-.~>0{info
)
NB.* appendArgsToScript: for photo directory x, add filename info to scripts.
appendArgsToScript=: 4 : 0
'scrflnm flnms'=. y NB. Script file names, photo file names
(LF,~LF,~'IPD=: <''',x,'''') fappend scrflnm NB. Photo dir as global
(')',~;LF,~&.>'PHFLS=: <;._2]0 : 0';dltb&.>flnms) fappend scrflnm
(LF,LF,~'onlyRuntime ''''') fappend scrflnm NB. Run code if standalone.
)
NB.EG Append code like this, with different "PHFLS", to multiple script files.
NB. IPD=: 'c:\amisc\pix\Photos\2010Q1\20100311\'
NB. PHFLS=: 0 : 0
NB. DSCF3837.jpg
NB. DSCF3838.jpg
NB. DSCF3839.jpg
NB. )
NB.* regflipphoto: flip a .jpg pic 1/4 counterclockwise as is most common.
regflipphoto=: 3 : 0
(1 0 2|:|."2 a.{~read_image y) write_image y
NB.EG
)
NB.* regflipphotos: flip all .jpg in dir 1/4 counterclockwise.
regflipphotos=: 3 : 0
y=. endSlash y
regflipphoto&.>(<y),&.>{."1 dir y,'*.jpg'
NB.EG 6!:2 'regflipphotos ''c:\pix\'''
)
NB.* getFrom1Drive: get photos from drive y -> \amisc\pix\photos\[yyyyQq] year and quarter
getFrom1Drive=: 3 : 0
phd=. endSlash y
topdirs=. dir phd,'*.'
NB. +----+------------------+-+---+------+
NB. |DCIM|2008 6 22 13 28 44|0|rw-|----d-| NB. Example top-level photo dir
NB. +----+------------------+-+---+------+
if. (<'DCIM') e. {."1 topdirs do. getPixDir phd,'DCIM\' else. 0 end.
NB.EG getFrom1Drive 'F:\'
NB.EG info;]6!:2 'info=: getFrom1Drive ''E:\'''
)
NB.* getPixDir: get pictures from directory on camera assuming standard names.
getPixDir=: 3 : 0
NB. subds=. {."1 dir phd,'DCIM\*.'
subds=. {."1 dir '*.',~y=. endSlash y
NB. +--------+--------+
NB. |102_FUJI|101_FUJI| NB. Example photo sub-dirs
NB. +--------+--------+
'destdir sumszs'=. accumSvInfo y
cmd=. (<'move "'),&.>(<y),&.>subds,&.><'\*.jpg" ',destdir
cmd=. cmd,(<'move "'),&.>(<y),&.>subds,&.><'\*.avi" ',destdir
rr=. shell&>cmd
rr=. >,&.>/_3{.&.><;._2&.>rr
destdir;(ratioKept destdir-.'"');sumszs
NB.EG 'destdir ratios totsz'=. getPixDir 'H:\DCIM\'
)
NB.* accumSvInfo: accumulate picture file information & save before altering.
accumSvInfo=: 3 : 0
inf=. >,&.>/dir&.>(<y),&.>({."1 dir y,'*.'),&.><'\*.jpg'
inf=. inf,>,&.>/dir&.>(<y),&.>({."1 dir y,'*.'),&.><'\*.avi'
inf=. inf/:1{"1 inf NB. Order by ascending date
NB. +----------------------+----------------------+
NB. |G:\DCIM\102_FUJI\*.jpg|G:\DCIM\101_FUJI\*.jpg|
NB. +----------------------+----------------------+
dtstr=. combineDates >~.3{.&.>1{"1 inf NB. Date string, e.g. 20080628-29
initDt=. 2{.>(<0 1){ inf NB. Characterize by initial date
dsd=. (":0{initDt),'Q',":>.3%~1{initDt NB. yyyyQq, e.g. 2008Q3<-3rd qtr 2008
shell 'mkdir ',destdir=. '"','"',~PHOTOP,dsd,'\',dtstr
saveInfoFl destdir;<(":&.>_2}."1 inf) NB. Info file->new dir
destdir;+/;2{"1 inf NB. Sum file sizes
)
NB.* ary2csv v represent enclosed array in csv format.
ary2csv=: 3 : 0
;(<@(,&LF)@}:@;)"1 ('"'&,@(,&'",')@(#~ >:@(=&'"'))) &.> ,&.> 8!:2 &.> y
)
NB.* saveInfoFl: save file of information on file sizes, save dates and times.
saveInfoFl=: 3 : 0
'ddir inf'=. y
ifl=. (ddir-.'"'),'\info.csv' NB. Append w/o header if file exists.
(ary2csv (fexist ifl)}.inf,~'Name';'Date';'Size') fappend ifl
ifl
)
NB.* ratioKept: estimate ratio of photos taken to those retained.
ratioKept=: 3 : 0
if. 1=#$y do. NB. Single, enclosed item is dir name.
fls=. {."1 jfi dir (y-.'"'),'\*.jpg'
else. fls=. {."1 y end.
if. 0=#nn=. countPicNums fls do. 0 0 0 else.
(|0 1-(#nn)%>:(>./-<./)nn),#fls end. NB. Ratio kept, discarded; # files
)
countPicNums=: 3 : 0
nn=. (4{. 4}.])&.>y NB. Get picture nums from file names.
nn=. ".&>nn#~isValNum&>nn NB. Only numbers
NB. 6667 and 3333 are arbitrary to catch break from 9999 -> 0 rollover.
if. ((6667+./ .>~])*. 3333+./ .<~]) nn do. nn=. nn+10000*nn<:6667 end.
)
NB.* combineDates: combine list of dates into string from earliest-latest,
NB. combining common high-order date parts; see examples below.
combineDates=: 3 : 0
dtlst=. ({.,:{:)4 _2 _2{.&.>"1]4 lead0s&.>/:~(_2{.1,$y)$,y
deq=. *./\=/dtlst NB. Where leading parts of dates are equal
dtstr=. ;(deq#0{dtlst),}:,(<'-'),.~(-.deq)#"1 dtlst
dtstr}.~-'-'={:dtstr
NB.EG combineDates ,:2008 6 30
NB.EG 20080630
NB.EG combineDates 2008 6 29,:2008 6 30
NB.EG 20080629-30
NB.EG combineDates 2008 6 29,2008 6 30,2008 7 1,:2008 7 3
NB.EG 20080629-0703
NB.EG combineDates 2008 12 30,2008 12 31,:2009 1 1
NB.EG 20081230-20090101
)
Parallel Photo Flipping Task Template
This is the template customized for each parallel task by the code above. This version implements mutex to prevent the multiple processes from over-writing each others’ information in their common timing file.
NB.* sampFlipPhotos.ijs: sample dedicated photo flipper.
load 'task images filefns ~Code/mutex.ijs'
coinsert 'fldir' [ coinsert 'mutex'
jpeg_quality_ima3_=: 99
NB.* regflipphoto: flip a .jpg pic 1/4 counterclockwise as is most common.
regflipphoto=: 3 : 0
(1 0 2|:|."2 a.{~read_image y) write_image y
NB.EG
)
onlyRuntime=: 3 : 0
NB.* onlyRuntime: only invoke if not loaded via interactive session.
if. (5{.&.>'-jijx';<'-rt') +./ . e. 5&{.&.>tolower&.>ARGV_z_ do.
1!:44 >IPD [ arg=. 1|.'""',}.;' ',&.>ARGV_z_ [ flout=. 'time.out'
getHold ''
(LF,~'Start flips (for ',arg,') @ ',":qts'') fappend flout
releaseHold ''
tm=. ":6!:2 'regflipphoto&.>IPD,&.>PHFLS'
getHold ''
(LF,~'Finished flips (for ',arg,') in ',tm,' seconds @ ',":qts'') fappend flout
releaseHold ''
2!:55 ''
end.
)
Parallel Directory Parsing
The following code demonstrates another task run in parallel. This code parses the contents of a specified set of files, usually starting from the root of a hard-drive, to create several global vectors of information about all the files and directories under the starting point. These vectors can be used for a number of things but are most commonly used to implement a flexible backup routine which allows us to copy a certain amount of the most recently-changed files.
There are three scripts here. The main one, parseDir.ijs contains the original, serial directory-parsing code. The parallel extensions to this comprise two scripts: the main routine parallelParseDir.ijs which implements the parsing in parallel, and the template script pllPDSub.ijs which is customized for each parallel task by the main routine.
NB.* ParseDir.ijs: tools to parse directory listing->backup most recently changed.
NB. Will look for command-line arguments SZLIM (max bytes to backup) and TARGDIR
NB. (target directory to which to write backups).
require '~Code/locationInfo.ijs' NB. On which machine are we running?
require '~Code/parallelParseDir.ijs'
DEBUGON=: 0 NB. Show messages as program progresses (1) or don't (0).
SESSWDW=: 0 NB. Open session window (1) or not (0)
DBGFL=: 'C:\Temp\parseDir.tmp' NB. These 3 things allow logging at load
sepLF=: 13 : ';((": :: ])y),10{a.' NB. time: DBGFL (log file), sepLF, and
fappendDHM=: 4 : '(,x) (#@[ [ 1!:3) :: _1: (([: < 8 u: >) ::]) y' NB. this.
3 : 0 ''
if. DEBUGON do. (;sepLF&.>'parseDir.ijs:13';ARGV_j_,<6!:0 '') fappendDHM DBGFL end.
)
NB. "C:\Program Files\J602\j.exe" c:\amisc\JSys\user\code\parseDir.ijs -jijx SZLIM 10e6 RUNANYDAY Y RUNFROMSAVED Y TARGDIR D:\
NB.* nameSpan: given 2 names like pfxYYYYMMDD, combine->pfxYYYYMMD1-D2.
NB.* consolidateBkpsTowardPast: copy all contents of successive dirs into
NB.* coreRunDaily: core code for "runDaily": daily backups.
NB.* buildBatFl: build .BAT file to create target dirs and copy files to them.
NB.* indicateSubdirs: from boolean selecting DIRNMS, indicate all subdirs.
NB.* rmEndSep: remove terminal path separator from string.
NB.* extractExclusions: extract names of target, exclude dirs, and files from global ExcludeUsual.
NB.* excludeFiles: exclude designated files from list to back up.
NB.* dirDependencies: convert list of full paths to index vector form of tree
NB.* getDirFlInfo: get info on dirs and files starting at node specified.
NB.* cvtDt2Num: convert Y M D h m s date to single num: YYYYMMDD.day fraction.
NB.* ExcludeUsual: list of usual files and directories to exclude from backup.
NB.* initFlsDir: parse memory-mapped file of directory listing->file, dir info.
NB.* getInfo: get directory into into file, memory-map and parse it.
NB.* process1Subdir: parse single subdir entry->files, parent dirs as globals.
NB.* extract1SubdirList: get first full sub-directory listing out of many.
NB.* addPath: put new parent/child index in tree from text of "dir\subdirs..."
NB.* Tst0addPath_tests_: test adding path to index vec tree from text.
NB.* mcopyto: text of DOS .BAT file to do multiple copies.
NB.* runDaily: backup to run every day: save some most recently changed files.
NB.* setGlobalParms: assign globals according to defaults or command-line overrides.
NB.* NYto01: convert 'N' or 'Y' to 0 or 1, respectively.
NB.* onlyRuntime: only invoke if not loaded via interactive session.
NB.* runFromSavedVars: Run backup assuming dir&file vars already saved.
NB.* nameSpan: given 2 names like pfxYYYYMMDD, combine->pfxYYYYMMD1-D2.
nameSpan=: 3 : 0
NB. y=. ({:,{.) dd NB. Earliest, latest
endlast=. 1 i.~ ~:/>y
(>0{y),'-',endlast}.>1{y
)
consolidateBkpsTowardPast=: 3 : 0
NB.* consolidateBkpsTowardPast: copy all contents of successive dirs into
NB. earliest starting with oldest->consolidated backups with newest version
NB. overwriting older ones.
y=. 2{.boxopen y NB. Files' prefix, dir in which to work
pfx=. openbox {.y NB. Prefix is "WL" for laptop, "WD" desktop
if. 0=#>1{y do.
wrkdir=. 'C:\Temp\'
else. wrkdir=. endSlash >1{y
end.
dd=. jd dir wrkdir,'*.' NB. Just directories
dd=. (<pfx,'[0-9]{8}$') rxfirst&.>dd NB. Only names like {pfx}YYYYMMDD
dd=. \:~dd-.a: NB. Order from most to least recent.
if. nameExists 'TOPDIR' do. svTD=. TOPDIR end.
TOPDIR=: wrkdir
2 moveDirOverAnother/\dd
newnm=. nameSpan ({:,{.) dd
cmd=. (TOPDIR{.~>:TOPDIR i. ':'),' && cd ',(TOPDIR}.~>:TOPDIR i. ':'),' && '
cmd=. cmd,'ren "',(>{:dd),'" "',newnm,'"'
shell cmd
if. nameExists 'svTD' do. TOPDIR=: svTD NB. Re-instate or
else. 4!:55 <'TOPDIR' end. NB. remove global.
NB.EG consolidateBkpsTowardPast 'WD';'C:\Temp\WrkDesk\'
)
coreRunDaily=: 0 : 0
NB.* coreRunDaily: core code for "runDaily": daily backups.
setGlobalParms '' NB. Cmd-line size limit, target dir, if present.
wkdy=. dow 3{.qts '' NB. Weekday number: 0=Sunday, 6=Saturday
wkdy e. 1 2 3 4 5 NB. Only Mon-Fri unless any
RUNANYDAY+.wkdy e. 1 2 3 4 5
sink=. runDaily '' [ 4!:55 <'ASSERR'
nameExists 'ASSERR'
)
NB. Display commonly-used set of commands in usual sequence.
smoutput 0 : 0
6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=. PllDirInfoEG ''C:\'''
6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=: getDirFlInfo SRCDIR'
(<'\Temp\') fileVar_WS_&.>'FLNMS';'FLDTS';'FLSZS';'FLPARENT';'DIRNMS';'DIRDEP'
'batfl cmds'=. buildBatFl_parseDir_ 700e6;'C:\Temp\Recent\'
shell batfl
)
coclass 'parseDir'
require 'jmf files dir dt logger filefns task'
coinsert 'base fldir'
USUVARS=: 'FLNMS';'FLDTS';'FLSZS';'FLPARENT';'DIRNMS';'DIRDEP'
EV=: getEnviVars ''
WINDIR=: endSlash ,>EV{~<1,~(toupper&.>0{"1 EV)i.<'WINDIR'
buildBatFl=: 3 : 0
NB.* buildBatFl: build .BAT file to create target dirs and copy files to them.
'szlim targ'=. y
xclud=. excludeFiles targ
dtdord=. xclud-.~\:FLDTS NB. File list indexes in date desc. order
toobig=. szlim<:dtdord{FLSZS NB. Exclude any single file > size limit
toobig=. toobig#dtdord NB. as this will truncate list prematurely.
dtdord=. dtdord-.toobig
ss=. +/\dtdord{FLSZS
cutoff=. 1 i.~ss>szlim
NB. Do better job noting which files excluded because too big; the following
NB. fails to account for cutoff.
if. DEBUGON do.
pl=. 's'#~1~:#toobig
logMsg_logger_ 'Excluding file',pl,' singly > size limit: ',":#toobig
end.
ix=. cutoff{.dtdord
cmds=. (0{DIRNMS),makeCopyCmds targ;<ix NB. 0{DIRNMS gives disk to which
tmpd=. getTempDir '' NB. to copy from.
cmds v2f batfl=. tmpd,'CDMDCopy.bat'
batfl;<cmds
NB.EG 'batfl cmds'=. buildBatFl 700e6;'C:\Temp\Recent\'
)
locChildren=: 3 : '(<^:(L. = 0:)y),~(<#~0~:#) I. DIRDEP e. >0{y'
indicateSubdirs=: 3 : '~.;;locChildren^:_&.>I. y'
indicateSubdirs0=: 3 : 0
NB.* indicateSubdirs: from boolean selecting DIRNMS, indicate all subdirs.
xdix=. I. y
childxd=. xdix-.~I. DIRDEP e. xdix
while. 0<#childxd do.
xdix=. ~.xdix,childxd
childxd=. xdix-.~I. DIRDEP e. xdix
end.
xdix NB. Index into DIRNMS of all subdirectories.
NB.EG xdix=. indicateSubdirs DIRNMS e. 'c:\amisc';'c:\Program Files'
)
NB.* rmEndSep: remove terminal path separator from string.
rmEndSep=: 3 : '(]}.~[:-PATHSEP_j_={:)"1 dtb y'
extractExclusions=: 3 : 0
NB.* extractExclusions: extract names of target, exclude dirs, and files from global ExcludeUsual.
targ=. rmEndSep y NB. Exclude target to avoid unwanted recursion.
sections=. '[ExcludeDirs]';'[ExcludeFiles]'
xu=. <;._1 LF,ExcludeUsual-.CR
xu=. xu#~&.>-.&.>+./\&.>(<'NB.')E.&.>xu NB. Exclude comments
xu=. xu#~0~:;#&.>xu
whsect=. >+./&.>sections E.&.>/ xu
secord=. /:sections i. ' '-.~&.>xu#~+./whsect
targ;secord{(+./whsect)<;._1 xu
NB.EG 'targ xd xf'=. extractExclusions y
)
excludeFiles=: 3 : 0
NB.* excludeFiles: exclude designated files from list to back up.
'targ xd xf'=. extractExclusions y
whxd=. DIRNMS}.~&.>(2*;':'e.&.>DIRNMS)*;DIRNMS i.&.>':' NB. Drop disk prefix.
whxd=. (toupper&.>whxd)e. PATHSEP_j_,&.>toupper&.>xd,<targ NB. Exclude target
whxd=. (1) (indicateSubdirs whxd)}whxd NB. to avoid recursion.
whxf=. (toupper&.>FLNMS)e. toupper&.>xf NB. Exclude files.
xclud=. I. whxf+.FLPARENT e. I. whxd
NB. xclud is list of indexes into FLNMS=files to exclude.
)
dirDependencies=: 3 : 0
NB.* dirDependencies: convert list of full paths to index vector form of tree
NB. showing directory and subdirectories as parent-child relations.
odir=. y
DIRDEP=: (#odir)$_1 NB. Dummy entry for 1st node: _1->no parent.
for_dc. i.#odir do.
cd=. PATHSEP_j_,~&.>dc{odir
len=. #>cd NB. Which prefixes match only current?
subs=. (;cd-:&.>len{.&.>odir)*.-.;PATHSEP_j_ e.&.>len}.&.>odir
DIRDEP=: subs}DIRDEP,:dc
end.
NB.EG dirDependencies 'c:';'c:\t1';'c:\t1\sb1';'c:\t2';'c:\t2\sb2';'c:\t1\sb3'
NB. _1 0 1 0 3 1 NB. Parent index for each input; _1 for no parent.
)
NB.* dirInfo: put directory info in more usable format: names, dates, sizes, dir flag.
dirInfo=: ([:((0{"1]) ; ([:>1{"1]) ; ([:;2{"1]) ; [:<'d'e.&>4{"1]) dir)
getDirFlInfo=: 3 : 0
NB.* getDirFlInfo: get info on dirs and files starting at node specified.
dskInf=. >(];[: dirInfo '\*',~]) generalWalkTree rmEndSep y
DIRNMS=: 0{"1 dskInf NB. Full names of all paths
DIRDEP=: (] i. (]{.~PATHSEP_j_ i:~])&.>) DIRNMS
DIRDEP=: (_1) (I. DIRDEP=i.#DIRDEP)}DIRDEP NB. Dirs' dependency tree: parent indexes
DIRDEP=: (_1) (I. DIRDEP>:#DIRDEP)}DIRDEP NB. "_1" is root (no parent).
isfl=. -.;4{"1 dskInf NB. Exclude dir info->only files.
FLNMS=: isfl#;1{"1 dskInf
ned=. 0~:#&>2{"1 dskInf NB. No empty directories
FLDTS=: isfl#;cvtTS21Num &.>ned#2{"1 dskInf NB. Date as single num: YYYYMMDD.day fraction
FLSZS=: isfl#;3{"1 dskInf NB. File size in bytes.
FLPARENT=: isfl#;(#&.>3{"1 dskInf)#&.>i.#DIRNMS
FLNMS;FLDTS;FLSZS;FLPARENT;DIRNMS;<DIRDEP
)
NB.* cvtDt2Num: convert Y M D h m s date to single num: YYYYMMDD.day fraction.
cvtDt2Num=: 3 : 0"1
NB. 90065 = 5 + 24 60 60#.24 60 60 NB. Max secs/day+5 fudge for leap secs.
(100#.3{.y)+90065%~24 60 60#._3{.y
NB.EG ' 20090803.32600899' -: 18j8":cvtDt2Num 2009 8 3 8 9 22
)
NB. Only distinguishes to about 1/10,000 second.
NB. Need to include a [regexp] exclusion section to apply to files, e.g. "saves-{d}*", "*~", etc.
NB. Exclude the usual files and directories from being copied given list of Files &
NB. Directories (result of munge_dir) & source Disk & target Disk[:\dir] names.
NB. Some DBs big enough to be done separately.
NB.* ExcludeUsual: list of usual files and directories to exclude from backup.
ExcludeUsual=: 0 : 0
[ExcludeDirs]
$avg
.emacs.d
CFGSAFE
C_DILLA
Documents and Settings
MSOCache
Program Files
Recycled
Recycler
WINDOWS
amisc\DCIM
i386
[ExcludeFiles] NB. Files to exclude from any directory
~
*.bz2
*.zip
AUTOEXEC.BAT
CONFIG.SYS
IO.SYS
MSDOS.SYS
NTDETECT.COM
NTLDR
SECURITY
SOFTWARE
SYSTEM
SYSTEM.ALT
Thumbs.db
boot.ini
eventlog.log
)
NB.*To do: actually use the wildcards in the file list above!!!
NB. Replace WINNT with actual Windows system dir from environment var.
ExcludeUsual=: ('WINNT';<'\'-.~WINDIR}.~WINDIR i. '\') stringreplace ExcludeUsual
NB. -------- Dir listing fns: parse text file directory listing:
NB. -------- this is a separate way to accomplish what has been done above.
initFlsDir=: 3 : 0
NB.* initFlsDir: parse memory-mapped file of directory listing->file, dir info.
JCHAR map_jmf_ 'DIRLSTFL';y
DBSTR=: LF,' Directory of '
WHDB=. (CR,DBSTR) E. DIRLSTFL NB. Where directory breaks are
WHDB=: (I. WHDB),<:#DIRLSTFL
DIRNMS=: FLNMS=: ''
FLSZS=: DIRPARENT=: FLPARENT=: i.0
FLDTS=: 0$0.0
)
3 : 0 ''
if. DEBUGON do. (;sepLF&.>'parseDir.ijs:383';<6!:0 '') fappendDHM DBGFL end.
)
getInfo=: 3 : 0
NB.* getInfo: get directory into into file, memory-map and parse it.
NB. winexec 'cmd /C dir /A /S C:\ > C:\allfls2.dir';1
dirlstfl=. y
if. 0=#dirlstfl do.
dirlstfl=. 'C:\allfls2.dir'
shell 'dir /A /S C:\ > ',dirlstfl
end.
initFlsDir dirlstfl
for_ix. i.<:#WHDB do.
ch=. extract1SubdirList WHDB{~ix+0 1
process1Subdir ch
end.
unmapall_jmf_ ''
)
process1Subdir=: 3 : 0
NB.* process1Subdir: parse single subdir entry->files, parent dirs as globals.
ch=. y
thisdir=. (($DBSTR)}.ch) {.~ 1 i.~ LF E. ($DBSTR)}.ch
'isnew thisdn'=. addPath thisdir
ch=. (<;._1 ch)-.a: NB. break into lines; no empty lines
ch=. ch#~' '~:;{.&.>ch NB. Get rid of lines beginning with space.
ch=. <;._1&.>' ',&.>dsp&.>ch NB. break apart lines by spaces
NB. re-join any names with embedded spaces
ch=. |:>(3{.&.>ch),&.><&.>(}.@;)&.>(' '&,)&.>&.>3}.&.>ch
chtit=. 'DATE';'TIME';'SIZE';'NAME' NB. row titles for "ch"
whmootdirs=. (ch{~chtit i. <'NAME')e. ,&.>'.';'..'
NB. "SIZE" column has "<DIR>" indicator for directory, size for file.
whdir=. ((ch{~chtit i. <'SIZE')e. <'<DIR>')*.-.whmootdirs
addPath&.>(<thisdir,'\'),&.>whdir#ch{~chtit i. <'NAME'
whfls=. -.whdir+.whmootdirs
ch=. whfls#"1 ch
FLNMS=: FLNMS,ch{~chtit i. <'NAME'
FLPARENT=: FLPARENT,(+/whfls)$thisdn
FLSZS=: FLSZS,;n2j&.>(ch{~chtit i. <'SIZE')-.&.>','
FLDTS=: FLDTS,;DateTimeCvt &.>,&.>/' ',&.>ch{~chtit i. 'DATE';'TIME'
thisdn
)
extract1SubdirList=: 3 : 0
NB.* extract1SubdirList: get first full sub-directory listing out of many.
'st end'=. y
ch=. CR-.~(st+i.>:end-st){DIRLSTFL
)
addPath=: 3 : 0
NB.* addPath: put new parent/child index in tree from text of "dir\subdirs..."
p2a=. <;._1 '\',y NB. Path to add, e.g. 'C:\top\mid\bottom'
p2a=. p2a-.a:
isnew=. 0 [ wh=. _1
for_nm. p2a do. NB. "wh" is parent index of current node...
if. 0=#wh2=. I. DIRNMS e. nm do. NB. new subdir
DIRPARENT=: DIRPARENT,wh
wh=. {.<:#DIRNMS=: DIRNMS,nm NB. will be parent of next node, if any
isnew=. 1
else.
if. 0=#wh2=. wh2#~wh=wh2{DIRPARENT do. NB. Name exists but with
DIRPARENT=: DIRPARENT,wh NB. different parent.
wh=. {.<:#DIRNMS=: DIRNMS,nm
isnew=. 1
else. wh=. {.wh2 end. NB. Name exists with
end. NB. same parent
end.
isnew,wh NB. 0 if path was already here
)
Tst0addPath_tests_=: 3 : 0
NB.* Tst0addPath_tests_: test adding path to index vec tree from text.
coinsert 'parseDir base'
d0=. DIRNMS=: 'C:';'Aegis';('\'-.~WINDIR}.~WINDIR i. '\');'Web';'printers';'foo';'Web'
dp0=. DIRPARENT=: _1 0 0 3 4 0 0
assert. 0 2-:addPath WINDIR NB. Shouldn't add it again.
assert. d0-:DIRNMS NB. Should not have changed
assert. dp0-:DIRPARENT NB. Should not have changed
td=. 'D:\foo\bar' NB. New path starting from new root
assert. 1 9-:addPath td NB. but with same-named sub as existing
assert. (d1=. DIRNMS)-:d0,<;._1 '\',td
assert. (dp1=. DIRPARENT)-:_1 0 0 0 3 4 0 0 _1 8 9
assert. 0 9-:addPath td NB. Shouldn't add it again.
assert. d1-:DIRNMS NB. Should not have changed
assert. dp1-:DIRPARENT NB. Should not have changed
assert. 1 10-:addPath WINDIR,'foo'
1
)
NB.* mcopyto: text of DOS .BAT file to do multiple copies (in case it's missing).
mcopyto=: 0 : 0
Rem MCopyTo.bat: Multiple COPY TO %1: copy %2, %3, %4, etc.
:START
If %1/==/ goto SHOWHOW
Set tmpnm=%1
:DO1
If %2/==/ goto BYEBYE
Copy %2 %tmpnm% > nul
Shift
Goto DO1
:SHOWHOW
Echo on
Rem MCOPY target source1 source2...sourceN
Echo off
Goto BYEBYE
:BYEBYE
Set tmpnm=
)
NB.* setTargetDir: Different target dir according to which known machine backed up.
setTargetDir=: 3 : 0
if. -.nameExists 'TARGDIR' do. TARGDIR=. '' end.
if. -.nameExists 'TMPDIR' do. TMPDIR=. getTempDir '' end.
if. 0=#TARGDIR do. NB. Assign dir prefix based on machine, if known.
pfx=. KMPFX
'machnm usernm'=. whoami ''
pfx=. >pfx{~KNMACH i. <machnm NB. Prefix based on computer name
today=. ;4 2 2 lead0s&.>3{.qts '' NB. today's date as 'yyyymmdd'
dirnm=: TMPDIR,pfx,today,'\'
else. dirnm=: endSlash TARGDIR end.
TMPDIR;dirnm
)
NB. ARGV_z_=: ARGV_z_,<'-jijx'
NB.* runDaily: backup to run every day: save some most recently changed files.
runDaily=: 3 : 0
TMPDIR=: 'C:\Temp\' [ setGlobalParms ''
today=. ;4 2 2 lead0s&.>3{.qts '' NB. today's date as 'yyyymmdd'
'machnm usernm'=. whoami ''
if. DEBUGON do.
logMsg_logger_ 'Running daily backup for machine, user: '
logMsg_logger_ ' ',machnm,', ',usernm,' at ','.',~showdate ''
end.
'TMPDIR dirnm'=. setTargetDir ''
ASSERR=: '*** Directory "',dirnm,'" does not exist.'
dodircheck=. 1
if. 3=#dirnm do. NB. if dir is root: 'C:\'
if. (PATHSEP_j_={:dirnm)*.':'=1{dirnm do. dodircheck=. 0 end.
end.
if. dodircheck do.
if. -.dirExists dirnm do. assert. createdir dirnm end.
end.
if. DEBUGON do.
logMsg_logger_ 'Created directory ',';',~dirnm
logMsg_logger_ 'starting "getDirFlInfo" at ','...',~showdate ''
end.
if. RUNFROMSAVED do. ASSERR=: '*** Error in runDaily at "if. RUNFROMSAVED".'
TMPDIR=. 'C:\Temp\' NB. [ 4!:55 <'TARGDIR'
'TMPDIR dirnm'=. setTargetDir ''
setGlobalParms ''
TMPDIR runFromSavedVars SZLIM;dirnm
else. ASSERR=: '*** Error in getDirFlInfo.'
tmpbkp=. 6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=: getDirFlInfo SRCDIR'
NB. 'FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'=. PllDirInfoEG 'C:\'
nm=. 'tmpbkp',today
".nm,'=: tmpbkp'
if. DEBUGON do.
showtime tmbkp
logMsg_logger_ 'Saving file and dir info in ',TMPDIR,'...'
end.
ASSERR=: '*** Error 0 saving file and dir info.'
nms=. nm;USUVARS
".&.>nms,&.>(<'_base_=: '),&.>nms
ASSERR=: '*** Error 1 saving file and dir info.'
xx=. (<TMPDIR) fileVar_WS_&.>nms
coclass 'parseDir' NB. Get back to this namespace after
ASSERR=: '*** Error 2 saving file and dir info.'
".&.>nms,&.>(<'=: '),&.>nms,&.><'_base_' NB. Ensure local names are same
coinsert 'base' NB. fileVar_WS_ moved us to "base".
ASSERR=: '*** Error 3 saving file and dir info.'
if. DEBUGON do.
logMsg_logger_ 'Save result:'
logMsg_logger_ >xx
end.
if. -.nameExists 'SZLIM' do. SZLIM=. 1e7 end.
ASSERR=: '*** Error building batch file.'
'batfl cmds'=: buildBatFl SZLIM;dirnm,~'C:'#~-.':'e.2{.dirnm
if. DEBUGON do.
logMsg_logger_ 'About to run batch file at ','...',~showdate ''
end.
ASSERR=: '*** Error running batch file ',batfl,'.'
shell batfl
end.
closeLog_logger_ '' [ wait 5
4!:55 <'ASSERR' NB. Eliminate global error message if we got this far.
)
NB.* setGlobalParms: set globals according to defaults or command-line overrides.
setGlobalParms=: 3 : 0
SZLIM=: ''$>1e7 lookupValAfterName ARGV_z_;<'SZLIM'
if. -.isNum SZLIM do. SZLIM=: ".SZLIM end.
RUNFROMSAVED=: NYto01 {.>0 lookupValAfterName ARGV_z_;<'RUNFROMSAVED'
TARGDIR=: ,>lookupValAfterName ARGV_z_;<'TARGDIR'
SRCDIR=: ,>'C:\' lookupValAfterName ARGV_z_;<'SRCDIR'
RUNANYDAY=: NYto01,>'N' lookupValAfterName ARGV_z_;<'RUNANYDAY'
if. -.nameExists 'DEBUGON' do.
DEBUGON=: NYto01,>0 lookupValAfterName ARGV_z_;<'DEBUGON'
end.
ans=. ans,:".&.>ans=. 'SZLIM';'RUNFROMSAVED';'TARGDIR';'RUNANYDAY';'DEBUGON'
)
NB.* NYto01: convert 'N' or 'Y' to 0 or 1, respectively.
NYto01=: 3 : 'if. -.isNum y do. ''Y''=toupper {.y else. y end.'
)
NB.* onlyRuntime: only invoke if not loaded via interactive session.
onlyRuntime=: 3 : 0
if. (5{.&.>'-jijx';<'-rt') +./ . e. 5&{.&.>tolower&.>ARGV_z_ do.
NB. Only invoke for runtime, not interactive (edit) session.
if. -.nameExists 'DEBUGON' do. DEBUGON=: 0 end.
if. -.nameExists 'SESSWDW' do. SESSWDW=: 0 end.
setGlobalParms '' NB. Cmd-line size limit, target dir, if present.
wkdy=. dow 3{.qts '' NB. Weekday number: 0=Sunday, 6=Saturday
if. RUNANYDAY+.wkdy e. 1 2 3 4 5 do. NB. Only Mon-Fri unless
try. sink=. runDaily '' [ 4!:55 <'ASSERR' NB. any day allowed.
catch. if. nameExists 'ASSERR' do. logMsg_logger_ ASSERR
else. msg=. '*** Unspecified error in "runDaily" at '
logMsg_logger_ msg,'.',~showdate ''
end.
end.
end.
if. -.SESSWDW do.
closeLog_logger_ '' [ wait 5 NB. 5 secs to read msg.
end.
end.
)
NB.* runFromSavedVars: Run backup assuming dir&file vars already saved.
runFromSavedVars=: 3 : 0
'\amisc\' runFromSavedVars y
:
bkpargs=. y NB. 'szlim targ'=. y
vn=. USUVARS
(<x,'\'#~'\'~:{:x) unfileVar_WS_&.>vn
".&.>(<'parseDir_'),&.>vn,&.>(<'_=. '),&.>vn
'batfl cmds'=. buildBatFl bkpargs
shell batfl
wait 2 NB. Give .BAT chance to start.
NB.EG runFromSavedVars 5e6;'C:\Temp\WL20050315\'
)
3 : 0 ''
if. DEBUGON do. (;sepLF&.>'parseDir.ijs:638';<6!:0 '') fappendDHM DBGFL end.
)
coclass 'base'
coinsert 'parseDir'
onlyRuntime ''
locationInfo.ijs
This script sets an arbitrary string to distinguish between the different machines on which I usually work. It has been anonymized here to use only made-up example names.
NB.* locationInfo.ijs: information to distinguish run-time location
NB.* KNMACH: Known machines: network IDs to distinguish different machines I use.
NB.* KMPFX: Known-machine prefixes to de-couple network ID from functionality.
NB.* KNIDS: Known-IDs associated with each machine - good idea?
NB.* whoami: find what machine and user-id I'm on (assume Novell net command).
KNMACH=: (<'\\'),&.>'OldDesktop';'LapHome1';'LapHome2';'WorkDesktop'
NB.* KMPFX: Known-Machine prefixs to use for backup dir.
KMPFX=: 'OD';'LH1';'LH2';'WDsk'
KNIDS=: 'MyOldHomeLogon';'MyHomeLogon';'MyHomeLogon';'MyWorkLogon'
NB. Home: old desktop, laptops; Work Main Machine (desk),
NB.* whoami: find what machine and user-id I'm on (assume Novell net command).
whoami=: 3 : 0
sess=. spawn 'net config workstation'
sess=. dsp&.><;._1 LF,sess-.CR NB. Vec of lines
sess=. ><;._1&.> ' ',&.>,sess NB. Mat of words
strs=. 'COMPUTER';<'USER' NB. Strings to key on
wh=. (<toupper&.>0{"1 sess)e.&.> <&.>strs NB. Which lines start with strings
wh=. ;b2i&.>(<(toupper&.>1{"1 sess)e. <'NAME')*.&.>wh
wh{2{"1 sess
NB.EG 'machine userid'=. whoami ''
)
Parallel Directory-Parsing Task Template
This is the template modified by the main routine and run as a distinct task for each top-level directory.
NB.* parallelParseDir.ijs: gather directory info in parallel.
usuUse=: 0 : 0
1!:44 dd=. 'C:\amisc\J\Parallel\ExampleProblems\DirInfo\'
load 'parallelParseDir.ijs'
6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=: PllDirInfoEG ''C:\'''
)
PllDirInfoEG=: 3 : 0
NB.* PllDirInfoEG: gather dirs' info accumulated in parallel.
'tmpd srcd flnms fldts flszs'=. pllParseDir y=. endSlash y
flszs=. ;flszs [ fldts=. cvtTS21Num&>fldts
flparent=. 0$~#flszs
dirdep=. _1,0$~#srcd [ dirnms=. y;srcd
vnms=. <;._1 ' FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'
for_ii. i. #tmpd do.
vals=. (3!:2)&.>fread&.>(<endSlash >ii{tmpd),&.>vnms,&.><'.DAT'
'flnms fldts flszs'=. (flnms;fldts;<flszs),&.>3{.vals
flparent=. flparent,(#dirnms)+>3{vals
dirnms=. dirnms,>4{vals
ddep=. (#dirnms)|dirnms i. (]{.~PATHSEP_j_ i:~])&.>dirnms
dirdep=. (ddep=i.#ddep)}ddep,:_1
end.
cleanupTempDirs 'PllDTmp'
flnms;fldts;flszs;flparent;dirnms;<dirdep
NB.EG 'flnms fldts flszs flparent dirnms dirdep'=. PllDirInfoEG 'C:\'
)
NB.* endSlash: ensure path has ending slash.
endSlash=: 13 : 'y,PATHSEP_j_#~PATHSEP_j_~:{:y'
SUBTASK=: <jpath '~Code/pllPDSub.ijs'
dq=: '"','"' ,~] NB.* dq: put double-quotes around y.
pllParseDir=: 3 : 0
NB.* pllParseDir: launch sub-tasks to parse each sub-directory under y.
if. 0=#y do. y=. 'C:\' end.
srcDirs=. jd dir '*',~y=. endSlash y
'flnms fldts flszs'=. <"1|:0 1 2{"1 jfi dir '*',~y
cleanupTempDirs 'PllDTmp'
tmpDirs=. (<'C:\Temp\PLLDTmp'),&.>":&.>i.#srcDirs NB. Don't end w/slash->escapes "
args=. SUBTASK,&.>(<' SRCDIR '),&.>dq&.>(<y),&.>srcDirs NB. Build command
args=. args,&.>(<' RESULTDIR '),&.>dq&.>tmpDirs NB. for each dir.
if. -.fexist BINPATH,'\PllPD.exe' do. (BINPATH,'\J.exe') fcopy BINPATH,'\PllPD.exe' end.
fork&>(<dq BINPATH,'\PllPD.exe'),&.>(<' -jijx '),&.>args
checkIfDonePPD 'PllPD'
tmpDirs;srcDirs;flnms;fldts;<flszs
NB.EG 'tmpd srcd flnms fldts flszs'=. pllParseDir 'C:\'
)
checkIfDonePPD=: 3 : 'while. 5<LF+/ . =shell ''pslist|egrep '',y do. wait 1 end.'
cleanupTempDirs=: 3 : 0
tmpdirs=. (<'C:\Temp\'),&.>jd dir 'C:\Temp\',y,'*'
shell&>(<'echo Y|del '),&.>tmpdirs,&.><'\*'
1[shell&>(<'rmdir '),&.>tmpdirs
)
0 : 0
NB.* Only invoke this if not loaded via interactive session.
if. (5{.&.>'-jijx';<'-rt') +./ . e. 5&{.&.>tolower&.>ARGV_z_ do.
'FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'=. PllDirInfoEG 'C:\'
(<'\Temp\') fileVar_WS_&.>'FLNMS';'FLDTS';'FLSZS';'FLPARENT';'DIRNMS';'DIRDEP'
'batfl cmds'=. buildBatFl_parseDir_ 12e6;'C:\Temp\Recent\'
shell batfl
end.
)
Mutual Exclusion
Here is the script implementing mutex by means of a semaphore file using the file-locking primitives in J. There are two scripts: the basic routines for mutex and a set of test cases to validate the correctness of these routines.
NB.* mutex.ijs: implement mutual exclusion via file locking.
coclass 'mutex'
require 'task' NB. Only for "spawn" in "whoami" - not necessary.
fopen=: [:1!:21< NB. Open file named by y.
fclose=: [:1!:22< NB. Close file name or number y.
flock=: 1!:31 NB. File number, index, length of region to lock
funlock=: 1!:32 NB. File number, index, length of region to unlock
whlocks=: 1!:30 NB. Which locks - 3-cols: file #, index, and length
whfiles=: 1!:20 NB. Which files - 2-cols: file #; name
getpid=: 2!:6 NB. Current process ID
qts=: 6!:0 NB. Timestamp -integers: Y M D h m s.s (millisec?)
LOCKFL=: 'Lock.fl' NB. Central hold file
LFINFO=: (i.0);'' NB. Hold file details while holding
NB.* getMyInfo: nice to have info identifying who has the lock and when.
getMyInfo=: 3 : '100{.(;'' '',~&.>whoami''''),":(getpid,qts)'''''
NB.* getHold: put hold on lock file.
getHold=: 3 : 0
fid=. fopen LOCKFL [ myinfo=. getMyInfo ''
if. -. fid e. ;0{"1 whlocks '' do. NB. If we don't have lock,
while. -.flock fid,0,100 do. wait 1 end. NB. keep waiting until we
myinfo fwrix_fldir_ fid;0 NB. get it; put ID info in
LFINFO=: fid (]{~[i.~[:;0{"1]) whfiles '' NB. file & update global.
end.
fid;myinfo
NB.EG 'fid myinfo'=. getHold '' NB. File ID, my ID info.
)
NB. funlock (>0{LFINFO) (]{~[i.~[:;0{"1]) whlocks ''
releaseHold=: 3 : 0
if. 0~:#>0{LFINFO do. 0*./ . =#&>LFINFO=: (i.0);'' [ fclose >0{LFINFO end.
)
NB.* getMUP: get Machine name, User ID, Process ID from my info string.
getMUP=: 13 : '3{.<;._1 '' '',y'
NB.* dsp: Despace - remove leading, trailing and redundant spaces.
dsp=: [:(#~(+.(1:|.(></\)))@(' '&~:))"1 (#~([:(+./\*. +./\.)' '&~:))"1
NB.* whoami: get machine and user ID - Windows only.
whoami=: 3 : 0
assert. 6-:9!:12 '' NB. Fail if not Windows
sess=. spawn 'net config workstation'
sess=. dsp&.><;._1 LF,sess-.CR NB. Vec of lines
sess=. ><;._1&.> ' ',&.>,sess NB. Mat of words
strs=. 'COMPUTER';<'USER' NB. Strings to key on
wh=. (<toupper&.>0{"1 sess)e.&.> <&.>strs NB. Which lines start w/ strings?
wh=. ;I.&.>(<(toupper&.>1{"1 sess)e. <'NAME')*.&.>wh
wh{2{"1 sess
NB.EG 'machine userid'=. whoami ''
)
Mutual Exclusion Test Cases
Here are routines to test various aspects of the mutex implementation.
NB.* mutex_TCs.ijs: test cases for mutex.ijs.
coclass 'mutexTest'
require '~Code/mutex.ijs'
coinsert 'mutex'
checkIfSelfLocked=: 3 : '(getMUP getMyInfo '''')-:getMUP y'
endSlash=: ],'\'-.{:
wait=: 6!:3
TC0=: 3 : 0
'fid myi'=. getHold ''
assert. -. 0*./ . =#&>LFINFO
releaseHold ''
assert. 0*./ . =#&>LFINFO
)
TC1=: 3 : 0
'fid myi'=. getHold ''
assert. checkIfSelfLocked fread fid
releaseHold ''
)
TC2=: 3 : 0
'fid myi'=. getHold ''
assert. checkIfSelfLocked myi
releaseHold ''
)
TC3=: 3 : 0
'fid0 myi0'=. getHold ''
'fid1 myi1'=. getHold ''
assert. fid0-:fid1
assert. checkIfSelfLocked myi0
assert. checkIfSelfLocked myi1
releaseHold ''
)
TC4=: 3 : 0
'fid myi'=. getHold ''
assert. fid=>0{LFINFO
assert. fid e. ;whlocks ''
assert. 3=#fid (]{~[i.~[:;0{"1]) whlocks ''
assert. LFINFO-:fid (]{~[i.~[:;0{"1]) whfiles ''
releaseHold ''
)
NB.* holdAndWait: hold file for a random time, then release.
holdAndWait=: 3 : 0
tm=. tm,qts'' [ 'fid myi'=. getHold '' [ tm=. ,:qts ''
rc=. *./checkIfSelfLocked &> myi;fread fid
rc=. rc,wait >:?y
releaseHold ''
rc;tm NB. If self-held, holding time; when asked and got hold.
)
NB.* holdsAndReleases: do number of holds and releases, waiting between each.
holdsAndReleases=: 3 : 0
'ni wl mhl'=. y NB. Number of iterations, wait length, max hold length.
rc=. 0 2$0 2;2 7$2
while. _1<ni=. <:ni do. wait wl [ rc=. rc,holdAndWait mhl end.
rc
)
NB.* startHRs: start holders and releasers as separate processes.
startHRs=: 3 : 0
if. 0=#y do.
args=. |.(10+|.i.10),. 2 2,2 3,3 2,3 3,2 4,4 2,4 4,2 5,5 2,:5 5
dd=. 'C:\amisc\J\Parallel\ExampleProblems\Holds\'
else. 'dd args'=. y end.
if. 0=#(1!:0)@<(]}.~[:-'\'={:) dd do. NB. Create dir if doesn't exist.
try. 1!:5 <dd catch. assert. 0 end. end.
start1HR"1 dd;"1 args
)
start1HR=: 3 : 0
'rundir arg'=. y
1!:44 rundir=. endSlash rundir [ svdir=. 1!:43 '' NB. Run in target dir.
basis=. fread jpath '~Code/mutex_TCs.ijs' NB. Get base code.
rndnm=. 8 (]{~[:?[$[:#]) 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
NB. Obfuscate rand name place-holder to avoid replacement here.
rplargs=. '{',&.>('RN';'arg';'rundir'),&.>'}'
addon=. (,rplargs,.rndnm;(":arg);rundir) stringreplace addon
if. fexist hrfl=. 'HR_',rndnm,'.ijs' do. ferase hrfl end.
(CR-.~basis,addon) fwrite hrfl NB. Create unique HR file.
exe=. (<'"',(jpath '~bin'),'\j.exe" -jijx "'),&.><endSlash rundir
fork&.>exe,&.>(<hrfl),&.>'"' NB. Run HR file.
1!:44 svdir
)
addon=: 0 : 0
varnm=. 'rctm{RN}'
(varnm)=: holdsAndReleases {arg}
(varnm)=: ('{RN}';(getpid ''),qts''),".varnm
(3!:1 (".varnm)) 1!:2 <'{rundir}',varnm,'.dat'
2!:55 ''
)
Checking Results of Mutual Exclusion Tests
The results of the startHRs tests need subsequently to be analyzed to ensure that no holds or releases overlap between separate processes, as shown here:
NB.* checkMutexOverlap: examine mutex hold and release times to find overlap.
NB.* Regularize vec of mats to 3-cols: time, ID, "H" (hold) or "R" (release),
NB. ordered by timestamp.
regMatz=: [:(] ,. ('H';'R')$~#) ([:<"1 [:,/[:>[:,/1{"1}.),.(<0 0){]
checkHRSeqc=: (2 {"1 ]) -: ('H';'R') $~ #
checkIDSeqc=: [:*./_2=/\1{"1]
egUse=: 0 : 0
dd=. 'C:\amisc\J\Parallel\ExampleProblems\HoldsWL2M100713\'
fls=. 0{"1 dir dd,'*.dat'
hldtms=. 3!:2&.>fread&.>(<dd),&.>fls
hh=. >,&.>/regMatz&.>hldtms
rc=. 0
assert. checkHRSeqc hh
assert. checkIDSeqc hh
rc=. 1
)