User:Marshall Lochbaum/Parallelize
This code gives a relatively clean way to parallelize "embarassingly parallel" problems--that is, those which are represented by verbs u with u-:u"_1. It can be run simply as u parallelize n y, and will fork off n tasks to run equally spaced sections of y. If one of these returns an error, the error is spat back using debug foreigns, and another thread is spawned to kill off the remaining processes as they terminate.
If it is necessary to load scripts before execution of u, they can be set with setloadfiles, which has the same arguments as load.
Tested on Linux and previous versions worked on Mac and Windows. Windows execution will give console windows that pop up for the duration of the computation. Currently does not support dyadic evaluation.
load 'strings files task' fork=: ([: 2!:1 '(',,&')&')`(fork_jtask_)@.IFWIN JEXE=: jpath IFWIN{:: '~bin/jconsole';'~bin\jconsole.exe' WAITTIME=: 0.05 NB. read, write a boxed J noun oread =: [: <@(3!:2)@:(1!:1)"0 boxopen owrite =: (3!:1@>@[ 1!:2 ])"0 boxopen getfname =: ((jpath '~user/')&,) : (((jpath '~user/') , ({.~i.&'.')@[) , ":@] , (}.~i.&'.')@[) &.> NB. (u parallelize n) y NB. assumes (u -: u"_1) y NB. divides y into n parts and executes u on them in parallel. parallelize=:2 :0 parts=. (</.~ [:<.n* (%~i.)@#) y 'inputs flags outputs kernels'=. <"_1|: files=. |: ('input';'flag';'output';'kernel.ijs') getfname/ i.n parts owrite inputs '0' 1!:2"0 flags (u buildtemplate) runkernels files result=.$0 while. +./ '1'~: read=. {.@fread"0 flags do. if. '2'e.read do. err =. ; oread (read i.'2'){outputs ferase (read i.'2'){files cleanup (read='0') # files 13!:8&>/ err end. 6!:3 WAITTIME end. result=. ; oread outputs ferase files result ) NB. run first y kernels runkernels=:4 :0"_ 1 'input flag output kernel'=. y sentence=. x rplc '{input}';input;'{flag}';flag;'{output}';output sentence fwrite kernel fork JEXE,' "',kernel,'"' ) NB. built a template string to use for runkernels. NB. assumes u is monadic. buildtemplate=:1 :0 if. _1= 4!:0 <'loadfiles' do. loadfiles=:'' end. if. 3 = 4!:0 <'u' do. u=.5!:5<'u' end. template rplc '{loadfiles}';loadfiles;'{verb}';u ) template =: ('\)';')') rplc~ 0 :0 {loadfiles} f=: {verb} 3 :0 '' try. '{flag}' (1!:2<)~ '1'[ '{output}' (1!:2<)~ f&.(3!:2) 1!:1 <'{input}' catch. '{flag}' (1!:2<)~ '2'[ '{output}' (1!:2<)~ 3!:1 (13!:12'') ; (13!:11'') end. \) 2!:55 ]0 ) cleanup =: 3 :0 c =. >getfname <'cleanup.ijs' c fwrite~ cleanupscript rplc '{files}';(5!:5<'y');'{WAITTIME}';(5!:5<'WAITTIME');'{c}';c fork JEXE,' "',c,'"' ) cleanupscript =: ('\)';')') rplc~ 0 :0 (3 :0) {files} try. while. #y do. for_i. y do. 'input flag output kernel'=.i if. '0'~:{.1!:1 <flag do. 1!:55 :: _1:"0 i y=.y-.i end. end. (6!:3) {WAITTIME} end. catch. 1!:55 :: _1:"0 y end. \) 1!:55 :: _1: <'{c}' 2!:55 ]0 ) NB. y is list of files to load (arguments to the verb load). setloadfiles=:3 :0 loadfiles=: 'load ''',y,'''' i.0 0 )
Contributed by Marshall Lochbaum.