NYCJUG/Projects/Backup/SaveWorkOutlineInitiation
Outline of "Save Work" Goals and Project Framework
[Sat 03/05/2005 | 14:37:16.10]
NYC-JUGgernauts -
I apologize for not getting this to you yesterday like I said I would.
I've had trouble firming up assignments because I need feedback from people. Also, we need to collaborate on the design of this whole thing and address a couple of important issues, but here's what I've done so far.
"What whole thing?" you may ask. I've decided we should work on the project I'd like to call "Save Work", i.e. a backup utility for easily saving one's PC work. The reasons for choosing this project are as follows.
This utility is something that everyone needs and for software to be good, good design is paramount. Furthermore, good design evolves through use and refinement of software. So, if we work on a piece of software each of us actually uses frequently (you _did_ back up your work yesterday, didn't you?) we should be able to come up with something well-designed: intuitive, effective, and customizable.
As you may know, I have a crude, command-line version of this utility I myself use every day. I'm willing to offer my work as version 0.1 of this project. This brings up a couple of important issues: ego and ownership.
On ego: many years ago, I read one of the best books ever written on software development, called "The Psychology of Computer Programming" by Gerry Weinberg. One of the things I most remember about this book was Weinberg's observation of how touchy programmers are about their code, how much of their ego and self-esteem is tied up in their work. This can be a big impediment to collaborating on software.
So, when I offer my work for people to learn from and to criticize, I'd like to think that I do so in a professional, egoless way. If we decide my code is terrible and we have to tear it apart completely or throw it away and start from scratch, I won't take this personally. I hope everyone else shares this goal of professional detachment.
On ownership: I made the mistake, or had the good luck, of mentioning our project to a friend of mine who is a lawyer. He suggested we agree in advance on who owns the product of our joint effort and how we would like it to be used in the future. I, at first, demurred because I see this as primarily a self-educational, non-commercial venture - I expect all of you will agree with this. However, as he pointed out, this potentially leaves us open to legal manipulations later on. For instance, if we simply release our software for public use, someone could take what we did, decide to sell a commercial version of it, and threaten us with legal action if we want to use our own software!
For this reason, I'd like to draft a simple statement of purpose to which we can all agree. Since I'm offering this first bit of code, I'll just say that I do so with the expectation that I reserve my right to use it as I see fit in the future, including its commercialization or public release or any other purpose. Sorry to bring up this legal stuff but it's important we deal with it up front.
So, before I dump about 1000 lines of code on you, let me lay out the preliminary assignments. These are subject to negotiation which we will do when we meet on Tuesday.
Jim Korn - I'd like you to look through this code in the role of a librarian. You'll see that there is the main module "parseDir.ijs" and an ancillary module "WS.ijs". The former is the guts of the task: it parses a directory structure (in two completely different ways - this is really two versions of the code stuck together) and produces six global variables used to construct backups.
The ancillary module, "WS.ijs" is a standalone set of functions for saving J variables to file and retrieving them. Since I had sent this to the J-Forum, I included a number of utility functions that would normally reside in different files in order to make the module self-sufficient. These sorts of utilities need to be grouped logically, or, in some cases, disposed of and replaced with existing J software code I didn't know about when I wrote them.
Dan Bron - I see you in the role of tool-builder. I expect you will be the one to rip apart my code and re-build it or re-factor it from scratch. There will be a number of generally useful utilities that will come out of this project and you'll probably do a lot of the work on those. For instance, the "WS.ijs" I mentioned above accomplishes a little part of a more general task: saving information between sessions. This includes both specific things like lists of files a user wants to save regularly and more general things like, perhaps, the order in which a user typically visits various input screens.
As a preliminary task, I'd like to see what you can come up with for saving and repopulating information from a window. This should be extendable to retain some history of what's been entered in the fields. Maybe we could track what groups of entries belong with each other, e.g. on Monday through Thursday I do only cursory backups of what's most recently changed but on Friday I do a more comprehensive backup that includes some large databases I excluded earlier in the week.
Harteg Wariyar - you can work on some of the preliminary input windows. As a specific example, let's say we have a "Quick Save" screen for our most commonly performed backup. It looks something like this:
Source: _________ [where the underlines represent input boxes]
Target: _________
maybe with a few radio buttons, check boxes, or other input fields to specify a couple of common parameters like:
_ Today's work only
_ Compressed
Think about how you commonly would want to save your work. Also, to get you started on this, J has a handy form-design tool you can bring up by selecting, from your J session window, File->new ijs (or Ctl-N), then, in this new window, Edit->Form Editor.
John Randall - since I know you the least well of the group, I'm
unsure what to have you work on.
So, I'll just ask you to do what I'd like everyone else to do as well:
1) attempt to use the code I've sent - you'll see that, when you load it, a few lines will appear because these are the three or four most common things I do with it. You'll see the following, assuming we've saved that code as "parseDir.ijs" in our "user" directory:
load '~user/parseDir.ijs'
6!:2 '''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''=: getDirFlInfo_parseDir_ ''C:\'''
(<'\amisc\') fileVar_WS_&.>'FLNMS';'FLDTS';'FLSZS';'FLPARENT';'DIRNMS';'DIRDEP'
'batfl cmds'=. buildBatFl_parseDir_ 700e6;'C:\Temp\Recent\'
winexec ('cmd /c ',batfl);1
The first line after the "load" line (which you will have entered), times how long it takes to build the six global variables with all the information on my "C:\" drive. This has taken from 200 to 1000 seconds on the machines on which I've run it, depending on the machine speed and how much disk space is being used. If you can't wait that long, try it on a sub-directory only, e.g.
'FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'=: getDirFlInfo_parseDir_ 'C:\Temp'
without the timing.
2) Think about the different ways in which you would like to save your own work. Is it largely based on recently changed files, entire project directories, different code associated with particular databases? Are there files you would always include or exclude? How do you find things once you've backed them up? Is there a need to archive, i.e. back up old files then remove them from your disk?
3) Keep a written record of the pitfalls and difficulties you encounter when using J to accomplish your tasks. For instance, just now when I wanted to tell Harteg that there is a form editor, I couldn't remember where it was or how to invoke it. Eventually I stumbled on the proper entry in the "User Manual" (Help->User Manual->Form Editor).
Those of you who are J novices have the most valuable insights to offer here. If I had used the form editor regularly, I'd remember how to invoke it and have forgotten my difficulty finding it at first.
That's (almost) all from me for now - three code modules follow. I just noticed that the parseDir code loads one of my own modules called "dt" which refers to "dateTime.ijs". For those of you who don't know, there's a standard global variable called "PUBLIC_j_" which has two columns. The first is a short name and the second is a file to which the name refers. So, I can enter load 'dt' instead of load '~user/code/datetime.ijs' because, in my "startup.ijs", I've added a row to PUBLIC (which resides in the "j" locale) like this:
('dt';'~user/code/datetime.ijs')
We may have to come up with a few common modules everyone will add to their startup file. Hopefully I didn't miss any other dependencies and you all can run my code. Please let me know if this isn't the case.
Good luck!
Devon
File:ParseDir.ijs (The following code reflects the state of the code when this letter was sent; the attachment should be more up-to-date.)
NB.* parseDir.ijs: tools to parse text of directory listing.
NB.* parseDir.ijs: tools to parse text of directory listing.
NB.* buildBatFl: build .BAT file to create target dirs and copy files to them.
NB.* makeCopyCmds: make a list of DOS copy commands after creating needed dirs.
NB.* indicateSubdirs: from boolean selecting DIRNMS, indicate all subdirs.
NB.* excludeFiles: exclude designated files from list to back up.
NB.* dirDependencies: convert list of full paths to index vector form of tree
NB.* getDirFlInfo: get info on dirs and files starting at node specified.
NB.* ExcludeUsual: list of usual files and directories to exclude from backup.
NB.* initFlsDir: parse memory-mapped file of directory listing->file, dir info.
NB.* getInfo: get directory into into file, memory-map and parse it.
NB.* process1Subdir: parse single subdir entry->files, parent dirs as globals.
NB.* extract1SubdirList: get first full sub-directory listing out of many.
NB.* addPath: put new parent/child index in tree from text of "dir\subdirs..."
NB.* Tst0addPath_tests_: test adding path to index vec tree from text.
NB.* mcopyto: text of DOS .BAT file to do multiple copies.
NB. 'FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'=: getDirFlInfo_parseDir_ 'C:\'
smoutput '6!:2 ''''''FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP''''=: getDirFlInfo_parseDir_ ''''C:\'''''''
NB. (<'\amisc\') fileVar_WS_&.>'FLNMS';'FLDTS';'FLSZS';'FLPARENT';'DIRNMS';'DIRDEP'
smoutput '(<''\amisc\'') fileVar_WS_&.>''FLNMS'';''FLDTS'';''FLSZS'';''FLPARENT'';''DIRNMS'';''DIRDEP'''
NB. coclass 'parseDir'
NB. 'FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'=: FLNMS_base_;FLDTS_base_;FLSZS_base_;FLPARENT_base_;DIRNMS_base_;<DIRDEP_base_
NB. 'batfl cmds'=. buildBatFl_parseDir_ 700e6;'C:\Temp\Recent\'
smoutput '''batfl cmds''=. buildBatFl_parseDir_ 700e6;''C:\Temp\Recent\'''
NB. winexec ('cmd /c ',batfl);1
smoutput 'winexec (''cmd /c '',batfl);1'
coclass 'parseDir'
load 'jmf files dir dt'
coinsert 'base'
buildBatFl=: 3 : 0
NB.* buildBatFl: build .BAT file to create target dirs and copy files to them.
'szlim targ'=. y.
xclud=. excludeFiles targ
dtdord=. xclud-.~\:FLDTS
ss=. +/\dtdord{FLSZS
cutoff=. 1 i.~ss>szlim
ix=. cutoff{.dtdord
cmds=. (0{DIRNMS),makeCopyCmds targ;<ix NB. 0{DIRNMS gives disk to which
tmpd=. getTempDir '' NB. to copy from.
cmds v2f batfl=. tmpd,'CDMDCopy.bat'
batfl;<cmds
NB.EG 'batfl cmds'=. buildBatFl 700e6;'C:\Temp\Recent\'
)
makeCopyCmds=: 3 : 0
NB.* makeCopyCmds: make a list of DOS copy commands after creating needed dirs.
'targ ix'=. y.
fls=. ix{FLNMS
drs=. DIRNMS{~ix{FLPARENT
udrs=. ~.drs
ord=. /:udrs i. drs
ptn=. 1,2~:/\ord{udrs i. drs
fls=. ptn<;.1 ord{fls
newdrs=. (<targ),&.>udrs}.~&.>2+&.>udrs i.&.>':'
mknew=. (<'mkdir "'),&.>newdrs,&.>'"'
NB. e.g. mkdir C:\Temp\Recent\amisc\jsys\user\code
cdnext=. (<'cd "'),&.>udrs,&.>'"'
NB. e.g. cd c:\amisc\jsys\user\code
cmds=. ''
for_ii. i. #udrs do.
mm=. mkMCopyTo 10;(>ii{newdrs);ii{fls
NB. e.g. mcopyto "C:\Temp\Recent\global\fof" "bmkcorr.ijx" "targetbands.xls"...
cmds=. cmds,(ii{cdnext),(ii{mknew),,mm
end.
cmds
)
mkMCopyTo=: 3 : 0
NB. mkMCopyTo 10;targ;fls NB. -> mcopyto targ fl1 fl2..fl10 ; mcopyto targ fl11 fl12...
'npc targ fls'=. y. NB. Number per copy, target dir, file list
ptn=. (#fls)$npc{.1
fls=. ;&.>ptn<;.1 '"',&.>fls,&.><'" '
cmd=. (<'call mcopyto "',targ,'" '),&.>fls
)
indicateSubdirs=: 3 : 0
NB.* indicateSubdirs: from boolean selecting DIRNMS, indicate all subdirs.
xdix=. b2i y.
childxd=. xdix-.~b2i DIRDEP e. xdix
while. 0<#childxd do.
xdix=. ~.xdix,childxd
childxd=. xdix-.~b2i DIRDEP e. xdix
end.
xdix NB. Index into DIRNMS of all subdirectories.
NB.EG xdix=. indicateSubdirs DIRNMS e. 'c:\amisc';'c:\Program Files'
)
excludeFiles=: 3 : 0
NB.* excludeFiles: exclude designated files from list to back up.
targ=. y.}.~-PATHSEP_j_={:y.
sections=. '[ExcludeDirs]';'[ExcludeFiles]'
xu=. <;._1 LF,ExcludeUsual-.CR
xu=. xu#~&.>-.&.>+./\&.>(<'NB.')E.&.>xu NB. Exclude comments
xu=. xu#~0~:;#&.>xu
whsect=. >+./&.>sections E.&.>/ xu
secord=. /:sections i. ' '-.~&.>xu#~+./whsect
'xd xf'=: secord{(+./whsect)<;._1 xu
xd=: xd,<targ NB. Exclude target dir to avoid recursion.
whxd=. DIRNMS}.~&.>(2*;':'e.&.>DIRNMS)*;DIRNMS i.&.>':'
whxd=. (toupper&.>whxd)e. PATHSEP_j_,&.>toupper&.>xd
whxd=. (1) (indicateSubdirs whxd)}whxd
whxf=. (toupper&.>FLNMS)e. toupper&.>xf
xclud=. b2i whxf+.FLPARENT e. b2i whxd
NB. xclud is list of indexes into FLNMS=files to exclude.
)
dirDependencies=: 3 : 0
NB.* dirDependencies: convert list of full paths to index vector form of tree
NB. showing directory and subdirectories as parent-child relations.
odir=. y.
DIRDEP=: (#odir)$_1 NB. Dummy entry for 1st node: _1->no parent.
for_dc. i.#odir do.
cd=. PATHSEP_j_,~&.>dc{odir
len=. #>cd NB. Which prefixes match only current?
subs=. (;cd-:&.>len{.&.>odir)*.-.;PATHSEP_j_ e.&.>len}.&.>odir
DIRDEP=: subs}DIRDEP,:dc
end.
NB.EG dirDependencies 'c:';'c:\t1';'c:\t1\sb1';'c:\t2';'c:\t2\sb2';'c:\t1\sb3'
NB. _1 0 1 0 3 1 NB. Parent index for each input; _1 for no parent.
)
getDirFlInfo=: 3 : 0
NB.* getDirFlInfo: get info on dirs and files starting at node specified.
tree=. dirtree y.
DIRDEP=: dirDependencies DIRNMS=. dirpath y. NB. Dir dependency tree
fpths=. (}.~(<:@-@(i.&PATHSEP_j_)@|.))&.>0{"1 tree NB. Paths w/o filenames.
FLPARENT=: DIRNMS i. fpths NB. Parent dirs of files
assert. FLPARENT *./ . <#DIRNMS NB. Found all paths?
FLNMS=. ((>:@#)&.>fpths)}.&.>0}"1 tree NB. bare filenames
FLDTS=: ;100#.&.>3{.&.>1{"1 tree NB. Num yyyymmdd dates
max=. 5+24 60 60#.24 60 60 NB. Max secs/day + 5 for any leap seconds.
FLDTS=: FLDTS+max%~;(<24 60 60)#.&.>_3{.&.>1{"1 tree
FLSZS=: ;2{"1 tree NB. Very large files have signed sizes
FLSZS=: FLSZS+(2^32)*FLSZS<0 NB. so adjust them.
FLNMS;FLDTS;FLSZS;FLPARENT;DIRNMS;<DIRDEP
NB.EG 'FLNMS FLDTS FLSZS FLPARENT DIRNMS DIRDEP'=: getDirFlInfo_parseDir_ 'C:\'
)
NB.* ExcludeUsual: list of usual files and directories to exclude from backup.
ExcludeUsual=: 0 : 0
NB. Need to include a [regexp] exclusion section to apply to files, e.g. "saves-{d}*", "*~", etc.
NB. Directories (result of munge_dir) & source Disk & target Disk[:\dir] names.
[ExcludeDirs] NB. XD: list of dirs to exclude.
barrasys
CFGSAFE
Data\Aegis
DATA\SCANSOFTDOCS
Data\DailyPx
Data\RussPx
Data\US\DailyPxs
Documents and Settings
FrontPage Webs
GIR
Global\logs
Lotus
MSSQL7
My Documents
Notes
Program Files
Recycled
Recycler
WINNT
WINNT\profile
amisc\code\R
amisc\comm\Juno
amisc\incoming
amisc\pix
amisc\sound\HarryPotter
amisc\sound\MP3
amisc\sound\music
charlesriver
data\barralin
data\aegis
factset
program files\opera6
samples
system volume information
temp
[ExcludeFiles] NB. Files to exclude from any directory
EAFE.MDB
EAFE.ZIP
EAFEMSA2K.mdb
NB. Some DBs big enough to be done separately.
Office97.zip
SECURITY
SEDOLS.MDB
SOFTWARE
SYSTEM
SYSTEM.ALT
SecyValOld.MDB
ShortInt.MDB
Trade.MDB
USER0000.log
Valuation.MDB
Valuation.zip
default.dlf
eventlog.log
fold0000.frm
hiberfil.sys
pagefile.sys
pspbrwse.jbf
)
getInfo=: 3 : 0
NB.* getInfo: get directory into into file, memory-map and parse it.
NB. winexec 'cmd /C dir /A /S C:\ > C:\allfls2.dir';1
dirlstfl=. y.
if. 0=#dirlstfl do.
dirlstfl=. 'C:\allfls2.dir'
winexec ('cmd /C dir /A /S C:\ > ',dirlstfl);1
end.
initFlsDir dirlstfl
for_ix. i.<:#WHDB do.
ch=. extract1SubdirList WHDB{~ix+0 1
process1Subdir ch
end.
unmapall_jmf_ ''
)
process1Subdir=: 3 : 0
NB.* process1Subdir: parse single subdir entry->files, parent dirs as globals.
ch=. y.
thisdir=. (($DBSTR)}.ch) {.~ 1 i.~ LF E. ($DBSTR)}.ch
'isnew thisdn'=. addPath thisdir
ch=. (<;._1 ch)-.a: NB. break into lines; no empty lines
ch=. ch#~' '~:;{.&.>ch NB. Get rid of lines beginning with space.
ch=. <;._1&.>' ',&.>dsp&.>ch NB. break apart lines by spaces
NB. re-join any names with embedded spaces
ch=. |:>(3{.&.>ch),&.><&.>(}.@;)&.>(' '&,)&.>&.>3}.&.>ch
chtit=. 'DATE';'TIME';'SIZE';'NAME' NB. row titles for "ch"
whmootdirs=. (ch{~chtit i. <'NAME')e. ,&.>'.';'..'
NB. "SIZE" column has "<DIR>" indicator for directory, size for file.
whdir=. ((ch{~chtit i. <'SIZE')e. <'<DIR>')*.-.whmootdirs
addPath&.>(<thisdir,'\'),&.>whdir#ch{~chtit i. <'NAME'
whfls=. -.whdir+.whmootdirs
ch=. whfls#"1 ch
FLNMS=: FLNMS,ch{~chtit i. <'NAME'
FLPARENT=: FLPARENT,(+/whfls)$thisdn
FLSZS=: FLSZS,;n2j&.>(ch{~chtit i. <'SIZE')-.&.>','
FLDTS=: FLDTS,;DateTimeCvt &.>,&.>/' ',&.>ch{~chtit i. 'DATE';'TIME'
thisdn
)
extract1SubdirList=: 3 : 0
NB.* extract1SubdirList: get first full sub-directory listing out of many.
'st end'=. y.
ch=. CR-.~(st+i.>:end-st){DIRLSTFL
)
addPath=: 3 : 0
NB.* addPath: put new parent/child index in tree from text of "dir\subdirs..."
p2a=. <;._1 '\',y. NB. Path to add, e.g. 'C:\top\mid\bottom'
p2a=. p2a-.a:
isnew=. 0
NB. if. 0=#wh=. b2i DIRNMS e. 0{p2a do. NB. Completely new root
NB. DIRPARENT=: DIRPARENT,_1 NB. "Parent Index" of _1 means root node.
NB. wh=. <:#DIRNMS=: DIRNMS,0{p2a
NB. isnew=. 1
NB. else. wh=. {.wh#~_1=wh{DIRPARENT end. NB. or same old root
wh=. _1
for_nm. p2a do. NB. "wh" is parent index of current node...
if. 0=#wh2=. b2i DIRNMS e. nm do. NB. new subdir
DIRPARENT=: DIRPARENT,wh
wh=. {.<:#DIRNMS=: DIRNMS,nm NB. will be parent of next node, if any
isnew=. 1
else.
if. 0=#wh2=. wh2#~wh=wh2{DIRPARENT do. NB. Name exists but with
DIRPARENT=: DIRPARENT,wh NB. different parent.
wh=. {.<:#DIRNMS=: DIRNMS,nm
isnew=. 1
else. wh=. {.wh2 end. NB. Name exists with
end. NB. same parent
end.
isnew,wh NB. 0 if path was already here
)
Tst0addPath_tests_=: 3 : 0
NB.* Tst0addPath_tests_: test adding path to index vec tree from text.
coinsert 'parseDir base'
d0=. DIRNMS=: 'C:';'Aegis';'WINNT';'Web';'printers';'foo';'Web'
dp0=. DIRPARENT=: _1 0 0 3 4 0 0
assert. 0 2-:addPath 'C:\WINNT' NB. Shouldn't add it again.
assert. d0-:DIRNMS NB. Should not have changed
assert. dp0-:DIRPARENT NB. Should not have changed
td=. 'D:\foo\bar' NB. New path starting from new root
assert. 1 9-:addPath td NB. but with same-named sub as existing
assert. (d1=. DIRNMS)-:d0,<;._1 '\',td
assert. (dp1=. DIRPARENT)-:_1 0 0 0 3 4 0 0 _1 8 9
assert. 0 9-:addPath td NB. Shouldn't add it again.
assert. d1-:DIRNMS NB. Should not have changed
assert. dp1-:DIRPARENT NB. Should not have changed
assert. 1 10-:addPath 'C:\WINNT\foo'
1
)
NB.* mcopyto: text of DOS .BAT file to do multiple copies.
mcopyto=: 0 : 0
Rem MCopyTo.bat: Multiple COPY TO %1: copy %2, %3, %4, etc.
:START
If %1/==/ goto SHOWHOW
Set tmpnm=%1
:DO1
If %2/==/ goto BYEBYE
Copy %2 %tmpnm% > nul
Shift
Goto DO1
:SHOWHOW
Echo on
Rem MCOPY target source1 source2...sourceN
Echo off
Goto BYEBYE
:BYEBYE
Set tmpnm=
)
coclass 'base'
[1] Many of the following have now been replaced by more standard, J-supplied equivalents.
NB. --------- General utilities follow ---------------
dsp=: deb"1@dltb"1
isNum=: 3 : '0=1{.0$y.' NB. 1 if numeric arg
nameExists=: 3 : '0<:4!:0 boxopen y.' NB. 1 if var or fnc name exists.
dirExists=: 3 : '0~:#(1!:0)@< (-''\''={:y.)}.y.' NB. 0: dir does not
NB. exist; 1: exists.
chgCase=: 3 : 0
NB.* chgCase: change text to Lower or Upper given 'L' or 'U' as right
NB. argument or numeric arg: 1=uppercase, else lower.
'L' chgCase y. NB. Default conversion to lowercase.
:
if. 0=1{.0$x. do. x.=. (1={.x){'LU' end.
dir=. 2|'LUl' i. {. x.
alp=. 'abcdefghijklmnopqrstuvwxyz';'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
FROM=. >(-.dir){alp [ TO=. >dir{alp
y.=. ,y. [ shape=. $y.
wh=. y. e. FROM
ix=. FROM i. wh#y.
y.=. shape$(ix{TO) (wh#i.$wh)} y.
)
baseNum=: 4 : 0
NB.* baseNum: return number y. in base with digits x.: positive
NB. integers only.
len=. #alph=. x. [ num=. y.
if. isNum num do.
digits=. >:<.len^.1>.num
alph{~(digits$len)#:num
else. NB. Assume "number" is character -
len#.alph i. num NB. convert to base 10 number.
end.
NB.EG '0123456789ABCDEF' baseNum 255
NB.EG '0123456789ABCDEF' baseNum 'FF'
)
f2v=: 3 : 0
NB.* f2v: File to Vector: read file -> vector of lines.
vec=. l2v 1!:1 <y.
)
v2f=: 4 : 0
NB.* v2f: Vector to File: write vector of lines to file as lines.
if. -.nameExists 'EOL' do. EOL=. LF end.
(;x.,&.><EOL) 1!:2 <y.
NB.EG ('line 1';'line 2';<'line 3') v2f 'C:\test.tmp'
)
l2v=: 3 : 0
NB.* l2v: Lines to Vec: convert lines terminated by LF to vector elements.
vec=. y.
if. 0~:#y. do.
vec=. vec-.CR NB. Only single-char for line delim
vec=. vec,(LF~:{:vec)#LF NB. Ensure terminal line delimiter
whq=. ~:/\'"'=vec NB. Mask interiors of quoted strings
ptn=. LF=vec NB. Partition lines but
ptn=. ptn*.-.whq*.ptn NB. exclude quoted end-of-lines
vec=. ptn <;._2 vec NB. Make into vector of lines
end.
vec
)
getTempPath=: 'GetTempPath' win32api
getTempDir=: 3 : 0
NB.* getTempDir: get name of temporary directory.
td=. >2{getTempPath 256;256$' '
td=. td{.~td i. 0{a.
)
cocurrent 'base'
coinsert 'WS'
NB. Initiated by Devon H. McCormick, 20030711.
.+--------------------+.
NB.* datetime.ijs: date and time fns.
NB. load '~addons/sfl/sfldates.ijs' NB. Use C++ Std Template Lib for dates
load 'dates'
NB.* DateTimeCvt: Convert between DOS-style time (YYYYMMDD hh:mm[a
p]) & day.fraction.
NB.* DOSTime2DayFrac: DOS date & time MM/DD/YYYY hh:mm[a|p] -> YYYYMMDD.day-fraction.
NB.* DateAdd: limited mimic of VB DateAdd - DateAdd <unit to add>;date;num
NB.* waitUntil: wait until time y. in [abbreviated] TS form: YYYY MM DD [[hh] [[mm] [[ss]]]]
NB.* adt2sqldt: MS Access date time 'M/D/Y h:m:s'->#M/D/Y#
NB.* yy2yyyy: 2-digit to 4-digit year; <:50 is pivot.
NB.* cvtIfDate2Num: assuming slash means [m]m/[d]d/[yy]yy date, convert to YYYYMMDD.
NB.* calcEOMdates: calculate End-Of-Month dates starting in year 0{y. for
NB.* cvtAT: convert vector of timestamps from MSAccess DB to numeric date YYYYMMDD.
NB.* JDCNV: Julian day conversion from http://www.astro.washington.edu/deutsch-bin/getpro/library01.html?JDCNV
NB.* TSDiff: timestamp difference - return difference between 2 timestamps
NB.* TSAdd: add 2 timestamps - return sum of 2 timestamps
NB.* cvtDateTime2numDt: convert multiple (e.g. MS Access) char date-times, e.g.
NB.* dow: Day-of-week for year month day: 0=Sunday
NB.* toJulian: convert YYYYMMDD date to Julian day number.
NB.* toGregorian: convert Julian day numbers to dates in form YYYYMMDD
NB.* cvtY4M2D22YMD: "2002-06-11" -> 20020611 (numeric)
NB.* cvtMDY2Y4M2D2: convert char [m]m/[d]d/[yy]yy -> yyyymmdd (numeric);
NB.* cvtY4M2D22MDY: convert yyyymmdd (numeric) -> [m]m/[d]d/yyyy (char)
NB.* addMos: add N months to date in form Y4M2 (YYYYMM): YYYYMM addMos N.
moAbbrevs=: 'Jan';'Feb';'Mar';'Apr';'May';'Jun';'Jul';'Aug';'Sep';'Oct';'Nov';<'Dec'
baseDPM=: 31 28 31 30 31 30 31 31 30 31 30 31 NB. Base # of days per month.
weekDayAbbrevs=: 'Sun';'Mon';'Tue';'Wed';'Thu';'Fri';<'Sat'
DateTimeCvt=: 3 : 0
NB.* DateTimeCvt: Convert between DOS-style time (YYYYMMDD hh:mm[a
p]) & day.fraction
NB. (single number=<Julian day number>.<day fraction>).
if. ' '={.0$,y. do. NB. Character ,T (Bnumber
'dt tm'=. <;._1 ' ',dsp ,y. NB. Seperate date and time
dt=. (toJulian cvtMDY2Y4M2D2 dt)+DOSTime2DayFrac tm
else. dt=. (cvtY4M2D22MDY toGregorian <.y.),' ',DOSTime2DayFrac 0|y.
end.
dt
)
DOSTime2DayFrac=: 3 : 0
NB.* DOSTime2DayFrac: DOS time hh:mm[a|p] -> 0.day-fraction or vice-versa.
'DOS' DOSTime2DayFrac y.
:
spd=. 5+*/24 60 60 NB. Seconds per day + 5 for any leap seconds.
targ=. 'U' chgCase x. NB. Target direction->'DOS' or 'DF' (day fraction)
if. -.isNum y. do. targ=. 'DF' end.
if. targ-:'DOS' do.
df=. (12|0{df) 0}df=. }:0 60 60#:<.0.5+spd*1|y.
df=. ((0{df)+12*0=0{df) 0}df NB. 5 leap-seconds
df=. (}:;(2 lead0s&.>df),&.>':'),(0.4999<:1|y.){'ap' NB. means noon<0.5.
else. df=. ;n2j&.><;._1 ':',y.#~y. e. ' 0123456789:' NB. '3:25p'->3 25
df=. ((0{df)+12*(12~:0{df)*.'pP'e.~{:y.) 0}df
df=. ((0{df)-12*(12= 0{df)*.'aA'e.~{:y.) 0}df
df=. 0.999999<.(60*60#.df)%spd
end.
df
)
monthAdd=: 4 : 0
mobase=. 0 12 31
100#.todate todayno 0 1 0+mobase#:(31*y.)+mobase#.0 _1 0+0 100 100#:x.
)
DateAdd=: 3 : 0
NB.* DateAdd: limited mimic of VB DateAdd - DateAdd <unit to add>;date;num.
NB. Date in form YYYYMMDD.
ts=. i. 0 [ units=. 'YMDW' NB. Units are Year, Month, Day, Week.
'unit dt addnum'=. y.
assert (unit=. {.toupper unit) e. units
dpu=. (372 31 1 7){~units i. unit NB. 372=12*31: provisional "year"
mobase=. 0 12 31
assert. dt>18000101 NB. todayno only goes back to this date
dt=. 0 100 100#:dt
select. unit
case. 'D';'W' do.
dt=. 100#.todate (dpu*addnum)+todayno dt
case. 'M';'Y' do.
fakedayno=. mobase#.0 _1 0+dt
dt=. 100#.todate todayno 0 1 0+mobase#:fakedayno+dpu*addnum
end.
assert. dt>18000101 NB. todayno only goes back to this date
NB. dtdifs=. 2-/\todayno&>(<0 100 100)#:&.> dtsa=. DateAdd&>(<'D';20050101),&.><&.>i:_10000
NB. 1 *./ . = dtdifs
NB. dtdifs=. 2-/\todayno&>(<0 100 100)#:&.> dtsa=. DateAdd&>(<'W';20050101),&.><&.>i:_10000
NB. 7 *./ . = dtdifs
)
DateAddOLD=: 3 : 0
NB.* DateAdd: limited mimic of VB DateAdd - DateAdd <unit to add>;date;num
ts=. i. 0 [ units=. 'YMDW' NB. Units are Year, Month, Day, Week.
'unit dt add'=. y.
unit=. {.'U' chgCase unit
assert unit e. units
if. 'W'=unit do. add=. 7*add [ unit=. 'D' end.
summand=. 6{.add (units i. unit)}(#units)$0
ts=. 100#.3{.>1{summand TSAdd 6{.0 100 100#:dt
NB.EG DateAdd 'D';20040524;_45
)
waitUntil=: 3 : 0
NB.* waitUntil: wait until time y. in [abbreviated] TS form: YYYY MM DD [[hh] [[mm] [[ss]]]]
6!:3 (0>.>0{y. TSDiff 6!:0 '')
NB.EG 'Happy New Year!' [ waitUntil 2004 12 31 23 59 59
)
adt2sqldt=: (1&|.)@('##'&,)@((]i.&' '){.]) NB.* MS Access date time 'M/D/Y h:m:s'->#M/D/Y#
yy2yyyy=: 3 : 0
NB.* yy2yyyy: 2-digit to 4-digit year; <:50 is pivot.
if. -.isNum yy=. y. do. yy=. _".yy end.
yy+(yy<:50){1900 2000
)
cvtIfDate2Num=: 3 : 0
NB.* cvtIfDate2Num: assuming slash (or dash) means [m]m/[d]d/[yy]yy date,
NB. -> character yyyymmdd.
if. +./'-/' e. y. do.
dlim=. {.y.-.'0123456789' NB. Assume delimiter is 1st non-numeral.
y.=. ":dlim cvtMDY2Y4M2D2 y.
end.
y. NB. Return input if not date.
)
cvtTS2Access=: 3 : 0
NB. cvtQTS2Access: convert J timestamp to MS Access date:
NB. YYYY MM DD hh mm ss.ms -> MM/DD/YYYY hh:mm:ss
if. 0=#y. do. dt=. qts ''
else. dt=. 6{.y. end.
dt=. (1 2 0 3 4 5 ){dt NB. Reorder YMD->MDY.
dt=. (roundNums 5{dt) 5}dt NB. Round to whole seconds.
dt=. <("1)2 3$dt NB. Separate date & time.
dt=. (>":&.>&.>dt),&.>("1 0)'/:' NB. M D Y ->"M/D/Y/" & h m s ->"h:m:s:"
dt=. (_3{.&.>(<'00'),&.>1{dt) 1}dt NB. Pad time w/leading 0s.
dt=. }:&.>dsp&.><"1 ;"1 dt NB. Drop excess trailing delimiters,
dt=. }:;dt,&.>' ' NB. join date & time -> "M/D/Y hh:mm:ss"
NB.EG cvtTS2Access 6!:0 ''
)
normAccessDate=: 3 : 0
NB. normAccessDate: convert MS Access dates with usually useless timestamp
NB. to simple date, e.g. YYYY-MM-DD hh:mm:ss -> (numeric) YYYYMMDD.
if. isNum 0{y. do. NB. YYYYMMDD -> 'MM/DD/YYYY'
xp=. 1 1 1 1 0 1 1 0 1 1 0
tmp=. y.
dts=. ''
while. 0<#tmp do.
dd=. }.4|.xp exp ": >0{tmp NB. 20030524 -> 05 24 2003
dts=. dts,<'/' (b2i -.}.4|.xp)}dd NB. -> 05/24/2003
tmp=. }.tmp
end.
else. NB. YYYY-MM-DD hh:mm:ss -> YYYYMMDD
dts=. boxopen y.
dts=. ;0".&.>'-'-.~&.>dts {.~ &.> dts i. &.> ' '
end.
dts
)
cvtAccessIODates=: 3 : 0
NB. cvtAccessIODates: stupid MS Access exports dates in form YYYY-MM-DD that
NB. it won't accept for import: convert between this and MM/DD/YYYY form.
y.=. y.{.~y. i. ' ' NB. 2002-01-23 12:34:46 -> 2002-01-23
if. '-' e. y. do. NB. Dash delimiter: YYYY-MM-DD -> MM/DD/YYYY
y.=. (y.='-')}y.,:'/'
dt=. y.
wh=. >:dt i. '/' NB. Pull off [YY]YY/
dt=. (wh}.dt),_1|.wh{.dt NB. YYYY/MM/DD -> MM/DD/YYYY
y.=. dt,y.}.~y. i. ' ' NB. Put date back on first.
elseif. '/' e. y. do. NB. Slash delimiter: MM/DD/YYYY -> YYYY-MM-DD
y.=. (y.='/')}y.,:'-'
if. ' ' e. y. do.
dt=. y.
wh=. ->:(|.dt) i. '-' NB. Pull off "-[YY]YY"
dt=. (wh}.dt),~1|.wh{.dt NB. MM-DD-YYYY -> YYYY-MM-DD
y.=. dt,y.}.~y. i. ' ' NB. Put date back on first.
end.
end.
y.
)
calcEOMdates=: 3 : 0
NB.* calcEOMdates: calculate End-Of-Month dates starting in year 0{y. for
NB. 1{y. years. Account for weekends but not any holidays.
'styr numyrs'=. y.
eomdts=. (styr+i.numyrs),&.>/(i.12) NB. Start with YYYY MM
eomdts=. eomdts,&.>(1{&.>eomdts) d14&.>0{&.>eomdts NB. ,last calendar day.
eomdts=. eomdts+&.><0 1 0 NB. Origin-1 months
dd=. dow&.>eomdts NB. Days-of-week
weekdayshift=. dd{&.><2 0 0 0 0 0 1 NB. Move back 2 days for Sunday,
eomdts=. eomdts-&.>(<0 0),&.>weekdayshift NB. or 1 for Saturday.
NB. Returns (numyrs,12)$<YYYY MM DD date numbers.
NB.EG eomdts=. calcEOMdates 2003 5
)
cvtAT=: 3 : 0
NB.* cvtAT: convert vector of timestamps between MS Access DB form
NB. 'MM/DD/YYYY hh:mm:ss' and numeric date YYYYMMDD.
if. isNum y. do.
cvtTS2Access 0 100 100#:y.
else. cvtDateTime2numDt y. end.
)
JDCNV=: 3 : 0
NB.* JDCNV: Julian day conversion from http://www.astro.washington.edu/deutsch-bin/getpro/library01.html?JDCNV
'yr mn day hr'=. y.
NB. yr = long(yr) & mn = long(mn) & day = long(day);Make sure integral
L=. (mn-14)%12 NB. In leap years, -1 for Jan, Feb, else 0
julian=. day-32075+(1461*(yr+4800+L)%4)+(367*(mn - 2-L*12)%12) - 3*((yr+4900+L)%100)%4
julian=. julian+(hr%24)-0.5
)
TSDiff=: 4 : 0
NB. Still has bug: will give year, month, or day of 0.
NB.* TSDiff: timestamp difference - return difference between 2 timestamps
NB. in form Y M D h m s= years months days hours minutes seconds.
timebase=. 0 12 31 24 60 60
'x. y.'=. (#timebase){.&.>x.;<y. NB. Pad trailing with 0s if missing.
secs=. ;(<timebase)#.&.>x.;<y.
swaphilo=. 0
if. </secs do. swaphilo=. 1 [ secs=. |.secs end.
diff=. timebase#: ds=. -/secs
if. swaphilo do.'ds diff'=. -&.>ds;diff end.
ds;diff NB. Difference in both seconds and in Y M D h m s.
NB.EG (6!:0 '') TSDiff ts [ 6!:3,10 [ ts=. 6!:0 '' NB. 10-sec delay
)
TSAdd=: 4 : 0
NB. Still has bug: will give year, month, or day of 0.
NB.* TSAdd: add 2 timestamps - return sum of 2 timestamps
NB. in form Y M D h m s= years months days hours minutes seconds.
timebase=. 0 12 31 24 60 60
'x. y.'=. (-#timebase){.&.>x.;<y. NB. Pad leading with 0s if missing.
secs=. ;(<timebase)#.&.>x.;<y.
sum=. timebase#: ds=. +/secs
ds;sum NB. Difference in both seconds and in Y M D h m s.
NB.EG (6!:0 '') TSAdd 10 15 NB. 10 min, 15 seconds from now.
)
showdate=: 3 : 0
NB. showdate: show given, or current if arg '', date & time in standard form.
NB. Can show 60 in seconds position because rounded to nearest second and
NB. carrying because of rounding gets too complicated.
y.=. ,y. NB. Must be vector.
if. 0=#y. do. y.=. 6!:0 '' end. NB. Arg is timestamp: YYYY MM DD hh mm ss
dt=. }:;(":&.>3{.y.),&.>'/'
if. 3=#y. do. tm=. '' else.
tm=. }:;(_2{.&.>(<'00'),&.>' '-.~&.>2j0":&.>3}.y.),&.>':' NB. Leading 0s
end.
dt,((0~:#tm)#' '),tm
NB.EG showdate 1959 5 24 8 9 0
NB.EG showdate '' NB. Current date and time
NB.EG >}.<;._1 ' ',showdate '' NB. Time only
NB.EG showdate 1992 12 16 NB. Date only
)
cvtDateTime2numDt=: 3 : 0
NB.* cvtDateTime2numDt: convert multiple (e.g. MS Access) char date-times, e.g.
NB. "1995-02-28 00:00:00", to numeric dates, e.g. 19950228.
dt=. boxopen y.=. ,y.
dlim=. ('-' e. >0{dt){'/-'
dt=. dt {.~ &.> ' 'i.~ &.> dt
dt=. ;dlim cvtY4M2D22YMD dt
NB.EG cvtDateTime2numDt '1995-05-31 12:34:56';<'1995-06-30 00:00:00'
)
cvtD2M3Y22MSDSY=: 3 : 0
NB. cvtD2M3Y22MSDSY: e.g. 31-Jul-02 -> 7/31/02.
if. 0~:#y. do.
y.=. ,y.
dlim=. {.~.y. -. '0123456789' NB. Assume delimiter whatever not num.
dt=. <;._1 dlim,y.
mo=. >:('U' chgCase &.> moAbbrevs) i. <'U' chgCase >1{dt
}:;(":&.>mo;0 2{dt),&.>'/'
else. y. end.
)
m11=: 0: ~:/ .= 4 100 400"_ |/ ] NB. Is y a leap year?
m12=: 28"_ + m11@] NB. Number of days in February of year y
d13=: 31"_ - 2: | 7: | [ NB. days in month x, not = 1
d14=: d13`m12@.([=1:) NB. Number of days in month x of year y
m22=: -/@:<.@(%&4 100 400)"0 NB. # of leap days in year yyyy (Clavian corr.)
m21=: >:@(365&* + m22)@(-&1601) NB. # of New Year's Day, Gregorian year y; m21 1601 is 1.
dowsoy=: 7&|@m21 NB. Day of week for start of year YYYY: 0=Sunday
dow=: 3 : 0
NB.* dow: Day-of-week for year month day: 0=Sunday
7|+/(dowsoy 0{y.),(;(i. 0>. <:1{y.) d14 &.> 0{y.),<:2{y.
)
isLeapYr=: 0: ~:/ .= 4 100 400"_ |/ ]
toJulian=: 3 : 0
NB.* toJulian: convert YYYYMMDD date to Julian day number.
dd=. |: 0 100 100 #: y.
mm=. (12*xx=. mm<:2)+mm=. 1{dd
yy=. (0{dd)-xx
nc=. <.0.01*yy
jd=. (<.365.25*yy)+(<.30.6001*1+mm)+(2{dd)+1720995+2-nc-<. 0.25*nc
NB.EG toJulian 19590524 19921216
)
toGregorian=: 3 : 0
NB.* toGregorian: convert Julian day numbers to dates in form YYYYMMDD
NB. (>15 Oct 1582). Adapted from "Numerical Recipes in C" by Press,
NB. Teukolsky, et al.
igreg=. 2299161 NB. Gregorian calendar conversion day (1582/10/15).
xx=. igreg<:ja=. <. ,y.
jalpha=. <.((<.(xx#ja)-1867216)-0.25)%36524.25
ja=. ((-.xx) expand (-.xx)#ja)+xx expand (xx#ja)+1+jalpha-<.0.25*jalpha
jb=. ja+1524
jc=. <.6680+((jb-2439870)-122.1)%365.25
jd=. <.(365*jc)+0.25*jc
je=. <.(jb-jd)%30.6001
id=. <.(jb-jd)-<.30.6001*je
mm=. mm-12*12<mm=. <.je-1
iyyy=. iyyy-0>iyyy=. (<.jc-4715)-mm>2
gd=. (10000*iyyy)+(100*mm)+id
)
NB. Date fns: convert between different date formats.
cvtY4M2Mos=: 3 : '12#.(0 100#:y.)-0 1' NB. 200011 -> 24011
cvtMo2Y4M2=: 3 : '100#.0 1+0 12#:y.' NB. 24010 -> 200010
cvtMDY2Y4M2=: (".)@((_4&{.),(2&{.)) NB. "11/30/1999" -> 199911 (numeric)
cvtY4M2D22YMD=: 3 : 0
NB.* cvtY4M2D22YMD: "2002-06-11" -> 20020611 (numeric)
'-' cvtY4M2D22YMD y.
:
if. 0~:#y. do.
dlim=. x.
dts=. (dts i. &.>' '){.&.>dts=. boxopen y.
dts=. ((5}.&.>dts),&.>dlim),&.>4{.&.>dts
dts=. dlim cvtMDY2Y4M2D2 &.> dts
else. y. end.
NB.EG dts=. cvtY4M2D22YMD '1959-5-24';'1992-12-16';<'1959-05-06'
)
cvtMDY2Y4M2D2=: 3 : 0
NB.* cvtMDY2Y4M2D2: convert char [m]m/[d]d/[yy]yy -> yyyymmdd (numeric);
NB. x. is optional delimiter character if not '/'.
'/' cvtMDY2Y4M2D2 y. NB. Assume slash delimiter.
:
if. 0~:#y. do.
ptn=. <;._1 ((x.~:0{y.)#x.),y.
ymd=. ".&.>ptn
if. 100 > year=. >2{ymd do.
if. 80 > year do. year=. year+2000
else. year=. year+1900 end.
ymd=. (<year) 2}ymd
end.
100#.;_1|.ymd
else. y. end.
NB. cvtMDY2Y4M2D2&.>'3/14/01';'5/9/99';'2/6/1999';'12/16/79';'2/18/80'
NB. +--------+--------+--------+--------+--------+
NB. |20010314|19990509|19990206|20791216|19800218|
NB. +--------+--------+--------+--------+--------+
)
cvtY4M2D22MDY=: 3 : 0
NB.* cvtY4M2D22MDY: convert yyyymmdd (numeric) -> mm/dd/yyyy (char)
NB. x. is optional delimiter character if not '/'.
'/' cvtY4M2D22MDY y. NB. Assume slash is delimiter.
:
if. 0~:#y. do.
y.=. ''$y.
dlim=. x.
dt=. (":&.>1|.0 100 100#:y.),&.>1 1 0{.&.>dlim
;_3 _3 _4{.&.>(<'0000'),&.>dt NB. Pad with leading 0s
else. y. end.
NB.EG cvtY4M2D22MDY 19730131
)
addMos=: 3 : 0
NB.* addMos: add N months to date in form Y4M2 (YYYYMM): YYYYMM addMos N.
(100 #. 2{.6!:0 '') addMos y. NB. Assume current date if none given.
:
1+100#.0 12#:y.+12#.0 100#:x.-1
NB.EG 199909 addMos 16
NB.EG 200101
)
NB. From: "RISKS List Owner" <risko@csl.sri.com>
NB. Date: Sat, 27 Jul 2002 12:41:31 PDT
NB. Subject: [risks] Risks Digest 22.18
NB. To: risks@csl.sri.com
NB.
NB. Date: Wed, 24 Jul 2002 18:37:22 +0100
NB. From: John Stockton <spam@merlyn.demon.co.uk>
NB. Subject: Possible day-of-week error - Zeller
NB.
NB. Algorithms for determining the day-of-week from year-month-day -
NB. whether or not truly Zeller's - can, for certain dates, compute a negative
NB. number mod 7, which does not yield the desired result. Zeller himself
NB. dealt with this.
NB.
NB. Tests using "current" dates in the later 1900's would not have seen
NB. this problem.
NB.
NB. A good test date is 2001-03-01 (1st March 2001); the algorithm can
NB. easily be run manually.
NB.
NB. The problem has been seen, for example, in C code in an Internet draft.
NB.
NB. Those whose systems do suitable run-time checking may already have
NB. discovered the problem.
NB.
NB. John Stockton, Surrey, UK. http://www.merlyn.demon.co.uk/programs/
NB. Dates: miscdate.htm moredate.htm js-dates.htm pas-time.htm critdate.htm
NB. etc.