David Lambert/dwldb and cryptocurrency

From J Wiki
Jump to navigation Jump to search

Warning: the file paths are oriented toward my personal configuration, including linux.

These two files are implemented as classes. The database. appendcol automatically resizes as needed. There is not an "append rows" functionality implemented. When open rw, the database first makes a tar archive of the columns. Otherwise, error recovery is not implemented.

 host 'cd {} && tar czf coldb.tar.z *.col' format~ <DIR
Note'00README'

 To the Hands
 Caroline Shaw

 (3333$a)appendcol__D'a'[D=:'/tmp'conew'dwldb'[load'~/j.ijs'

 setup:  make a directory, install files  fieldName.col  using createjmf_jmf_
 load this file into a session

 all files in the directory with extension name.col are mapped as columns.
 VALID is a reserved column name
 
 require'~lambertdw/src/j/dwldb.ijs'
 db=: DIRECTORY conew dwldb

 Setup for 1st use.
  put *.col files into a directory.  Each of these will be a database field.
  for example, date.col will map to the variable date.
  These files are created with
  require'jmf'
  createjmf_jmf_'date.col';666
 subsequent uses
  create takes arguments dirpath ;  readonly flag
  The default flag is 0, ie. rw.
  It maps all the columns, storing the variables into the base locale.
)

coclass'dwldb'

require'jmf'

Filter=:(#~`)(`:6)

Dyad=: [: :
Monad=: :[:

ls=: 1!:0@boxopen Monad
read=:1!:1@boxopen Monad
write=:(1!:2 boxopen)~ Dyad
append=:(1!:3 boxopen)~ Dyad
mkdir=:1!:5@boxopen Monad
open=:1!:21@boxopen Monad
close=:1!:22@boxopen Monad
pwd=:1!:43 Monad
cd=:1!:44@boxopen Monad
rm=:1!:55@boxopen Monad
host=:2!:0 Monad
spawn=:2!:1 Monad
getenv=:2!:5 Monad
getpid=:2!:6 Monad
exit=:2!:55 Monad

nameClass=:4!:0@boxopen
erase=: [: 4!:55 ;: ::]

format=: ;@:((a: , ":&.>)@:[ ,. '{}'&([ }.L:_1@:(E. <;._1 ]) ,)@:])
fail=: format assert 0:

rank=:#@$
showmap=: ,:^:(2>rank)@:showmap_jmf_

create_dwldb_=: 3 : 0
 NB. default is open for rw
 'dir ro'=. 2 {. (boxopen y) , < 0
 'readonly flag must be 0 or 1'assert ro e. 0 1
 DIR=: PATHJSEP_j_ ,~^:(~:{:) jpath dir
 a=. ls }: DIR
 'expected 1 match' assert 1 = # a
 'not a directory' assert 'd' = {. (0 _1) {:: a
 opendb ro
)

valid_columns=: 3 : 0
 NB. y is the desirable nameclass, 0 for existing nouns, _1 for unused names
 a=. ([: -. (1 e. '_'&e.)&>)Filter {."1 ls DIR , '*.col'
 (_2 < ([: nameClass ('_' , '_' ,~ > COCREATOR) ,~ _4&}.)&>)Filter a
)

closedb=: ([: unmap_jmf_ _4&}.)&>@:valid_columns

destroy=: 3 :0
 closedb''
 coerase y
)

opendb=: 3 :0 :.closedb
 NB. y is readonly flag
 if. y = 0 do. NB. make backup
  host 'cd {} && tar czf coldb.tar.z *.col' format~ <DIR
 end.
 files=. valid_columns _1
 ([: map_jmf_ ('' ; y) ,~ _4&}. ; DIR&,)&> ([([:echo 'opening '&;)) files
)

appendcol=: 4 :0 ::resize
 name=. fullname_jmf_ y
 if. # name~ do. (name)=: name~ , x else. (name)=: x end.
 EMPTY
)

resize=: 4 :0
 name=. fullname_jmf_ y
 map=. showmap''
 ({. map) =. map&(((i.~ {."1)~ <) { [) ::(fail&'{} not mapped') name
 tempname=. DIR,'growth_here'
 createjmf_jmf_ tempname ; {. fsize + msize
 map_jmf_'growth_here';tempname
 growth_here_name=: fullname_jmf_ 'growth_here'
 (growth_here_name)=: name~
 unmap_jmf_&> y ; growth_here_name
 host 'mv {} {}' format~ (;  ('bak' ,~ _3&}.)) fn
 host 'mv {} {}' format~ tempname ; fn
 map_jmf_  y ; fn
 x appendcol y
)

Example using dwldb, uses wget to load the json data from the www.coindesk.com/price/, making some gross assumptions that they won't change their web page format.

coclass'cryptocurrency'

Note'use'
 $ ~/downloads/installs/j903/jconsole.sh
       if the ro flag is 0, update the database, then open read only.
       if the ro flag is 1, just open the db read only.
       roflag conew
    c=:0 conew'cryptocurrency'[load'/home/lambertdw/src/j/cryptocurrency/cryptocurrency.ijs'[13!:0[1
    $@".L:0 ;:'HIGH OPEN VALUE MARKETCAP CIRCULATINGSUPPLY RETURNS24 TIMEPERIOD TS TO PERCENT NAME LOW ISO FROM CLOSE TOTALEXCHANGEVOLUME'
    avg=:(+/%#)
    avg HIGH
    +/CIRCULATINGSUPPLY
    NAME{~(i.>./)PERCENT
    10{.(\:PERCENT){NAME
    10{.(\:PERCENT){PERCENT
)


NB. c=:''conew'cryptocurrency'[load'/home/lambertdw/src/j/cryptocurrency/cryptocurrency.ijs'

require'files'
require'~addons/convert/json/json.ijs'
load'/home/lambertdw/src/j/dwldb.ijs'

DBDIR=: '/home/lambertdw/src/j/cryptocurrency/'


Dyad=: [: :
Monad=: :[:

ls=: 1!:0@boxopen Monad
read=:1!:1@boxopen Monad
write=:(1!:2 boxopen)~ Dyad
append=:(1!:3 boxopen)~ Dyad
mkdir=:1!:5@boxopen Monad
open=:1!:21@boxopen Monad
close=:1!:22@boxopen Monad
pwd=:1!:43@boxopen Monad
cd=:1!:44@boxopen Monad
rm=:1!:55@boxopen Monad
host=:2!:0 Monad
spawn=:2!:1 Monad
getenv=:2!:5 Monad
getpid=:2!:6 Monad
exit=:2!:55 Monad

nameClass=:4!:0@boxopen
erase=: [: 4!:55 ;: ::]

create=: 3 :0
 NB. y is read only flag
 if. 0 ~: y do.
  db=: (DBDIR ; 1) conew'dwldb'
 else.
  if. fexist DBDIR,'ISO.col' do.
   db=: DBDIR conew'dwldb'
  else.
   ([: createjmf_jmf_ 1024 ;~ '.col' ,~ DBDIR&,)&> ;: 'ISO NAME OPEN HIGH LOW CLOSE PERCENT VALUE MARKETCAP CIRCULATINGSUPPLY TIMEPERIOD TO FROM TOTALEXCHANGEVOLUME RETURNS24 TS'
   db=: DBDIR conew'dwldb'
   ISO                =: 0  5$''
   NAME               =: 0 21$''
   OPEN               =: 0  1$0.0
   HIGH               =: 0  1$0.0
   LOW                =: 0  1$0.0
   CLOSE              =: 0  1$0.0
   PERCENT            =: 0  1$0.0
   VALUE              =: 0  1$0.0
   MARKETCAP          =: 0  1$0.0
   CIRCULATINGSUPPLY  =: 0  1$0.0
   TIMEPERIOD         =: 0  3$''
   TO                 =: 0  1$00
   FROM               =: 0  1$00
   TOTALEXCHANGEVOLUME=: 0  1$00
   RETURNS24          =: 0  1$0.0
   TS                 =: 0  1$00
  end.
  acquire db
  closedb__db''
  erase 'ISO NAME OPEN HIGH LOW CLOSE PERCENT VALUE MARKETCAP CIRCULATINGSUPPLY TIMEPERIOD TO FROM TOTALEXCHANGEVOLUME RETURNS24 TS'
  opendb__db 1  NB. readonly
 end.
)

NB. location of the temp file for wget
DIR=:      '/tmp'
FILENAME=: 'bitcoin'

Note '00README'

 wget https://www.coindesk.com/price  information source
 https://www.CryptoDataDownload.com   historical data in .csv, many coins

 plan
  use a home-brew non-sql database
   create a directory for the database
   auto-extending
  data fields and types
   dates stored as int, see timestamp verb
   data comes from json section of the web page
)

rm=: 1!:55@:boxopen
pwd=: 1!:43
cd=: 1!:44
host_command =: 2!:0
recent_error=:13!:12

ufail=: 'verb u failed'"_

MapFile=: adverb define
 NB. u is a verb to act on the file named as y
 NB. MapFile wraps the mapping
 JCHAR map_jmf_'A' ; y
 result=. u (fullname_jmf_'A')~
 unmap_jmf_'A'
 result
)

UnderDir=: conjunction define
 savedir=. pwd''
 cd n
 result=. u ::ufail y
 cd savedir
 result
)

UnderTmpFile=: adverb define
 rm ::empty y
 result=. u ::ufail y
 rm ::empty y
 result
)

get_data_from_internet=: 3 :0
 host_command'wget https://www.coindesk.com/price/',y
 ([: , (13&}.;.1~ '"currencies":[{'&E.) MapFile) y
)

yyyymmddhhmmss=: 0 ". 1 1 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 &#
yyyyxmmxddxhhxmmxss=: _ 100 100 100 100 100&#.inv
timestamp=: yyyymmddhhmmss :.yyyyxmmxddxhhxmmxss
assert 2021 4 7 23 4 47 -: ]&.timestamp'2021-04-07T23:04:47.851+00:00'

acquire=: 3 :0
 a=: get_data_from_internet UnderTmpFile UnderDir DIR FILENAME
 b=: dec_json_json_ ,@((];._2~ '},"dataLayer":'&E.)) a

 NB. make sure all fields are available
 $@".L:0 ;:'HIGH OPEN VALUE MARKETCAP CIRCULATINGSUPPLY RETURNS24 TIMEPERIOD TS TO PERCENT NAME LOW ISO FROM CLOSE TOTALEXCHANGEVOLUME'

 NB.literal   5  iso                         (<1;0)&{::S:_1
 'ISO'appendcol__db~(<1;0)&{::S:_1 b

 NB.literal  21  name                        (<1;1)&{::S:_1
 'NAME'appendcol__db~(<1;1)&{::S:_1 b

 NB.floating open                        ([: |: [: > 1 {"2 (<1;6)&{::S:_1)
 NB.floating high
 NB.floating low
 NB.floating close
 (([: |: [: > 1 {"2 (<1;6)&{::S:_1) b)(appendcol__db >)"1 0;:'OPEN HIGH LOW CLOSE'

 NB.floating percent                     ([: |: [: > 1 {"2 (<1;7)&{::S:_1)
 NB.floating value
 (([: |: [: > 1 {"2 (<1;7)&{::S:_1) b)(appendcol__db >)"1 0;:'PERCENT VALUE'

 NB.floating marketCap                   (<1;8)&{::S:_1
 'MARKETCAP'appendcol__db~(<1;8)&{::S:_1 b

 NB.floating circulatingSupply
 'CIRCULATINGSUPPLY'appendcol__db~(<1;9)&{::S:_1 b

 NB.literal   3  timePeriod              ([: |: [: ,/ _ _1 _ {.  (<1;_6)&{::S:_1)
 NB.floating  29  from  timestamp
 NB.floating  29  to    timestamp
 t=. ([: |: [: ,/ _ _1 _ {.  (<1;_6)&{::S:_1) b
 'TIMEPERIOD'appendcol__db~ > 0 { t
 'FROM'appendcol__db~ timestamp S:0 ] 1 { t
 'TO'appendcol__db~ timestamp S:0 ] 2 { t

 NB.integer  totalExchangeVolume         ([: |: [: (;"2)_ _1 2 {. (<1;_5)&{::S:_1)
 NB.floating returns24
 t=. ([: |: [: (;"2)_ _1 2 {. (<1;_5)&{::S:_1) b
 'TOTALEXCHANGEVOLUME'appendcol__db~ <. {. t
 'RETURNS24'appendcol__db~ {: t

 NB.integer  ts                          (<1;_2)&{::S:_1
 'TS'appendcol__db~ (<1;_2)&{::S:_1 b
)