David Lambert/dwldb and cryptocurrency
Jump to navigation
Jump to search
Warning: the file paths are oriented toward my personal configuration, including linux.
These two files are implemented as classes. The database. appendcol automatically resizes as needed. There is not an "append rows" functionality implemented. When open rw, the database first makes a tar archive of the columns. Otherwise, error recovery is not implemented.
host 'cd {} && tar czf coldb.tar.z *.col' format~ <DIR
Note'00README' To the Hands Caroline Shaw (3333$a)appendcol__D'a'[D=:'/tmp'conew'dwldb'[load'~/j.ijs' setup: make a directory, install files fieldName.col using createjmf_jmf_ load this file into a session all files in the directory with extension name.col are mapped as columns. VALID is a reserved column name require'~lambertdw/src/j/dwldb.ijs' db=: DIRECTORY conew dwldb Setup for 1st use. put *.col files into a directory. Each of these will be a database field. for example, date.col will map to the variable date. These files are created with require'jmf' createjmf_jmf_'date.col';666 subsequent uses create takes arguments dirpath ; readonly flag The default flag is 0, ie. rw. It maps all the columns, storing the variables into the base locale. ) coclass'dwldb' require'jmf' Filter=:(#~`)(`:6) Dyad=: [: : Monad=: :[: ls=: 1!:0@boxopen Monad read=:1!:1@boxopen Monad write=:(1!:2 boxopen)~ Dyad append=:(1!:3 boxopen)~ Dyad mkdir=:1!:5@boxopen Monad open=:1!:21@boxopen Monad close=:1!:22@boxopen Monad pwd=:1!:43 Monad cd=:1!:44@boxopen Monad rm=:1!:55@boxopen Monad host=:2!:0 Monad spawn=:2!:1 Monad getenv=:2!:5 Monad getpid=:2!:6 Monad exit=:2!:55 Monad nameClass=:4!:0@boxopen erase=: [: 4!:55 ;: ::] format=: ;@:((a: , ":&.>)@:[ ,. '{}'&([ }.L:_1@:(E. <;._1 ]) ,)@:]) fail=: format assert 0: rank=:#@$ showmap=: ,:^:(2>rank)@:showmap_jmf_ create_dwldb_=: 3 : 0 NB. default is open for rw 'dir ro'=. 2 {. (boxopen y) , < 0 'readonly flag must be 0 or 1'assert ro e. 0 1 DIR=: PATHJSEP_j_ ,~^:(~:{:) jpath dir a=. ls }: DIR 'expected 1 match' assert 1 = # a 'not a directory' assert 'd' = {. (0 _1) {:: a opendb ro ) valid_columns=: 3 : 0 NB. y is the desirable nameclass, 0 for existing nouns, _1 for unused names a=. ([: -. (1 e. '_'&e.)&>)Filter {."1 ls DIR , '*.col' (_2 < ([: nameClass ('_' , '_' ,~ > COCREATOR) ,~ _4&}.)&>)Filter a ) closedb=: ([: unmap_jmf_ _4&}.)&>@:valid_columns destroy=: 3 :0 closedb'' coerase y ) opendb=: 3 :0 :.closedb NB. y is readonly flag if. y = 0 do. NB. make backup host 'cd {} && tar czf coldb.tar.z *.col' format~ <DIR end. files=. valid_columns _1 ([: map_jmf_ ('' ; y) ,~ _4&}. ; DIR&,)&> ([([:echo 'opening '&;)) files ) appendcol=: 4 :0 ::resize name=. fullname_jmf_ y if. # name~ do. (name)=: name~ , x else. (name)=: x end. EMPTY ) resize=: 4 :0 name=. fullname_jmf_ y map=. showmap'' ({. map) =. map&(((i.~ {."1)~ <) { [) ::(fail&'{} not mapped') name tempname=. DIR,'growth_here' createjmf_jmf_ tempname ; {. fsize + msize map_jmf_'growth_here';tempname growth_here_name=: fullname_jmf_ 'growth_here' (growth_here_name)=: name~ unmap_jmf_&> y ; growth_here_name host 'mv {} {}' format~ (; ('bak' ,~ _3&}.)) fn host 'mv {} {}' format~ tempname ; fn map_jmf_ y ; fn x appendcol y )
Example using dwldb, uses wget to load the json data from the www.coindesk.com/price/, making some gross assumptions that they won't change their web page format.
coclass'cryptocurrency' Note'use' $ ~/downloads/installs/j903/jconsole.sh if the ro flag is 0, update the database, then open read only. if the ro flag is 1, just open the db read only. roflag conew c=:0 conew'cryptocurrency'[load'/home/lambertdw/src/j/cryptocurrency/cryptocurrency.ijs'[13!:0[1 $@".L:0 ;:'HIGH OPEN VALUE MARKETCAP CIRCULATINGSUPPLY RETURNS24 TIMEPERIOD TS TO PERCENT NAME LOW ISO FROM CLOSE TOTALEXCHANGEVOLUME' avg=:(+/%#) avg HIGH +/CIRCULATINGSUPPLY NAME{~(i.>./)PERCENT 10{.(\:PERCENT){NAME 10{.(\:PERCENT){PERCENT ) NB. c=:''conew'cryptocurrency'[load'/home/lambertdw/src/j/cryptocurrency/cryptocurrency.ijs' require'files' require'~addons/convert/json/json.ijs' load'/home/lambertdw/src/j/dwldb.ijs' DBDIR=: '/home/lambertdw/src/j/cryptocurrency/' Dyad=: [: : Monad=: :[: ls=: 1!:0@boxopen Monad read=:1!:1@boxopen Monad write=:(1!:2 boxopen)~ Dyad append=:(1!:3 boxopen)~ Dyad mkdir=:1!:5@boxopen Monad open=:1!:21@boxopen Monad close=:1!:22@boxopen Monad pwd=:1!:43@boxopen Monad cd=:1!:44@boxopen Monad rm=:1!:55@boxopen Monad host=:2!:0 Monad spawn=:2!:1 Monad getenv=:2!:5 Monad getpid=:2!:6 Monad exit=:2!:55 Monad nameClass=:4!:0@boxopen erase=: [: 4!:55 ;: ::] create=: 3 :0 NB. y is read only flag if. 0 ~: y do. db=: (DBDIR ; 1) conew'dwldb' else. if. fexist DBDIR,'ISO.col' do. db=: DBDIR conew'dwldb' else. ([: createjmf_jmf_ 1024 ;~ '.col' ,~ DBDIR&,)&> ;: 'ISO NAME OPEN HIGH LOW CLOSE PERCENT VALUE MARKETCAP CIRCULATINGSUPPLY TIMEPERIOD TO FROM TOTALEXCHANGEVOLUME RETURNS24 TS' db=: DBDIR conew'dwldb' ISO =: 0 5$'' NAME =: 0 21$'' OPEN =: 0 1$0.0 HIGH =: 0 1$0.0 LOW =: 0 1$0.0 CLOSE =: 0 1$0.0 PERCENT =: 0 1$0.0 VALUE =: 0 1$0.0 MARKETCAP =: 0 1$0.0 CIRCULATINGSUPPLY =: 0 1$0.0 TIMEPERIOD =: 0 3$'' TO =: 0 1$00 FROM =: 0 1$00 TOTALEXCHANGEVOLUME=: 0 1$00 RETURNS24 =: 0 1$0.0 TS =: 0 1$00 end. acquire db closedb__db'' erase 'ISO NAME OPEN HIGH LOW CLOSE PERCENT VALUE MARKETCAP CIRCULATINGSUPPLY TIMEPERIOD TO FROM TOTALEXCHANGEVOLUME RETURNS24 TS' opendb__db 1 NB. readonly end. ) NB. location of the temp file for wget DIR=: '/tmp' FILENAME=: 'bitcoin' Note '00README' wget https://www.coindesk.com/price information source https://www.CryptoDataDownload.com historical data in .csv, many coins plan use a home-brew non-sql database create a directory for the database auto-extending data fields and types dates stored as int, see timestamp verb data comes from json section of the web page ) rm=: 1!:55@:boxopen pwd=: 1!:43 cd=: 1!:44 host_command =: 2!:0 recent_error=:13!:12 ufail=: 'verb u failed'"_ MapFile=: adverb define NB. u is a verb to act on the file named as y NB. MapFile wraps the mapping JCHAR map_jmf_'A' ; y result=. u (fullname_jmf_'A')~ unmap_jmf_'A' result ) UnderDir=: conjunction define savedir=. pwd'' cd n result=. u ::ufail y cd savedir result ) UnderTmpFile=: adverb define rm ::empty y result=. u ::ufail y rm ::empty y result ) get_data_from_internet=: 3 :0 host_command'wget https://www.coindesk.com/price/',y ([: , (13&}.;.1~ '"currencies":[{'&E.) MapFile) y ) yyyymmddhhmmss=: 0 ". 1 1 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 &# yyyyxmmxddxhhxmmxss=: _ 100 100 100 100 100&#.inv timestamp=: yyyymmddhhmmss :.yyyyxmmxddxhhxmmxss assert 2021 4 7 23 4 47 -: ]&.timestamp'2021-04-07T23:04:47.851+00:00' acquire=: 3 :0 a=: get_data_from_internet UnderTmpFile UnderDir DIR FILENAME b=: dec_json_json_ ,@((];._2~ '},"dataLayer":'&E.)) a NB. make sure all fields are available $@".L:0 ;:'HIGH OPEN VALUE MARKETCAP CIRCULATINGSUPPLY RETURNS24 TIMEPERIOD TS TO PERCENT NAME LOW ISO FROM CLOSE TOTALEXCHANGEVOLUME' NB.literal 5 iso (<1;0)&{::S:_1 'ISO'appendcol__db~(<1;0)&{::S:_1 b NB.literal 21 name (<1;1)&{::S:_1 'NAME'appendcol__db~(<1;1)&{::S:_1 b NB.floating open ([: |: [: > 1 {"2 (<1;6)&{::S:_1) NB.floating high NB.floating low NB.floating close (([: |: [: > 1 {"2 (<1;6)&{::S:_1) b)(appendcol__db >)"1 0;:'OPEN HIGH LOW CLOSE' NB.floating percent ([: |: [: > 1 {"2 (<1;7)&{::S:_1) NB.floating value (([: |: [: > 1 {"2 (<1;7)&{::S:_1) b)(appendcol__db >)"1 0;:'PERCENT VALUE' NB.floating marketCap (<1;8)&{::S:_1 'MARKETCAP'appendcol__db~(<1;8)&{::S:_1 b NB.floating circulatingSupply 'CIRCULATINGSUPPLY'appendcol__db~(<1;9)&{::S:_1 b NB.literal 3 timePeriod ([: |: [: ,/ _ _1 _ {. (<1;_6)&{::S:_1) NB.floating 29 from timestamp NB.floating 29 to timestamp t=. ([: |: [: ,/ _ _1 _ {. (<1;_6)&{::S:_1) b 'TIMEPERIOD'appendcol__db~ > 0 { t 'FROM'appendcol__db~ timestamp S:0 ] 1 { t 'TO'appendcol__db~ timestamp S:0 ] 2 { t NB.integer totalExchangeVolume ([: |: [: (;"2)_ _1 2 {. (<1;_5)&{::S:_1) NB.floating returns24 t=. ([: |: [: (;"2)_ _1 2 {. (<1;_5)&{::S:_1) b 'TOTALEXCHANGEVOLUME'appendcol__db~ <. {. t 'RETURNS24'appendcol__db~ {: t NB.integer ts (<1;_2)&{::S:_1 'TS'appendcol__db~ (<1;_2)&{::S:_1 b )