Jd/Release

The Jd wiki is dynamic. It has the latest corrections and changes. It reflects not just the latest release, but development work that hasn't been finalized or released. It reflects the bleeding edge and may not be directly relevant for the version you are running.

9 Feburary 2024 - version 4.47

• update pandas - pandas.ijs and tutorials

4 April 2023 - version 4.46

• pandas support for parquet/feathers/orc/hdf5/... extended
• write Jd tables to parquet/... files
• pandas tutorials improved

  jdrt'pandas_install' NB. install and verify python3/pandas
  jdrt'pandas_load'    NB. load parquet file of yellow cab trip data
  jdrt'pandas_clean'   NB. basic table cleaning - missing values and outliers
  jdrt'pandas_write'   NB. write Jd tables to parquet/... files

22 March 2023 - version 4.45

• parquet tutorial expanded to cover cleaning the table
  • missing values and low/high outlier values replaced by the median value

  jdrt'parquet'

20 March 2023 - version 4.44

• load parguet/feathers/orc/hdf5/... files into Jd table

  jdrt'parquet'

• csvcdefs sets escape as \ - previously it was NO
• csvcdefs does a better job of setting row delimiter
  • previously one CRLF would override all the LFs
• csvcdefs does a better job of setting col delimiter
  • previously it did not ignore chars in "s
• dropdb - bugs fixed, removes all admin for the folder, enables subsequent createdb
• createdb - bugs fixed, must follow dropdb in same session

2 March 2023 - version 4.43

• tables used in this version will be marked damaged if used with a previous version!!!
  • Run Production - protect your production Jd code base and databases before running this version
  • you can revert 4.43 tables to work with previous versions

  load 'path-to-Jd-4.43-or-later-folder','/tools/fix_tlen.ijs'
  fix_tlen '' NB. display help
  NB. follow instructions

• performance improvement for insert/upsert
  • ~20% faster than 4.42 to insert 1 row of 20 cols
  • Tlen (table row count) maintained in jdtlen mapped file
  • writestate file op not required on every insert/upsert
  • Tlen performance improvement makes reverting difficult

27 February 2023 - version 4.42

• accept new 9.4 (new JVERSION format)
• macos universal binary build (M1) in Jd shared library

16 February 2023 - version 4.41

• csv ops that read csv files no longer assert .csv suffix
• all op overhead reduced by 3 file operations (fread and 2 fexist)
  • required for validation and damage/repair state and not necessary in streamlined path
  • measurable performance improvement for all ops - ~10% faster than 4.40 or more

5 March 2021 - version 4.40

• improvements to tutorials jds, jds_client, and node

10 February 2021 - version 4.39

• see Server for significant new facilities (json/nodejs/https)

  jdrt'jds'
  jdrt'jds_client'
  jdrt'node'

4 November 2020 - version 4.38

• read code now in standard location - previously read ran in a temporary locale
  • this makes it easier to use jpm (performance monitor) for read

27 October 2020 - version 4.37

• special code for read with single by column
  • this common case runs faster

24 October 2020 - version 4.36

• special code for read from single table with empty where clause
  • full support for aggregations, by, and order by
  • 3 times faster than previous use of the general case code

17 October 2020 - version 4.35

• new tutorial that downloads NYC taxi tripdata csv file

jdrt'taxi'

• fixed csv load problem where headers had spurious chars (extra LF)
• minor code adjustments ahead of next JE 902 beta release

13 October 2020 - version 4.34

• fix JE overfetch crash - longstanding bug
  • triggered by some queries where data was too close to file size page boundary
  • mapped cols now ensure pad before file size page boundary

• insert performance profiling allowed refactoring so that inserts run 2 times faster

8 September 2020 - version 4.33

Lots of changes in sensitive areas. Deploy in production systems with caution.

• csv_load tutorial
  • easy to use utils for getting your csv file into J with minimal Jd knowledge

jdrt'csv_load'

• renamecol updates column_create_order for default read selection
• col names can have any printable char
  • easier to work with other systems (especially with csv)
  • Jd can now accept/create col names compatible with everyone
  • see names for details on rules for names

'new'jdadmin'test'
jd'createtable f'
jd'createcol f "a/ =\" \b" int'
jd'insert f';'a/ =" \b';i.5
jd'read from f where "a/ =\" \b"=3'
jdaddq_jd_'a" /'    NB. " if required by " or blank
jdremq_jd_'"a\" /"' NB. remove "s

23 May 2020 - version 4.32

• jd'createcol /derived ...' - see tutorial createcol_derived
• jd'createcol /derived_mapped ...' - see tutorial createcol_derived_mapped
• jd'createdcol ...' - deprecated - replaced by /derived
• csvcdefs - null fields ignored in determining col type

8 May 2020 - version 4.31

• fix ptable bug in insert and key

28 April 2020 - version 4.30

• experimental jconsole task manager - jdrt'jctask'

16 April 2020 - version 4.29

• experimental blob support - jdrt'blob'
• experimental json/jbin string interface - jdrt'jdserver'
• experimental http server that manages multiple Jd tasks on same db - jdrt'mtm'
• jdrt'json'
• jd'list open' - removed

26 February 2020 - version 4.28

• Jd shared library - thread safe except for csv load routines
  • load routines could be threadsafe with a bit more work
• csvcdefs - changes to improve guesses on col types
  • will not set types date/datex/datetime/datetimex due to efs confusion
  • csvcdefs is called from csvrd to create cdefs file if there isn't one
• csvrd - changes for bad or missing data for numeric types
• csvreport /errors option for table of csvrd errors
• csvreport /f option had opposite effect - fixed
• gen types table rows
• Jd windows shared library now uses same C runtime as J (/MT)
• Jd shared library - localtime changed to localtime_r to be threadsafe

7 January 2020 - version 4.27

• pm/bench improvements
• countdat extended over types for better colcheck and validate
• mtm support for concurrent MTRW and multiple MRRO tasks
• jdadminnew - deprecated - will be removed in a future release
• jdadminx - removed
• 'access'jdaminx‘‘ - removed
• jdadmin takes mt (map type) parameter
• jmf addon updated for map types MTMRW, MTRO (read only), MTCW (copy on write)

17 August 2019 - version 4.26

• query_test.ijs code change for 901
• 901 JE has sfe/efs support as foreigns 6!:16/17
  • 6!:16 sfe is 6 times faster
  • 6!:17 efs is 10 times faster
  • old dtmn and new d039 precision indicators are supported
  • csv loader continues to use previous version
  • bad date (imin_jd_) allowed in insert of sfe string
    • previously bad date only allowed with insert int
    • bad date allowed in csv rd/wr

2019 May 19 - version 4.25

• jd jcs server improved - jdrt'jcs'
• likeci/unlikeci - case insensitive
• key changed to use first match (similar to left1 change)

2019 May 4 - version 4.24

• reads /table foo - create new table from reads result
• tutorials reworked and improved - better base for future work
• listing of a run of all tutorials is at tutorials

2019 April 19 - version 4.23

• incompatible changes
  • left1 join matches first row (previously matched last row)
  • read /types includes shape in byte type - (byte 4) - and proper agg type
• experimental,incomplete features
  • read /table foo - creates table foo from read result
  • read /file abc - creates (3!:1) file FILEFOLDER/abc from read result
• significant rework on parsing options
• like/unlike shared lib calls adjusted to work for 901
• bug fixed - info validatebad does not report derived cols as bad because of filesize
• info avoids unnecessary mapping - derived col performance problems

2019 March 19 - version 4.22

• jdrt 'jcs' - document jcs/zmq based Jd server
• fix local html doc problems

2019 February 23 - version 4.21

• createdcol op - create derived col - jdrt'createdcol'
• info derived - report derived cols
• byten op - changes trailing shape of byte N col
• bug - read epoch cols incorrectly displayed as int instead of string
• bug - read with more than 1 blank between agg and col
• readtc op removed

2019 January 18 - version 4.20

• reduced start overhead for all ops - simple read/insert run 20% faster
• fixed reported bug that prevented deleting a Jd folder

2019 January 3 - version 4.19

• replicate - jdrt'replicate'
• jdadminnew'...' NB. create new db (jdadminx'...' and 'new'jdadmin'...' deprecated)
• see Run Production - production Jd code base vs casual use of latest addon update code base
• setscriptlists run directly for tests and tuts
• forcecopy - memu instead of a:{]
• requires J807
• dbl_jd_ current DB locale
• misc fixes

2018 August 30 - version 4.18

• allow table to join to itself

2018 July 3 - version 4.17

• csvscan bug with cd call (new rules) fixed
• csvwr crash with byte col 0 avoided
• update with 0 rows to update allows scalars as well as no data

2018 May 24 - version 4.16

sort syntax changed to be same as read order by clause

sort bug with multiple cols fixed

2018 May 15 - version 4.15

automatic free non-commercial key

jdtests_jd_‘‘ NB. run tests

2018 April 18 - version 4.14

fix MAXROWCOUNT bug

tutorials improved - see basic intro_a b and c

Lam setdirty/jdmap bug fix

2018 April 13 - version 4.13

improve ref arg validation

fix ref /left bug - read with more than 1 join

fix ref /left bug - read with where clause

2018 April 3 - version 4.12

basic-pairs tutorial

op examples in docs moved to tutorials

demos moved to tutorials

jdex decommitted

jdrt changes

'new'jdadmin'...'

createtable /replace

2018 March 29 - version 4.11

table_from_pairs - new tutorial (replaces table_from_data tutorial)

read[s] /types

createtable /pairs /types

performance monitor off - previously always on - see pmhelp_jd_ for details

2018 March 14 - version 4.10

pcre (not new pcre2) always in cd folder

small adjustments for 807

minor cleanup

2018 January 13 - version 4.9

break large source file into smaller op oriented files

fix bugs in int1/2/4 support

add intx op to convert between int types

2018 January 11 - version 4.8

update where _ selects all rows

jdadmin 0 - fix bug - error if no db opened (showed up in convert jd3-jd4)

csv support for int1/int2/int4 types

int1/int2/int4 types

ref /left can only be used in single joins - f,f-g,g-h signals an error

fixpairs routine (adjust pairs for insert etc) improved and made consistent across all ops

2017 December 11 - version 4.7

linux cd folder contains libjpcre.so as it is not included in J deb install

cd folder contains only jd platform binary

where range clause runs faster

FLUSHAUTO has been decommitted. Previously FLUSHAUTO_jd_ (default value 1) flushed buffers when a close, csvrd, createcol, ref, or set was done. This slowed things down (e.g., jdtests took twice as long) and wasn't required except for critical, production systems.

Use jdcloseflush to mark a db to require a flush when a close is done.

Use flush to flush buffers explicitly when it make sense for the application.

2017 November 29 - version 4.6

documentation reorg

csv colsep/rowsep AUTO support added back in

damaged db

repair tool can repair a damaged db

delete and update ops can use keys

sort op

upsert op

key op

keys - one or more cols form a key

col-data-pairs reworked

2017 November 4 - version 4.5

new tutorial - quandl_eod_stock_data

jdtests‘‘ NB. run tests

rationalize, reorganize, and clean up folders

2017 November 1 - version 4.4

old keys no longer work - new purchased keys are required

quandl_ibm - new tutorial - load table with historical ibm pricing from quandl csv

jdrt/jdex cleaned up

validate streamlined to run faster

docs updated to reflect join changes

jd'reads from a,a.b where b.id=2' - bug fix - incorrectly included missing rows

jd'ref /left ...' - left join - left1 and inner derived from left

left1 - preferred join if appropriate - fast

jdadmin'none' - proper error report

2017 October 26 - version 4.3

join tutorial has example of multiple refs between tables (bug fixed)

some problems with jdex examples fixed

Jd paypal page added to web site and docs updated

new style keys supported

old style keys still work

OSX problems with unicode table and column names

• previously failed because of OSX file system decomposed unicode
• jdtests now flag this OSX error
• will be fixed in a future release

806 new labs

• tutorials use new 806 labs if available
• sp.ijs loaded only if spx required to run labs
• sptable defined (in case sp.ijs not loaded)

2017 September 28 - version 4.2

Snapshot of Jd wiki with new proposed licensing and prices.

Addition of tutorial server-zmq - high-performance, scalable zmq based Jd server.

2017 May 17 - version 4.1

Jd 4.1 is a major release and is incompatible with previous versions. A db from a previous version has to be converted to be used by this version.

See Convert for info on converting a Jd3 database to Jd4.

Overall the changes were to simplify and streamline code for the use cases that have proved common and to remove unwarranted complications. The big changes were removing jdactive and support for other than a simple join.

pacman 2.3.7

• rewrite createcol

• rewrite insert/update data validation to have similiar conformabilty rules

• fix createtable parse problem with , separated coldefs

• toJ for builddemo jdcsvrefs.txt

pacman 2.3.6

• new Jd4 jdkey required (Jd4 key in ~confg and Jd3 key in ~addons/data/jd/config)

• new shared library - version 8

• jdactive col decommitted
  • delete immediately removes rows from the table
  • not dealing with rows marked as deleted is a major code simplification
  • everything (except delete) runs faster and cleaner

• left1 is the only join - all others are decommitted

• ref now supports multiple cols

• reference col decommitted
  • was required for non left1 joins
  • creating reference was slow on large tables
  • code to dynamically track changes was complicated

• hash col decommitted (primarily for reference)

• update in place (rather than delete/insert)
  • much simpler code and much faster
  • update varbyte done in place (if space) or new data appended to val end

• modify decommitted (update does it all)

• insert
  • errors normally caught before insert starts - table unchanged on error
  • an error after the insert starts now reverts - table unchanged on error

• when col is opened, its length is validated against Tlen - db damaged if wrong

• tableappend decommitted (tableinsert does it cleaner and faster)

• info changed to reflect other changes (deleted/reference/dynamic)

• csvdump /replace
• csvrestore /replace
• csvdump creates jdcsvrefs.txt and csvrestore uses it to create refs
• dropdynamic and createdynamic decommitted

2017 April 27 - version 3.3

• Jd wiki - latest Jd documentation at: http://code.jsoftware.com/wiki/Jd/Index
• Jd wiki snapshot (release specific) at: ~addons/data/ide/doc/Index.htm
• update bug - bad shape deleted record
• modify bug - bad shape truncated byte data
• info validate/validate - col c marks bad tlen - col s marks bad fsize
• new jd shared library version 7
• db log.txt limited in size to 16mb
• table... changes - srcdan removed as arg from all except tablemove
• csv column names - bugs in utf8 and special chars fixed
• csv cdefs cols names can now be any valid Jd col name
• csv BOM UTF8 (239 187 191{.a.) skipped
• csv cdefs # starts a comment line
• csv csvcdefs creates cdefs with default " and NO for escape
• csv cdefs AUTO no longer supported (handled in csvcdefs)
• csvprobe op to examine csv file
• csvscan - scans csv file and updates cdefs to have proper byte widths
• csvcdefs /u no longer takes value - value is derived from data
• csv tutorial reworked
• csv_load tutorial added

2016 May 12 - version 3.2

• partitioned table - significant new facility - see tutorial ptable
• invalid epoch data passes precision assertions
• default epoch and sfe display does not have Z suffix
• table/column/dan name validation (General|Names)
• read op parsing reworked and some bugs fixed
• documentation of ops reorganized and improved
• readtc requires jdtc clause (:::s::: no longer supported)
• validate no longer done before/after insert/update/modify/delete
• Tlen (active+deleted rows) maintained in table state
• S_deleted (deleted rows) maintained in table state
• info summary uses Tlen/S_deleted - fast as no read of jdactive
• info improved and enhanced to handle ptable
• csv loader bug with fields longer than 1000 fixed
• where clause not bug fixed
• datatune support removed
• regex cd problem with linux debian installs fixed

2016 January 12 - version 3.1

• showbox bug fixed for jconsole (non linedraw) box chars
• showbox replaced by sptable (from simple project)
• insert/update datatune bugs fixed
• modify now does datatune
• tutorial datatune updated
• datatune doc
• key enforced by shared library
• Jd binary and code version match enforced
• csv now supports epoch cols (Ops csv)
• csvrd default reads all rows (rather than just 10)
• tutorial csv updated
• install over previous install (orphans) now OK (scriptlists.ijs)
• JAL version no longer tied to jd'list version'

2015 December 12 - version 2.2.11

• dropcol does not update column_create_order.txt - bug fix
• csvappend fails in some cases - bug fix
• min-max/epochdt with no records works (not domain error)
• logging and db validation improved (Technical|Damaged)
• tutorial log
• validate - detects db problems (Ops misc|validate)
• info validate - info on all db mappings (Ops info)
• info validatebad - validate mappings with problems (Ops misc|validate)
• jdadmin does validate
• validate done before/after insert/update/modify/delete
• insert revert bug - detailed log and marks db damaged

2015 October 1 - version 2.2.10

• linux lock bug fixed - concurrent Jd task not locked out
• info dynamic - (Ops info)
• createhash/createunique 2 to 10 times faster on large tables
• createhash /nc
• createunique simplified and changed
• insert unique simplified and changed
• flush - flush mapped file dirty buffers (Ops misc|flush)
• ref changed and improved
• performance tutorial - table size vs ram

2015 June 18 - version 2.2.9

• where cola,colb = 1,2 doc and implementation bug fixed
• jddropstop prevents drops (General|Drop/Delete)
• jddeletefolder fails if criteria not met (General|Drop/Delete)
• jddeletefolder more robust vs background tasks (Technical|Windows search service)
• jddeletefolderok allows otherwise blocked jddeletefolder
• welcome replaced by short message after jdtests run
• modify with indexes as well as where clause - jdex_jd_'modify'
• bugs fixed in where col in ()
• rough version of pm (performance measurement) - pmhelp_jd_
• droptable /reset
• info jd - info on jd... cols (Ops info)
• better createhash/createunique performance
• fix bug in createhash/createunique for multiple cols
• libjd.so built with stable linux (RH 4/Centos 5) (Technical|libjd.so)

2015 May 11 - version 2.2.7

• where cola range (23,64) - tutorial reads
• where cola,colb = 1,2 - tutorial where
• jdadminx'a' (single letter) bug fixed
• new doc page (General)
• consistent rules for dan/table/col names (General|Names)
• modify bugs fixed
• createtable /a option must be before table name
• create table from array - tutorial table_from_array
• better error messages

2015 April 14 - version 2.2.6

• CDEFSFILE (tutorial csv)
• gethashlen bug fixed
• where in () bug fixed
• jdtests reports elapsed time
• performance measurement - pmhelp_jd_
• read.ijs sortfrom bug fixed
• createcol shape validated
• droptable allowed with hash (fails with reference)
• some aggregates sped up
• better error message for insert/update/modify
• jdlast last line shuffled
• renametable implemented
• renamecol implemented
• modify implemented - update in place
• libjd_centos.so (Technical|libjd.so)

2015 March 10 - version 2.2.1

• dropdynamic for specified columns
• info unique

2015 March 9 - version 2.2

• ref join - simpler than reference but with restrictions (tutorial join,Technical|Ref)
• improved handle management and related errors (Technical|File handles)
• epochdt new features - tutorial epochdt
• epochdt allows blank (as well at T) as delimiter of time fields
• epochdt bug with UTC offsets fixed
• join with empty table bug fixed
• join with createunique crash now signals error
• csvrd bug with float .x vs 0.x fixed

2014 Jan 5 - version 2.1

• key required to run Jd
• epoch datetime col types (tutorial epochdt, General|Types)
• smarter file allocation and resize
• all column files mapped when db is opened
• list lists version and open databases (Ops info|list)
• info works only on current database

2014 April 22 - version 2.0

• Jd released as addon