Guides/Named Noun Internals

The following is an overview of the internals of J named nouns. The examples assume J32, where ints and pointers are both 4 bytes.

Memory Foreigns

The following definitions are used:

memr=: 15!:1        NB. read memory: address,byte_offset,count [,type]
memw=: 15!:2        NB. write memory
mema=: 15!:3        NB. allocate memory

Named Nouns

A J named noun is implemented in three parts: a symbol table entry, a header, and the data. The symbol table entry points to the header which points to the data.

Suppose a new noun abc is defined as:

abc=: 10 + i.2 3

Then, a symbol table entry is created, which starts with two pointers:

1. pointer to name abc
2. pointer to header

This structure may change in the future and symget is considered deprecated. Starting with J 9.04, the pointer to the header is accessed directly with 15!:12.

The header is list of 7 integers, followed by the shape which is itself a list of integers. In this case:

72 0 1 4 1 6 106037250

1. offset from header to data
2. internal flags
3. formerly: reserved size for data - at least large enough to store data
4. data type
5. reference count
6. data length (*/shape)
7. a union, the significance of which depends on context; the six lowest bits indicate the rank
8. shape (list of integers)

The data is stored in ravelled order. Typically, it is stored immediately after the header, but this is not required. For example, it would not be the case when mapping files that only contain data. The data is best accesses using 15!:14.

The following verb returns the symbol table pointers, fixed header, shape and data:

viewnoun=: 3 : 0
s=. symget <y               NB. pointer to symbol table entry
sys=. memr s,0 2 4          NB. pointers to name, header
h=. 1{sys                   NB. header pointer
assert h = 15!:12 <y        NB. J 9.04
fheader=. memr h,0 7 4      NB. fixed header
'offset flag size type refcount len misc'=. fheader
rank =. 16b3f (17 b.) misc
shape=. memr h,56,rank,4    NB. shape
assert (h+offset) = 15!:14 <y  NB. J 9.04
data=. memr (h+offset),0,len,type
sys;fheader;shape;data
)

For example:

   viewnoun 'abc'
┌───────────────────────────┬──────────────────────┬───┬─────────────────┐
│2969659998784 2969663040576│72 0 1 4 1 6 106037250│2 3│10 11 12 13 14 15│
└───────────────────────────┴──────────────────────┴───┴─────────────────┘

Now define def as copy of abc. Note that the header pointer for def is the same as for abc, i.e. both nouns point to the same memory; and that the reference count is incremented to 2:

   def=: abc
   viewnoun 'abc'
┌───────────────────────────┬──────────────────────┬───┬─────────────────┐
│2969659998784 2969663040576│72 0 1 4 2 6 106037250│2 3│10 11 12 13 14 15│
└───────────────────────────┴──────────────────────┴───┴─────────────────┘
   viewnoun 'def'
┌───────────────────────────┬──────────────────────┬───┬─────────────────┐
│2969659989568 2969663040576│72 0 1 4 2 6 106037250│2 3│10 11 12 13 14 15│
└───────────────────────────┴──────────────────────┴───┴─────────────────┘

Now redefine def as a vector. This creates a new memory area for def and sets the rank and shape.

   def=: ,def
   viewnoun 'def'
┌───────────────────────────┬──────────────────────┬─┬─────────────────┐
│2969659989568 2969663038144│64 0 1 4 1 6 101056513│6│10 11 12 13 14 15│
└───────────────────────────┴──────────────────────┴─┴─────────────────┘

Now add a new row to abc. The new data fits into the reserved size, so there is no data movement, i.e. the header pointer is unchanged:

   abc=: abc,7
┌───────────────────────────┬────────────────────┬───┬───────────────────────┐
│2969659998784 2969660045888│72 0 1 4 1 9 1835010│3 3│10 11 12 13 14 15 7 7 7│
└───────────────────────────┴────────────────────┴───┴───────────────────────┘

Creating a J noun

We can create a noun by creating the data, the header and the symbol table entry. For example, we create noun mydata with value i.2 3 :

   d=: mema 24
   (i.6) memw d,0 6 4
   h=: allochdr 40
   hdr=: (d-h),0 24 4 1 6 2 2 3
   hdr memw h,0 9 4
   mydata=: symset h
   mydata
0 1 2
3 4 5

   viewnoun'mydata'           NB. in this case, header and data are not contiguous
┌───────────────────┬─────────────────────┬───┬───────────┐
│149241240 146224904│_9536336 0 24 4 2 6 2│2 3│0 1 2 3 4 5│
└───────────────────┴─────────────────────┴───┴───────────┘

Mapped Files

A mapped noun is a noun where the data, or both the data and header, are mapped to a file. Changing the noun changes the file.

The verb map_jmf_ does this mapping - this verb handles several minor details related to mapping a noun, but the core logic is straightforward.

A mapped file can have one of two forms:

• it contains both header and data, and hence is self-describing

• it contains only the data, in which case a suitable header is constructed when mapping the file

When mapping a file, the verb map_jmf_ maps the file to get a memory address to it, and creates a new symbol table entry. Then,

• for self-describing files, the symbol table entry points to the file memory address, which is the address of the header

• for data-only files, a new header is created in memory, and assigned appropriate values. The address of this new header is set in the symbol table entry, while the memory address to the file is set as an offset in the new header.