NYCJUG/2015-04-14

Beginner's regatta

Reversing a String

A search on the web for how to reverse a string in Excel returned the following examples. [From http://www.excel-easy.com/vba/examples/reverse-strings.html]

Dim text As String, reversedText As String, length As Integer, i As Integer
text = InputBox("Enter the text you want to reverse")
length = Len(text)
For i = 0 To length - 1
    reversedText = reversedText & Mid(text, (length - i), 1)
Next i

This brings up a dialog box that allows you to enter the string by hand. --- [From http://www.extendoffice.com/documents/excel/1146-excel-reverse-string-word-order.html#a2] Here we have two methods: one using a formula, the other using a VBA module that requires you to enter the range on which to work; this one replaces the existing text.

=IF(LEN(A1)<1,"",MID(A1,LEN(A1),1))&IF(LEN(A1)<2,"",MID(A1,LEN(A1)-1,1))&IF(LEN(A1)<3,"",MID(A1,LEN(A1)-2,1))&IF(LEN(A1)<4,"",MID(A1,LEN(A1)-3,1))&IF(LEN(A1)<5,"",MID(A1,LEN(A1)-4,1))

And this monstrosity

Sub ReverseText()
'Updateby20131128
Dim Rng As Range
Dim WorkRng As Range
On Error Resume Next
xTitleId = "KutoolsforExcel"
Set WorkRng = Application.Selection
Set WorkRng = Application.InputBox("Range", xTitleId, WorkRng.Address, Type:=8)
For Each Rng In WorkRng
xValue = Rng.Value
xLen = VBA.Len(xValue)
xOut = ""
For i = 1 To xLen
getChar = VBA.Right(xValue, 1)
xValue = VBA.Left(xValue, xLen - i)
xOut = xOut & getChar
Next
Rng.Value = xOut
Next
End Sub

--- [From http://excelexperts.com/reverse-string]

Perhaps the best of the lot, this uses a formula similar to the one above, though it is sensitive to the length of the string, returning an incorrect result if the length is not 5, though it is easily extendable.

=LEFT(RIGHT(A1,1),1)&LEFT(RIGHT(A1,2),1)&LEFT(RIGHT(A1,3),1)&LEFT(RIGHT(A1,4),1)&LEFT(RIGHT(A1,5),1)

Reversing a String in J

In J, we use “|.” to reverse a string:

  |. 'hallelujah!'

!hajulellah

Of course, this works with other vectors as well:

   |. 1 2 3
3 2 1

It also applies to higher-dimensional arrays in the same way as all similar J verbs:

   ]mat=. 'Madam','I''m',:'Adam'
Madam
I'm  
Adam 
   |.mat
Adam 
I'm  
Madam

Like other verbs, it can also be modified with the “rank” conjunction to change how it is applied:

   |."1 |.mat
 madA
  m'I
madam

Used dyadically, the same verb extends to other useful functionality:

   4|.'not really '
really not 
   1|.mat
I'm  
Adam 
Madam
   1|."1 mat
adamM
'm  I
dam A

2D Probability Distribution Plot

from: Ryan <reckbo@bwh.harvard.edu> to: Programming forum <programming@jsoftware.com> date: Tue, Jan 20, 2015 at 11:48 PM subject:[Jprogramming] 2D probability distribution plot I have a matrix of probabilities that I'd like to visualise similar to this: where the bigger the probability, the bigger the square. The closest I've come is a simple density plot mymat=? 27 27$0 load 'graphics/plot viewmat' 'density' plot mymatrix NB. or viewmat mymatrix I've looked through plot's source code but don't see an option for something like this. Does someone know if it's possible to do? thanks for any help, Ryan

---

from: Raul Miller <rauldmiller@gmail.com> date: Wed, Jan 21, 2015 at 1:55 AM It's certainly possible. You didn't give any data, and I do not feel like trying to replicate your original numbers from the image you gave, so I will just use some arbitrary numbers: data=: ?10 10$0 Now, let's say that your squares are 25 pixels on a side. We can form squares like this: $square=: (>./~|i:12)%12 25 25 That's a bit big for email, so here's a smaller version: 4%~>./~|i:4 1 1 1 1 1 1 1 1 1 1 0.75 0.75 0.75 0.75 0.75 0.75 0.75 1 1 0.75 0.5 0.5 0.5 0.5 0.5 0.75 1 1 0.75 0.5 0.25 0.25 0.25 0.5 0.75 1 1 0.75 0.5 0.25 0 0.25 0.5 0.75 1 1 0.75 0.5 0.25 0.25 0.25 0.5 0.75 1 1 0.75 0.5 0.5 0.5 0.5 0.5 0.75 1 1 0.75 0.75 0.75 0.75 0.75 0.75 0.75 1 1 1 1 1 1 1 1 1 1 (That's basically the same expression, I just moved the division over to the left hand side, and changed the 12 to a 4, so it's 5 by 5. The point isn't the expression so much as the resulting data. It's arranged as a square with values ranging from 0 in the center to 1 at the edge.) Now, we just need to compare this to each of our original values, form them into a square, and pass that on to viewmat: require'viewmat' viewmat ,/0|:,/0 2|:data >:/ square Or would an explanation of the steps in any of those expressions help? (You can shave off parts of an expression to see what was passed along, and you can shrink the data so you can inspect those intermediate results easier. You can also use J's trace facility and/or dissect. You can also of course read up on things in the dictionary. But sometimes, especially after you've tried some of those other approaches, it can be good to have someone else give their perspective - not so much because it's not simple but because having other people's words can help you think about things from a slightly different perspective.) Thanks,

---

from: Devon McCormick <devonmcc@gmail.com> date: Wed, Jan 21, 2015 at 8:37 AM Raul's example assumes your data is between 0 and 1, inclusive, so you may need to scale it: 13 : 'y-<./,y' NB. Subtract the smallest from each ] - [: <./ , 13 : 'y%>./,y' NB. Divide each by the largest ] % [: >./ , scale=: (] % [: >./ ,)@(] - [: <./ ,) NB. Combine to scale from 0 to 1 So, if data=: scale +/~i.10 then, doing what Raul showed: viewmat ,/0|:,/0 2|:data >:/ square

---

from: Ryan <reckbo@bwh.harvard.edu>
date: Tue, Jan 27, 2015 at 2:28 AM

Wow, that's awesome, thank you! Exactly what I need. The way you merge the squares is something I haven't seen until now.

---

from: Ryan <reckbo@bwh.harvard.edu>
date: Tue, Jan 27, 2015 at 4:33 AM

Thanks Devon.

FYI I'm learning J by implementing the exercises in Information theory, inference, and learning algorithms (Mackay 03) http://www.inference.phy.cam.ac.uk/mackay/itila/

Show-and-tell

We looked at slides from a recent presentation, at the Data Rave Meetup, about using J with large data arrays.

Working with Large Correlation Matrixes Using J

Displaying and Working with Matrixes

from: j-programming@googlegroups.com
to: Digest recipients <j-programming@googlegroups.com>
date: Mon, Jan 26, 2015 at 1:56 AM

"Björn Helgason" <gosinn@gmail.com>: Jan 25 03:43PM

Working with matrixes in J are easy once you have crossed some initial barriers.
Over the years I have been interested in having something like spreadsheet facility in J to work with tables.
There have been some grid examples and demos introduced.
Many of them have been interesting.
Recently a new facility was introduced in JHS.
It looked interesting but I have not looked at it more than at a glance until now.
I have been playing with it a bit and at first it looked a bit limited and in a traditional J fashion the info was a bit cryptic and terse.
The example given is a small table is small.
Just a few rows and columns.
I tested what would happen with a table with 444 rows and 444 columns.
At first all rows showed but the columns only showed the columns A up to U.
I thought this was a limit that needed configuration.
Investigated all kinds of stuff but found few clues.
By some experiments I accidentally discovered that trying to scroll to the right the columns started to appear!!
I was happy with this and the columns past Z came as AA nd so on and after AZ came BA so it seems to work fine.
No problem editing contents of cells.
Add cells, rows and columns here and there inside the table.
This is a really great facility and working with tables is easy.
You can use text and/or numbers.
It is like a complete spreadsheet added inside all of the J functionality.
I am really impressed and I am sure it will assist many.
While I was stuck during my dilemma with the large table I sent out an SOS and no-one replied so I guess there may be a surprise for many others too how nice this is.

Try:

   n=.33 44$i.111
   jtable 'e1';'n'

scroll to the right.
add 9 in the column AT which was empty and a new column AU appears.
You can right click on a cell and then add a column or row.
After you save with the left top button you have a new n with all the changes you made.
You can then work with this n in the jijx window or your verbs and see the rows and columns you added and the cells you changed in the table.
With the right click you can also remove rows and/or columns.
I tried a mixed matrix and got an assertion failure so that does not work as far as I can tell but I do not consider that a serious limitation.
You can display and work with boxed chars or strings.
So use numbers or chars but not both at the same time.

Try:

   s=.22 22$'aa';'b';'c';'dd';'this is a test'
   jtable 'e2';'s'

If adding a matrix inside cells like:

   s=.22 22$'aa';'b';'c';'dd';22 3$'this is a test'
   jtable 'e2';'s'

The matrix in the last cell will be stretched to a vector and after save s will be changed that way too.
Not a serious limitation as I see it.
All in all a great facility.

---

"Björn Helgason" <gosinn@gmail.com>: Jan 27 12:53PM

NB. Demo of jtable use
 
PEOPLE=:'Larry';'Tom';'Harry';'Siggi';'Kalli';'Sigga';'Magga';'Bob'
HEIGHT=:100+82 63 96 81 63 85 64 59
WEIGHT=: 80 75 163 35 68 73 168 111
 
jtable 'e2';'s' [ s=:(,.PEOPLE),. (":&.> HEIGHT),. ":&.> WEIGHT

NB. Table demo 

 A B C D
1 Larry 182 80
2 Tom 163 75
3 Harry 196 163
4 Siggi 171 45
5 Kalli 163 68
6 Sigga 185 73
7 Magga 164 168
8 Bob 159 111
9
 
NB. I added the line jhh1 into the jtable.ijs
 
NB. HBS=: 0 : 0
NB. jhh1'Table demo'
 
   BMI=:6j0 ": &.> WEIGHT%(2^~HEIGHT%100)
   jtable 'e3';'t' [ t=:s,.BMI
 
NB. Table demo
 
A B C D E
1 Larry 182 80 24
2 Tom 163 75 28
3 Harry 196 163 42
4 Siggi 171 45 11
5 Kalli 163 68 26
6 Sigga 185 73 21
7 Magga 164 168 62
8 Bob 159 111 44
9
 
   jtable 'e4';'v' [ v=:(|:('Name';'Height';'Weight';'BMI'),.<8$'-'),t
 
NB. Table demo
 
A B C D E
1 Name Height Weight BMI
2 -------- -------- -------- --------
3 Larry 182 80 24
4 Tom 163 75 28
5 Harry 196 163 42
6 Siggi 171 45 11
7 Kalli 163 68 26
8 Sigga 185 73 21
9 Magga 164 168 62
10 Bob 159 111 44
11
 
NB. The above tables display in the spreadsheet facility jtable and each on their own page/tab
NB. It is possible to edit the display, add lines etc
 
NB. Displaying v on its own in jijx will only display in fixed boxes in the session manager
   v
┌────────┬────────┬────────┬────────┐
│Name    │Height  │Weight  │BMI     │
├────────┼────────┼────────┼────────┤
│--------│--------│--------│--------│
├────────┼────────┼────────┼────────┤
│Larry   │182     │80      │ 24     │
├────────┼────────┼────────┼────────┤
│Tom     │163     │75      │ 28     │
├────────┼────────┼────────┼────────┤
│Harry   │196     │163     │ 42     │
├────────┼────────┼────────┼────────┤
│Siggi   │171     │45      │ 11     │
├────────┼────────┼────────┼────────┤
│Kalli   │163     │68      │ 26     │
├────────┼────────┼────────┼────────┤
│Sigga   │185     │73      │ 21     │
├────────┼────────┼────────┼────────┤
│Magga   │164     │168     │ 62     │
├────────┼────────┼────────┼────────┤
│Bob     │159     │111     │ 44     │
└────────┴────────┴────────┴────────┘

Advanced topics

We look at some work Jon Hough did on ordinary differential equations or ODEs.

Working with ODEs

from: Jon Hough <jghough@outlook.com>
date: Wed, Apr 8, 2015 at 12:09 PM
subject:Re: [Jprogramming] Solving Differential Eqns with J

I came up with my own solution to analytically solving very simple 1st and second order ODEs (i.e. returns the function/ verb solution) with boundary conditions. For example to solve

y +2y' +y = 0, with y(0) = 1 and y'(0) = 0

I do

   (0 1; 0 0; 1 0) SOLVE_ODE 1 2 1
0.999999999999999667&*@:(^@:(_1x&*)) + 0.999999999999999667&*@:(] * ^@:(_1x&*)) NB. i wish it would round to one

It also took care of the repeated root in the second summand.

The three elements of the boxed array in the left argument of SOLVE_ODE are: the derivatives of the boundary conditions; the arguments of the boundary conditions; the results of the functions on the arguments.

Another example: y+4y'-3y=0, with y(1) = 5 and y(-1) = 5

   solution =: (0 2; 1 _1; 5 5) SOLVE_ODE _3 4 1

Result is:

0.00196954874057786878&*@:(^@:(_4.64575131106459072&*)) + 2.62133261178797961&*@:(^@:(0.645751311064590605&*))

   solution 1

gives 5.

My solution also handles some first order linear ODEs :

e.g. 2y' +y = 0, with y'(0) = 1

   a=: ( 1; 0 ; 1 ) SOLVE_ODE 1 2

returns

_2&(^@:(_0.5&*))

My solution didn't follow the (better) matrix approach. I decided to solve it this way just to see how to do such a thing in J. It is also clumsy handling complex solutions, which in theory should be able to be reduced to sin / cosine functions. But I haven't figured a way to do that.

It also cannot handle non-zero arguments on the right side of the ODE.

e.g. y + y' - y = cos(x)

is out of the question.

Anyway, I have added my source below. I would appreciate any comments for improving the general J-ness of my code. If you look at the end of BOUND_CONDITIONS, I ran into a problem trying to return the verb on the last line (before the final end.) Returning conditions&r1 did not give me the actual verb solution. Any explanation of what my mistake was would be appreciated.

Also, I was able to write a shorter program by calculating both summands together (i.e. both exponentials). The problem then is that I cannot call d. on this, e.g.

   F =: +/@:((1 2)&*)@:^@:(_1 1&*)
   (F d. 1) 1 NB. differentiate and evaluate at x = 1.

This gives an error. It seems d. only handles scalar arguments of verbs (?). And I really want to be able to differentiate my solution.

My code:

ROOTS=: >@:(1&{)@p.

G=: 1 : '^@:(m&*)'

SOLVE=: 1 : ' (ROOTS m) G2'

SOLVEn=: 2 : '(n{ (ROOTS m)) G2'

NB. Gets the solution
G2=: 1 : 0
   sols=: m
   if. (# sols ) = 2 do.
       if. =/ sols do.
           ((>:)*(^@:(_1x&*)))
       elseif. 1 do. ^@:(m&*)
       end.
    else.
       ^@:(m&*)
    end.
)

NB. Returns general solution verb, without constant coefficients.
SOLVE=: 1 : ' (ROOTS m) G2'

NB. Solution for duplicate roots.
SOLVE_DUPLICATE=: 2 : 0
   root=. 0{ (ROOTS n)
   ((0{m)&*@:(^@:(root&*)))+ ((1{m)&*@:(]*(^@:(root&*))))
)

NB. Boundary Conditions. Returns the coefficients
NB. of the soluton's summands, calculated from
NB. the given boundary conditions.
BOUND_CONDITIONS=: conjunction define
   coeffs=. y      NB. polynomial coefficients
   deriv=. >0{ x NB. the orders of derivatives
   val=. >1{ x NB. values to put in
   res=. >2{ x NB. the values' results, i.e. y^(deriv)(val) = res
   if. (# coeffs) = 3 do. NB. Quadratic ODE (e.g. y''+y'+y = 0)
       if. =/ (ROOTS y) do.
           c1=. 1 :'^@:((0{ (ROOTS m))&*)'
           c2=. 1 :']*(^@:((0{ (ROOTS m))&*))'
           r1=. y c1
           r2=. y c2
NB. non-repeating roots. Get the two summands of the
NB. solution independently.
       else.
           r1=. y SOLVEn 0
       r2=. y SOLVEn 1
   end.
NB. solutions - for quadratics, differentiate the
NB. summands and input the values.
NB. This gets four numbers, we can put into a matrix to find
NB. the 2 coefficients of the summands.
   dr1=. (r1 d. (0{deriv)) (0{val)
   dr2=. (r2 d. (0{deriv)) (0{val)
   dr3=. (r1 d. (1{deriv)) (1{val)
   dr4=. (r2 d. (1{deriv)) (1{val)
NB. matrixify, get the coefficient constants
   mat=. (2 2) $ dr1, dr2, dr3, dr4
   mat=. %. mat
   constants=. mat (+/ . *) res
   constants NB. these are the two constants for summands 1 and 2.
   if. =/ (ROOTS y) do.
       ((0{constants)&*@:(y c1)) + ((1{constants)&*@:(y c2))
   else.
       ((0{constants)&*@:(y SOLVEn 0)) + ((1{constants)&*@:(y SOLVEn 1))
   end.
   elseif. (# coeffs) = 2 do. NB. order 1 eqn (e.g. y'+y = 0)
       r1=. y SOLVEn 0 NB. ge the general solution, without coefficient.
       dr1=. (r1 d. (0{deriv)) (0{val)
       constant=. res % dr1 NB. coefficient.
       constant &(y SOLVEn 0)    NB. returning constant&r1 doesn't work, so (y SOLVEn 0) needed to be called again here.
   end.
)

SOLVE_ODE =: conjunction define
   if. (# n) = 3 do.
       if. =/( ROOTS n) do.
           (m BOUND_CONDITIONS n)
       else.
           (m BOUND_CONDITIONS n )
       end.
       elseif. (# n ) = 2 do.
           (m BOUND_CONDITIONS n )
   end.
)

Threading Multiple J Instances

Date: Tue, 24 Feb 2015 08:05:42 -0500
From: Joe Bogner <joebogner@gmail.com>
To: source@jsoftware.com
Subject: [Jsource] threading / multiple J instances

I'm building a server that will create a J instance per request. JInit() works differently on windows and linux.

It looks like nearly all of the interpreter's state is held on a local J struct. The struct is created using JInit() and subsequently is used in JDo / JSetM / JGetM

J seems reasonably thread safe on windows. I know there is some static behavior with dll callbacks and possibly some other areas, but can I use a new J instance on each thread reliably if I stay away from those areas?

On Linux, the behavior is different, JInit is implemented as:

J JInit(void){
  J jt;
  /* jtglobinit must be done once when dll is first loaded
     Windows does it in dll load routine  - thread safe
     Unix does it here once, but this is not thread safe */

  static J g_jt=0;
  if(!g_jt)
  {
    g_jt=malloc(sizeof(JST));
    if(!g_jt) R 0;
    memset(g_jt,0,sizeof(JST));
    if(!jtglobinit(g_jt)){free(g_jt);g_jt=0; R 0;}
  }
  RZ(jt=malloc(sizeof(JST)));
  memset(jt,0,sizeof(JST));
  if(!jtjinit2(jt,0,0)){free(jt); R 0;};
  R jt;
}

//
https://github.com/openj/core/blob/18fd23bbdc2f50770eb3047e978cd5e4e3b47039/io.c

More importantly, JFree doesn't do anything:

int JFree(J jt){return 0;}

This means my JInit/JFree pattern won't work on linux as I'll continue to accumulate memory for each thread.

Questions:

1. Does J support multiple instances within the same process, or do I need to be spinning up new processes for each request?

2. What is the purpose of g_jt? I don't see it called elsewhere. Is this just a flag to determine if jtglobinit was called? jtglobinit appears to set up an instance of a J struct and discard it. I can't figure out what jtglobinit is doing unless it's affecting some static/shared variable that's not readily apparent

Thanks, Joe

---

Date: Tue, 24 Feb 2015 08:23:32 -0500
From: Brian Schott <schott.brian@gmail.com>

Joe,

Have you looked at JUM with JHS? It may not be relevant, but my understanding is that it enables JHS to be serve multiple users and may address some of your questions. -- (B=)

---

Date: Tue, 24 Feb 2015 09:57:18 -0500
From: Joe Bogner <joebogner@gmail.com>

Thanks Brian, I hadn't looked at JUM previously. It looks like it spawns a new jconsole jhs process & port per user and the user connects over the new port. That is less than ideal in my situation. Thank you for suggesting it though.

It looks like there's a fundamental difference in linux memory management and windows memory management that I hadn't encountered. This will make the idea much more complex to implement on linux, which means I may need to go back to spawning processes and doing some form of inner process communication.

On windows, memory is allocated from a private heap per J instance (jt->heap)

JFree can wipe out that entire heap and not deallocate objects individually.

Linux doesn't have a similar 'out of the box' capability. It looks like I would need to roll my own allocate using mmap to be able to free all allocated memory at the end of the request. Alternatively, I could try to keep track of all allocations in a linked list and free them, which might be possible but could also be problematic.

The simple answer on linux may be to just fork the process instead

#if SY_WIN32 && !SY_WINCE
#define FREE(a)     HeapFree(jt->heap,0,a)
#define MALLOC(n)   (void*)HeapAlloc(jt->heap,0,n)
#else
#define FREE(a) free(a)
#define MALLOC(n) malloc(n)
#endif

---

Date: Tue, 24 Feb 2015 10:41:38 -0500
From: Thomas Costigliola <tcostigl@gmail.com>

Hi, Joe, it's great to see someone interested in and attempting this. My understanding is that JInit is not thread safe, so, for example you cannot start up your threads and call JInit in each. But you can call JInit multiple times in the same thread and pass the J pointers to your worker threads after.

> 2. What is the purpose of g_jt? I don't see it called elsewhere....

The static g_jt is to makes sure that the global init function is not called twice. This sets up global variables that lay outside of the JT struct. Most are read only and can be used safely from multiple threads. There are a few I am not sure about, depending on what kind of J code your threads will be executing.

I would be interested in discussing this further with you, it could be helpful to pick each other's brains.

---

Date: Tue, 24 Feb 2015 11:40:37 -0500
From: Joe Bogner <joebogner@gmail.com>

Hi Thomas, thanks for the feedback.

Here's my opinion on the thread safety, but I'm not an expert. The main reason that JInit() isn't thread-safe is the use of the static g_jt. There could be a race condition if multiple threads are accessing that global at the same time. It may be possible to call JInit in a startup routine and then let each thread call it subsequently once the first call has completed.

For more background, I'd like to publish a multi-threaded J web server. I'm currently running down the path of having go (golang) call J through it's C interface. The go httphandler calls JInit and JFree for each request. On windows, I've been able to get a respectable # of requests/per second. Calling J only seems to be a 10x slowdown, so I'm running at roughly 1000 requests/sec vs 12000 requests/seq with stock golang and a hello world handler. I've tested sleeps in the J code and everything seems to be working well so far. The memory remains stable through a load test and properly threads as the

Here's a load test on windows (2 cpus, 50 concurrent requests, and 10,000 total requests)

C:\dev\tools\boom>boom -c 50 -n 10000 -cpus 2 http://127.0.0.1:8080/
10000 / 10000 Booooooooooooooooooooooooooooooooooooooooooooooooooooo!
100.00 %

Summary:
  Total:        9.9266 secs.
  Slowest:      0.1780 secs.
  Fastest:      0.0020 secs.
  Average:      0.0495 secs.
  Requests/sec: 1007.3976
  Total Data Received:  250014 bytes.
  Response Size per Request:    25 bytes.

Status code distribution:
  [200] 10000 responses

Response time histogram:
  0.002 [14]    |
  0.020 [255]   |??
  0.037 [2035]  |?????????????????
  0.055 [4608]  |????????????????????????????????????????
  0.072 [1970]  |?????????????????
  0.090 [722]   |??????
  0.108 [256]   |??
  0.125 [89]    |
  0.143 [32]    |
  0.160 [13]    |
  0.178 [6]     |

Latency distribution:
  10% in 0.0320 secs.
  25% in 0.0380 secs.
  50% in 0.0450 secs.
  75% in 0.0580 secs.
  90% in 0.0740 secs.
  95% in 0.0850 secs.
  99% in 0.1120 secs.

The http-server-j.exe process maxes out at about 100MB and uses upwards of 50% cpu

On linux, I tried porting over the same code and that's where I ran into the issue of memory leaks, but I may have been able to work around it. The server would grow at roughly 1MB per request and quickly exhausted the available memory on my host

On the gpl source, I switched malloc and free to use talloc ( https://talloc.samba.org/talloc/doc/html/index.html)

It was a small change to m.h:

#if SY_WIN32 && !SY_WINCE
#define FREE(a)     HeapFree(jt->heap,0,a)
#define MALLOC(n)   (void*)HeapAlloc(jt->heap,0,n)
#else
#include <talloc.h>
---> #define FREE(a) talloc_free(a) //was free(a)
---> #define MALLOC(n) talloc_size(jt->heap,n) //was malloc(n)
#endif

As you can see, it operates similarly to the windows HeapAlloc by using a private memory pool per J instance.

in io.c, I had to add a logic to initialize and free it:

J JInit(void){
  J jt;
  /* jtglobinit must be done once when dll is first loaded
     Windows does it in dll load routine  - thread safe
     Unix does it here once, but this is not thread safe */

  static J g_jt=0;
  if(!g_jt)
  {
        talloc_enable_leak_report_full();
        g_jt=malloc(sizeof(JST));
        if(!g_jt) R 0;
        memset(g_jt,0,sizeof(JST));
        g_jt->heap = _talloc(NULL,0);
        printf("allocating\n");
        fflush(stdout);
        if(!jtglobinit(g_jt)){free(g_jt);g_jt=0; R 0;}
  }
  RZ(jt=malloc(sizeof(JST)));
  memset(jt,0,sizeof(JST));
  jt->heap = _talloc(NULL,0);
  if(!jtjinit2(jt,0,0)){free(jt); R 0;};
  R jt;
}

int JFree(J jt){
  //printf("all memory: %d\n", talloc_total_size(NULL));
  //printf("freeing: %d\n", talloc_total_size(jt->heap));
  //fflush(stdout);
  talloc_free(jt->heap);
  free(jt);
  return 1;
}

Now, on my $5 digital ocean linux box, I can run a similar load test:

joebo@joebo:~/dev/tools/bin$ ./boom -c 50 -n  10000  -cpus 1
http://localhost:8080/foo
10000 / 10000
Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo!
100.00 %

Summary:
  Total:        12.4409 secs.
  Slowest:      0.4593 secs.
  Fastest:      0.0009 secs.
  Average:      0.0621 secs.
  Requests/sec: 803.8021
  Total Data Received:  40000 bytes.
  Response Size per Request:    4 bytes.

Status code distribution:
  [200] 10000 responses

Response time histogram:
  0.001 [1]     |
  0.047 [4682]  |????????????????????????????????????????
  0.093 [3577]  |??????????????????????????????
  0.138 [1173]  |??????????
  0.184 [328]   |??
  0.230 [157]   |?
  0.276 [42]    |
  0.322 [22]    |
  0.368 [10]    |
  0.413 [5]     |
  0.459 [3]     |

Latency distribution:
  10% in 0.0234 secs.
  25% in 0.0348 secs.
  50% in 0.0494 secs.
  75% in 0.0772 secs.
  90% in 0.1137 secs.
  95% in 0.1443 secs.
  99% in 0.2222 secs.

This is all experimental at this stage, but results look promising. My work in progress is at:

https://github.com/joebo/lang-lab/tree/master/go/http-j

This is the windows go server: https://github.com/joebo/lang-lab/blob/master/go/http-j/http-j.go and the linux one: https://github.com/joebo/lang-lab/blob/master/go/http-j/http-j-linux.go

On windows, the pre-requisites I used are:

1. golang
2. tinyc (to build shim dll)
3. j803 (no changes)

On linux:

1. golang
2. gcc (uses cgo for the interop)
3. talloc
4. J gpl source with modifications (not yet published)

---

Date: Tue, 24 Feb 2015 12:30:00 -0500
From: Joe Bogner <joebogner@gmail.com>

I have a running example up at http://csilo.com:8080/

I'm not sure how long it will be up but I'll keep it up for today at least. Lots to do still as this just scratches the surface

---

Date: Tue, 24 Feb 2015 17:34:45 -0500
From: Raul Miller <rauldmiller@gmail.com>

On a perhaps related note:

The last time I looked (almost a decade ago), spawning a process on linux had about the same cost as spawning a thread on windows.

Thanks,

-- Raul

---

Date: Tue, 24 Feb 2015 19:59:58 -0500
From: Joe Bogner <joebogner@gmail.com>

Yes, forking or spawning processes is fairly cheap on linux. I opted against it because I wanted a similar server implementation on windows and linux. Forking wasn't a viable option on windows. I have messed with forking awhile back: https://github.com/joebo/j-fork-server .. which used WSADuplicateSocket and spawned the process to handle the request

I chose go because I wasn't particularly interested in writing a http parser and high performance socket handling. I figured I'd leave that (for now) to someone else and use a compiled language. I intend to write my J code against an abstraction that will allow me to swap it out in the future if needed. There's no reason that JHS or a different J http server couldn't be used at some point down the road for development. The code could then be deployed to a concurrent web server like this for production traffic

---

Date: Tue, 24 Feb 2015 20:36:45 -0500
From: Raul Miller <rauldmiller@gmail.com>

For what it's worth... I built an "http server" this weekend, and so far it has been working reasonably well for my purposes. (http://github.com/rdm/geoh)

For my purposes, an "http server" was something that listens for connections, reads up to a blank line, and then hands that block of text over to something else which grovels through it, picking out what seems relevant. And "working reasonably well" means serving a few thousand connections at any one time without any measurable load on the server.

I'm still reviewing that particular bit of code, but I think I've fixed most of the worst problems. (And I'm under no illusions that it will solve all problems which it will face - what with distributed denial of service issues and a world full of criminals, script kiddies and wanna-bes, not to mention all the failings of our infrastructures and economies, it's only a matter of time before something goes wrong.)

But as long as you can constrain your problem, you can often do surprisingly well with simple approaches.

Anyways -- other than the sorts of issues noted at http://www.cs.yale.edu/homes/perlis-alan/quotes.html -- I suspect that there's nothing wrong with Go.

Thanks,

-- Raul

---

Date: Tue, 24 Feb 2015 22:31:12 -0500
From: Joe Bogner <joebogner@gmail.com>

Thanks for sharing. I skimmed through the code and it looked solid as far as I could discern in a quick once over. My needs for a HTTP server is less constrained. I need to be able to support GET / query strings, POST data, cookies, redirects, etc. It's mostly just messing around with the HTTP header and body, but still not what I wanted to spend my time solving yet. I also wanted it to be portable across Windows and Linux since I use both operating systems.

It was interesting to see your use of SO_REUSEPORT instead of the traditional forking mechanism. I see it's relatively new to linux. Alas, not supported on Windows

I also needed to handle multiple requests at once so that a longer running request wouldn't block shorter ones.

HTTP pipelining is not something I've done much with either. I would traditionally implement it as a method that accepted multiple values and provided an array of responses or defer to http keep-alive

Your custom server route lets you optimize for the constraints, which can be beneficial. For example, you can mmap the file once per process. It would be interesting to compare/contrast the code, maintenance, performance of the different approaches but I'm sure we all have more than enough to do.

---

Date: Wed, 25 Feb 2015 00:44:15 -0500
From: Raul Miller <rauldmiller@gmail.com>

On Tue, Feb 24, 2015 at 10:31 PM, Joe Bogner <joebogner@gmail.com> wrote:

> Thanks for sharing. I skimmed through the code and it looked solid as far > as I could discern in a quick once over. My needs for a HTTP server is less > constrained. I need to be able to support GET / query strings, POST data, > cookies, redirects, etc. It's mostly just messing around with the HTTP > header and body, but still not what I wanted to spend my time solving yet. > I also wanted it to be portable across Windows and Linux since I use both > operating systems.

I'll not comment on this aspect, since I don't have complete (nor adequate) specs.

> It was interesting to see your use of SO_REUSEPORT instead of the > traditional forking mechanism. I see it's relatively new to linux. Alas, > not supported on Windows

I guess I'm glad that Linux had it. (My original versions of this code were done on OpenBSD - to make sure I got the basic networking right, then I did some work on os x yosemite, but I'm using an ubuntu system to run it on.)

That said, I've found that (for most of what I do) supporting Windows eats far more time than I'm willing to invest. I prefer to have other people deal with Windows, when I can. The exceptions are when stuff only works on Windows (and someone else is willing to foot the time and tools bill).

I don't mind using Windows, it can be entertaining and it can do simple useful stuff, but as you point out, things that work well elsewhere often do not work well on Windows (and vice versa).

> I also needed to handle multiple requests at once so that a longer running > request wouldn't block shorter ones.

The server I pointed you at limits itself to 1023 concurrent requests in its current implementation. And my current test deployment runs 16 instances of it. This isn't an issue for me, in my current context.

But of course, you're doing something very different.

> HTTP pipelining is not something I've done much with either. I would > traditionally implement it as a method that accepted multiple values and > provided an array of responses or defer to http keep-alive

I have mixed feelings about pipelining. On the one hand, it just about doubles the complexity of the server. On the other hand, tcp connection overhead is probably the most expensive part of the system, for my application at least.

At some point, design decisions become non-transferable (or cost more than they're worth, in whatever the new context is).

> Your custom server route lets you optimize for the constraints, which can > be beneficial. For example, you can mmap the file once per process. It > would be interesting to compare/contrast the code, maintenance, performance > of the different approaches but I'm sure we all have more than enough to do.

Yeah... I mostly implemented it this way because it was going to be more trouble figuring out how to get an existing http server (apache, or node's http server) to behave the way I needed to behave.

Sometimes you need to think carefully, but sometimes you just need to get something done.

Thanks,

---

Date: Fri, 27 Feb 2015 15:56:47 -0500
From: Joe Bogner <joebogner@gmail.com>

It looks like profile crash is caused by concurrent calls to setbreak, which is in boot.ijs. I'm not surprised by that. I've commented it out and loading the profile from multiple instances/threads seems stable

The memory mapped file test is also running stable when the mapping is read only. It still crashes with read/write mapped files, which is not an issue for me since I am using the mapped file as a read-only database

It's quite slow, but at least it's stable. I can work with slow and speed it up...

vagrant@precise64:~/tools/wrk$ ./wrk -t3 -c10 -d5m
http://127.0.0.1:8080/ipLookup/
Running 5m test @ http://127.0.0.1:8080/ipLookup/
  3 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   551.51ms  155.60ms   1.59s    81.75%
    Req/Sec     5.16      1.16     9.00     69.83%
  5085 requests in 5.00m, 1.71MB read
Requests/sec:     16.94
Transfer/sec:      5.82KB

Learning and Teaching J

Here we look at some excerpts from an essay on programming paradigms.

(excerpts from) Programming Paradigms

Structured Programming

Structured programming is a kind of imperative programming where the control flow is defined by nested loops, conditionals, and subroutines, rather than via gotos. Variables are generally local to blocks (have lexical scope).

result = [];
for i = 0; i < length(people); i++ {
    p = people[i];
    if length(p.name)) > 5 {
        addToList(result, toUpper(p.name));
    }
}
return sort(result);

Early languages emphasizing structured programming: Algol 60, PL/I, Algol 68, Pascal, C, Ada 83, Modula, Modula-2. Structured programming as a discipline is sometimes though to have been started by a famous letter by Edsger Dijkstra entitled Go to Statement Considered Harmful.

---

Functional Programming

In functional programming control flow is expressed by combining function calls, rather than by assigning values to variables.

let(
  f, fun(
    people,
    if(equals(people, emptylist),
        emptylist,
        if(greater(length(name(head(people))), 5),
          append(to_upper(name(head(people))), f(tail(people))),
          f(tail(people))))),
    sort(f(people)))

Of course, there's usually syntactic sugar

let
    fun f [] = []
      | f (p :: ps) =
          if p.name.length() > 5 then p.name.to_upper()::(f ps)
          else (f ps)
in
    sort(f(people))

The real power of this paradigm comes from passing functions to functions (and returning functions from functions).

sort(
    filter((λs. s.length() > 5),
        map((λp. p.name.to_upper()), people)

Read Joel Spolsky's article on map and reduce. [Following: “Can Your Programming Language Do This? ]

[How we might do this in J:

   #&>people=. 'ava';'freddy';'mike';'patsy';'patrick'
3 6 4 5 7
   ]`toupper @. (5 >~ #) &.> people
+---+------+----+-----+-------+
|ava|FREDDY|mike|patsy|PATRICK|
+---+------+----+-----+-------+
]

With functional programming • There are no commands, only side-effect free expressions • Code is much shorter, less error-prone, and much easier to prove correct • There is more inherent parallelism, so good compilers can produce faster code

Some people like to say • Functional, or Applicative, programming is programming without assignment statements: one just applies functions to arguments. Examples: Scheme, Haskell, Miranda, ML. • Function-level programming does away with the variables; one combines functions with "functionals". Examples: FP, FL, J.

Exercise: Write the above example in Miranda, ML, and J.

Python has a neat little thing called list comprehensions that combine map and filter.

sorted([p.name.upper() for p in people if len(p.name) > 5])

[From https://www.info.ucl.ac.be/~pvr/paradigms.html]

Explanations

See “Concepts, Techniques, and Models of Computer Programming”.

The chart classifies programming paradigms according to their kernel abstractions can be defined). Kernel languages are ordered according to the creative extension principle: a new concept is added when it cannot be encoded with only local transformations. Two languages that implement the same paradigm can nevertheless have very different "flavors" for the programmer, because they make different choices about what programming techniques and styles to facilitate.

When a language is mentioned under a paradigm, it means that part of the language is intended (by its designers) to support the paradigm without interference from other paradigms. It does not mean that there is a perfect fit between the language and the paradigm. It is not enough that libraries have been written in the language to support the paradigm. The language’s kernel language should support the paradigm. When there is a family of related languages, usually only one member of the family is mentioned to avoid clutter. The absence of a language does not imply any kind of value judgment.

State is the ability to remember information, or more precisely, to store a sequence of values in time. Its expressive power is strongly influenced by the paradigm that contains it. We distinguish four levels of expressiveness, which differ in whether the state is unnamed or named, deterministic or nondeterministic, and sequential or concurrent. The least expressive is functional programming (threaded state, e.g., DCGs and monads: unnamed, deterministic, and sequential). Adding concurrency gives declarative concurrent programming (e.g., synchrocells: unnamed, deterministic, and concurrent). Adding nondeterministic choice gives concurrent logic programming (which uses stream mergers: unnamed, nondeterministic, and concurrent). Adding ports or cells, respectively, gives message passing or shared state (both are named, nondeterministic, + local cell and concurrent). Nondeterminism is important for real−world interaction (e.g., client/server). Named state is important for modularity.

Axes orthogonal to this chart are typing, aspects, and domain−specificity. Typing is not completely orthogonal: it has some effect on expressiveness. Aspects should be completely orthogonal, since they are part of a program’s specification. A domain−specific language should be definable in any paradigm (except when the domain needs a particular concept).

Metaprogramming is another way to increase the expressiveness of a language. The term covers many different approaches, from higher−order programming, syntactic extensibility (e.g., macros), to higher−order programming combined with syntactic support (e.g., meta−object protocols and generics), to full−fledged tinkering with the kernel language (introspection and reflection). Syntactic extensibility and kernel language tinkering in particular are orthogonal to this chart. Some languages, such as Scheme, are flexible enough to implement many paradigms in almost native fashion. This flexibility is not shown in the chart.

---

Can Your Programming Language Do This?

by Joel Spolsky / Tuesday, August 01, 2006 One day, you're browsing through your code, and you notice two big blocks that look almost exactly the same. In fact, they're exactly the same, except that one block refers to "Spaghetti" and one block refers to "Chocolate Moose."

   // A trivial example:
   
   alert("I'd like some Spaghetti!");
   alert("I'd like some Chocolate Moose!");

These examples happen to be in JavaScript, but even if you don't know JavaScript, you should be able to follow along. The repeated code looks wrong, of course, so you create a function:

   function SwedishChef( food )
   {  alert("I'd like some " + food + "!");
   }

   SwedishChef("Spaghetti");
   SwedishChef("Chocolate Moose");

OK, it's a trivial example, but you can imagine a more substantial example. This is better code for many reasons, all of which you've heard a million times. Maintainability, Readability, Abstraction = Good! Now you notice two other blocks of code which look almost the same, except that one of them keeps calling this function called BoomBoom and the other one keeps calling this function called PutInPot. Other than that, the code is pretty much the same.

   alert("get the lobster");
   PutInPot("lobster");
   PutInPot("water");

   alert("get the chicken");
   BoomBoom("chicken");
   BoomBoom("coconut");

Now you need a way to pass an argument to the function which itself is a function. This is an important capability, because it increases the chances that you'll be able to find common code that can be stashed away in a function.

   function Cook( i1, i2, f )
   {   alert("get the " + i1);
       f(i1);
       f(i2);
   }

   Cook( "lobster", "water", PutInPot );
   Cook( "chicken", "coconut", BoomBoom );

Look! We're passing in a function as an argument. Can your language do this? Wait... suppose you haven't already defined the functions PutInPot or BoomBoom. Wouldn't it be nice if you could just write them inline instead of declaring them elsewhere?

   Cook( "lobster", 
         "water", 
         function(x) { alert("pot " + x); }  );
   Cook( "chicken", 
         "coconut", 
         function(x) { alert("boom " + x); } );

Jeez, that is handy. Notice that I'm creating a function there on the fly, not even bothering to name it, just picking it up by its ears and tossing it into a function. As soon as you start thinking in terms of anonymous functions as arguments, you might notice code all over the place that, say, does something to every element of an array.

   var a = [1,2,3];

   for (i=0; i<a.length; i++)
   {   a[i] = a[i] * 2;
   }

   for (i=0; i<a.length; i++)
   {   alert(a[i]);
   }

Doing something to every element of an array is pretty common, and you can write a function that does it for you:

   function map(fn, a)
   {   for (i = 0; i < a.length; i++)
       {    a[i] = fn(a[i]);
       }
   }

Now you can rewrite the code above as:

   map( function(x){return x*2;}, a );
   map( alert, a );

Another common thing with arrays is to combine all the values of the array in some way.

   function sum(a)
   {   var s = 0;
       for (i = 0; i < a.length; i++) s += a[i];
       return s;
   }
   
   function join(a)
   {   var s = "";
       for (i = 0; i < a.length; i++) s += a[i];
       return s;
   }
   
   alert(sum([1,2,3]));
   alert(join(["a","b","c"]));

sum and join look so similar, you might want to abstract out their essence into a generic function that combines elements of an array into a single value:

   function reduce(fn, a, init)
   {   var s = init;
       for (i = 0; i < a.length; i++) s = fn( s, a[i] );
       return s;
   }
   
   function sum(a)
   {    return reduce( function(a, b){ return a + b; }, a, 0 );
   }
   
   function join(a)
   {    return reduce( function(a, b){ return a + b; }, a, "" );
   }

Many older languages simply had no way to do this kind of stuff. Other languages let you do it, but it's hard (for example, C has function pointers, but you have to declare and define the function somewhere else). Object-oriented programming languages aren't completely convinced that you should be allowed to do anything with functions.

Java required you to create a whole object with a single method called a functor if you wanted to treat a function like a first class object. Combine that with the fact that many OO languages want you to create a whole file for each class, and it gets really klunky fast. If your programming language requires you to use functors, you're not getting all the benefits of a modern programming environment. See if you can get some of your money back. How much benefit do you really get out of writing itty bitty functions that do nothing more than iterate through an array doing something to each element?

Well, let's go back to that map function. When you need to do something to every element in an array in turn, the truth is, it probably doesn't matter what order you do them in. You can run through the array forward or backwards and get the same result, right? In fact, if you have two CPUs handy, maybe you could write some code to have each CPU do half of the elements, and suddenly map is twice as fast.

Or maybe, just hypothetically, you have hundreds of thousands of servers in several data centers around the world, and you have a really big array, containing, let's say, again, just hypothetically, the entire contents of the internet. Now you can run map on thousands of computers, each of which will attack a tiny part of the problem.

So now, for example, writing some really fast code to search the entire contents of the internet is as simple as calling the map function with a basic string searcher as an argument.

The really interesting thing I want you to notice, here, is that as soon as you think of map and reduce as functions that everybody can use, and they use them, you only have to get one supergenius to write the hard code to run map and reduce on a global massively parallel array of computers, and all the old code that used to work fine when you just ran a loop still works only it's a zillion times faster which means it can be used to tackle huge problems in an instant.

Lemme repeat that. By abstracting away the very concept of looping, you can implement looping any way you want, including implementing it in a way that scales nicely with extra hardware. And now you understand something I wrote a while ago where I complained about CS students who are never taught anything but Java:

Without understanding functional programming, you can't invent MapReduce, the algorithm that makes Google so massively scalable. The terms Map and Reduce come from Lisp and functional programming. MapReduce is, in retrospect, obvious to anyone who remembers from their 6.001-equivalent programming class that purely functional programs have no side effects and are thus trivially parallelizable. The very fact that Google invented MapReduce, and Microsoft didn't, says something about why Microsoft is still playing catch up trying to get basic search features to work, while Google has moved on to the next problem: building Skynet^H^H^H^H^H^H the world's largest massively parallel supercomputer. I don't think Microsoft completely understands just how far behind they are on that wave.

Ok. I hope you're convinced, by now, that programming languages with first-class functions let you find more opportunities for abstraction, which means your code is smaller, tighter, more reusable, and more scalable. Lots of Google applications use MapReduce and they all benefit whenever someone optimizes it or fixes bugs.

And now I'm going to get a little bit mushy, and argue that the most productive programming environments are the ones that let you work at different levels of abstraction. Crappy old FORTRAN really didn't even let you write functions. C had function pointers, but they were ugleeeeee and not anonymous and had to be implemented somewhere else than where you were using them. Java made you use functors, which is even uglier. As Steve Yegge points out, Java is the Kingdom of Nouns.

Problems Caused For Students By The Two Languages Of Math

2015/02/26 SIXWINGEDSERAPH

The two languages of math

Mathematics is communicated using two languages: Mathematical English and the symbolic language of math (more about them in two languages).

This post is a collection of examples of the sorts of trouble that the two languages cause beginning abstract math students. I have gathered many of them here since they are scattered throughout the literature. I would welcome suggestions for other references to problems caused by the languages of math.

In many of the examples, I give links to the literature and leave you to fish out the details there. Almost all of the links are to documents on the internet.

There is an extensive list of references.

Conjectures

Scattered through this post are conjectures. Like most of my writing about difficulties students have with math language, these conjectures are based on personal observation over 37 years of teaching mostly computer engineering and math majors. The only hard research of any sort I have done in math ed consists of the 426 citations of written mathematical writing included in the Handbook of Mathematical Discourse.

Disclaimer

This post is an attempt to gather together the ways in which math language causes trouble for students. It is even more preliminary and rough than most of my other posts. • The arrangement of the topics is unsatisfactory. Indeed, the topics are so interrelated that it is probably impossible to give a satisfactory linear order to them. That is where writing on line helps: Lots of forward and backward references. • Other people and I have written extensively about some of the topics, and they have lots of links. Other topics are stubs and need to be filled out. I have probably missed important points about and references to many of them. • Please note that many of the most important difficulties that students have with understanding mathematical ideas are not caused by the languages of math and are not represented here. I expect to revise this article periodically as I find more references and examples and understand some of the topics better. Suggestions would be very welcome.

Intricate symbolic expressions

I have occasionally had students tell me that have great difficulty understanding a complicated symbolic expression. They can’t just look at it and learn something about what it means.

Example

Consider the symbolic expression

Now, I could read this expression aloud as if it were text, or more precisely describe it so that someone else could write it down. But if I am in math mode and see this expression I don’t “read” it, even to myself.

I am one of those people who much of the time think in pictures or abstractions without words. (See references Press/?p=9668#maththinking here.)

In this case I would look at the expression as a structured picture. I could determine a number of things about it, and when I was explaining it I would point at the board, not try to pronounce it or part of it: • The denominator is always positive so the expression is defined for all reals. • The exponent is even so the value of the expression is always nonnegative. I would say, “This (pointing at the exponent) is an even power so the expression is never negative.” • It is zero in exactly one place, namely x=10−−√3. • Its derivative is also 0 at 10−−√3. You can see this without calculating the formula for the derivative (ugh).

There is much more about this example in Zooming and Chunking.

Algebra in high school

There are many high school students stymied by algebra, never do well at it, and hate math as a result. I have known many such people over the years. A revealing remark that I have heard many times is that “algebra is totally meaningless to me”. This is sometimes accompanied by a remark that geometry is “obvious” or something similar. This may be because they think they have to “read” an algebraic expression instead of studying it as they would a graph or a diagram.

Conjecture

Many beginning abstract math students have difficulty understanding a symbolic expression like the one above. Could this be cause by resistance to treating the expression as a structure to be studied?

Context-sensitive pronunciation

A symbolic assertion (“formula” to logicians) can be embedded in a math English sentence in different ways, requiring the symbolic assertion to be pronounced in different ways. The assertion itself is not modified in any way in these different situations.

I used the phrase “symbolic assertion” in abstractmath.org because students are confused by the logicians’ use of “formula“.

In everyday English, “H2O” is the “formula” for water, but it is a term, not an assertion.

Example

“For every real number x>0 there is a real number y such that x>y>0.” • In the sentence above, the assertion “x>0” must be pronounced “x that is greater than 0” or something similar. • The standalone assertion “x>0” is pronounced “x is greater than 0.” • The sentence “Let x>0” must be pronounced “Let x be greater than 0”.

The consequence is that the symbolic assertion, in this case “x>0”, does not reveal that role it plays in the math English sentence that it is embedded in. Many of the examples occurring later in the post are also examples of context-sensitive pronunciation.

Conjectures

Many students are subconsciously bothered by the way the same symbolic expression is pronounced differently in different math English sentences.

This probably impedes some students’ progress. Teachers should point this phenomenon out with examples.

Students should be discouraged from pronouncing mathematical expressions.

For one thing, this could get you into trouble. Consider pronouncing “3+5−−−−√+6”. In any case, when you are reading any text you don’t pronounce the words, you just take in their meaning. Why not take in the meaning of algebraic expressions in the same way?

Parenthetic assertions

A parenthetic assertion is a symbolic assertion embedded in a sentence in math English in such a way that is a subordinate clause.

Example

In the math English sentence

“For every real number x>0 there is a real number y such that x>y>0”

mentioned above, the symbolic assertion “x>0” plays the role of a subordinate clause.

It is not merely that the pronunciation is different compared to that of the independent statement “x>0”. The math English sentence is hard to parse. The obvious (to an experienced mathematician) meaning is that the beginning of the sentence can be read this way: “For every real number x, which is bigger than 0…”.

But new student might try to read it is “For every real number x is greater than0 …” by literally substituting the standalone meaning of “x>0” where it occurs in the sentence. This makes the text what linguists call a garden path sentence.The student has to stop and start over to try to make sense of it, and the symbolic expression lacks the natural language hints that help understand how it should be read.

Note that the other two symbolic expressions in the sentence are not parenthetic assertions. The phrase “real number” needs to be followed by a term,and it is, and the phrase “such that” must be followed by a clause, and it is.

More examples

• “Consider the circle S1⊆C=R2.” This has subordinate clauses to depth 2. • “The infinite series ∑k=1∞1k2 converges to ζ(2)=π26≈1.65” • “We define a null set in I:=[a,b] to be a set that can be covered by a countable of intervals with arbitrarily small total length.” This shows a parenthetical definition. • “Let F:A→B be a function.”

A type declaration is a function? In any case, it would be better to write this sentence simply as “Let F:A→B”.

David Butler’s post Contrapositive grammar has other good examples.

Math texts are in general badly written. Students need to be taught how to read badly written math as well as how to write math clearly. Those that succeed (in my observation) in being able to read math texts often solve the problem by glancing at what is written and then reconstructing what the author is supposedly saying.

Conjectures

Some students are baffled, or at least bothered consciously or unconsciously, by parenthetic assertions, because the clues that would exist in a purely English statement are missing. Nevertheless, many if not most math students read parenthetic assertions correctly the first time and never even notice how peculiar they are. What makes the difference between them and the students who are stymied by parenthetic assertions?

There is another conjecture concerning parenthetic assertions below.

Context-sensitive meaning

“If” in definitions

Example

The word “if” in definitions does not mean the same thing that it means in other math statements. • In the definition “An integer is even if it is divisible by 2,” “if” means “if and only if”. In particular, the definition implies that a function is not even if it is not divisible by 2. • In a theorem, for example “If a function is differentiable, then it is continuous”, the word “if” has the usual one-way meaning. In particular, in this case, a continuous function might not be differentiable.

Context-sensitive meaning occurs in ordinary English as well. Think of a strike in baseball.

Conjectures

The nearly universal custom of using “if” to mean “if and only if” in definitions makes it a harder for students to understand implication.

This custom is not the major problem in understanding the role of definitions. See my article Definitions.

Underlying sets

Example

In a course in group theory, a lecturer may say at one point, “Let F:G→Hbe a homomorphism”, and at another point, “Let g∈G”.

In the first sentence, G refers to the group, and in the second sentence it refers to the underlying set of the group.

This usage is almost universal. I think the difficulty it causes is subtle. When you refer to R, for example, you (usually) are referring to the set of real numberstogether with all its canonical structure. The way students think of it, a real number comes with its many relations and connections with the other real numbers, ordering, field properties, topology, and so on.

But in a group theory class, you may define the Klein 4-group to be Z2×Z2. Later you may say “the symmetry group of a rectangle that is not a square is the Klein 4-group.” Almost invariably some student will balk at this. Referring to a group by naming its underlying set is also an example of synecdoche.

Conjecture

Students expect every important set in math to have a canonical structure. When they get into a course that is a bit more abstract, suddenly the same set can have different structures, and math objects with different underlying sets can have the same structure. This catastrophic shift in a way of thinking should be described explicitly with examples.

Way back when, it got mighty upsetting when the earth started going around the sun instead of vice versa. Remind your students that these upheavals happen in the math world too.

Overloaded notation

Identity elements

A particular text may refer to the identity element of any group as e. This is as far as I know not a problem for students. I think I know why: There is a generic identity element. The identity element in any group is an instantiation of that generic identity element. The generic identity element exists in the sketch for groups; every group is a functor defined on that sketch. (Or if you insist, the generic identity element exists in the first order theory for groups.) I suspect mathematicians subconsciously think of identity elements in this way.

Matrix multiplication

Matrix multiplication is not commutative. A student may forget this and write (A2B2=(AB)2. This also happens in group theory courses.

This problem occurs because the symbolic language uses the same symbol for many different operations, in this case the juxtaposition notation for multiplication. This phenomenon is called overloaded notation and is discussed in abstractmath.org here.

Conjecture

Noncommutative binary operations written using juxtaposition cause students trouble because going to noncommutative operations requires abandoning some overlearned reflexes in doing algebra.

Identity elements seem to behave the same in any binary operation, so there are no reflexes to unlearn. There are generic binary operations of various types as well. That’s why mathematicians are comfortable overloading juxtaposition. But to get to be a mathematician you have to unlearn some reflexes.

Negation

Sometimes you need to reword a math statement that contains symbolic expressions. This particularly causes trouble in connection with negation.

Ordinary English

The English language is notorious among language learners for making it complicated to negate a sentence. The negation of “I saw that movie” is “I did not see that movie”. (You have to put “d** not” (using the appropriate form of “do”) before the verb and then modify the verb appropriately.) You can’t just say “I not saw that movie” (as in Spanish) or “I saw not that movie” (as in German).

Conjecture

The method in English used to negate a sentence may cause problems with math students whose native language is not English. (But does it cause math problems with those students?)

Negating symbolic expressions

Examples • The negation of “n is even and a prime” is “n is either odd or it is not a prime”. The negation should not be written “n is not even and a prime” because that sentence is ambiguous. In the heat of doing a proof students may sometimes think the negation is “n is odd and n is not a prime,” essentially forgetting about DeMorgan. (He must roll over in his grave a lot.) • The negation of “x>0” is “x≤0”. It is not “x<0”. This is a very common mistake.

These examples are difficulties caused by not understanding the math. They are not directly caused by difficulties with the languages of math.

Negating expressions containing parenthetic assertions

Suppose you want to prove:

“If f:R→R is differentiable, then f is continuous”.

A good way to do this is by using the contrapositive. A mechanical way of writing the contrapositive is:

“If f is not continuous, then f:R→R is not differentiable.”

That is not good. The sentence needs to be massaged:

“If f:R→R is not continuous, then f is not differentiable.”

Even better would be to write the original sentence as:

“Suppose f:R→R. Then if f is differentiable, then f is continuous.”

This is discussed in detail in David Butler’s post Contrapositive grammar.

Conjecture

Students need to be taught to understand parenthetic assertions that occur in the symbolic language and to learn to extract a parenthetic assertion and write it as a standalone assertion ahead of the statement it occurs in.

Scope

The scope of a word or variable consists of the part of the text for which its current definition is in effect.

Examples

• “Suppose n is divisible by 4.” The scope is probably the current paragraph or perhaps the current proof. This means that the properties of n are constrained in that section of the text. • “In this book, all rings are unitary.” This will hold for the whole book.

There are many more examples in the abstractmath.org article Scope.

If you are a grasshopper (you like to dive into the middle of a book or paper to find out what it says), knowing the scope of a variable can be hard to determine. It is particularly difficult for commonly used words or symbols that have been defined differently from the usual usage. You may not suspect that this has happened since it might be define once early in the text. Some books on writing mathematics have urged writers to keep global definitions to a minimum. This is good advice.

Finding the scope is considerably easier when the text is online and you can search for the definition.

Conjecture

Knowing the scope of a word or variable can be difficult. It is particular hard when the word or variable has a large scope (chapter or whole book.)

Variables

Variables are often introduced in math writing and then used in the subsequent discussion. In a complicated discussion, several variables may be referred to that have different statuses, some of them introduced several pages before. There are many particular ways discussed below that can cause trouble for students. This post is restricted to trouble in connection with the languages of math. The concept of variable is difficult in itself, not just because of the way the math languages represent them, but that is not covered here.

Much of this part of the post is based on work of Susanna Epp, including three papers listed in the references. Her papers also include many references to other work in the math ed literature that have to do with understanding variables.

See also Variables in abstractmath.org and Variables in Wikipedia.

Types

Students blunder by forgetting the type of the variable they are dealing with. The example given previously of problems with matrix multiplication is occasioned by forgetting the type of a variable.

Conjecture

Students sometimes have problems because they forget the data type of the variables they are dealing with. This is primarily causes by overloaded notation.

Dependent and independent

If you define y=x2+1, then x is an independent variable and y is a dependent variable. But dependence and independence of variables are more general than that example suggests. In an epsilon-delta proof of the limit of a function (example below,) ε is independent and δ is dependent on ε, although not functionally dependent.

Conjecture

Distinguishing dependent and independent variables causes problems, particularly when the dependence is not clearly functional.

I recently ran across a discussion of this on the internet but failed to record where I saw it. Help!

Bound and free

This causes trouble with integration, among other things. It is discussed in abstractmath.org in Variables and Substitution. I expect to add some references to the math ed literature soon.

Instantiation

Some of these variables may be given by existential instantiation, in which case they are dependent on variables that define them. Others may be given by universal instantiation, in which case the variable is generic; it is independent of other variables, and you can’t impose arbitrary restrictions on it.

Existential instantiation

A theorem that an object exists under certain conditions allows you to name it and use it by that name in further arguments.

Example

Suppose m and n are integers. Then by definition, m divides n if there is an integer q such that n=qm. Then you can use “q” in further discussion, but q depends on m and n. You must not use it with any other meaning unless you start a new paragraph and redefine it.

So the following (start of a) “proof” blunders by ignoring this restriction:

Theorem: Prove that if an integer m divides both integers n and p, then m divides n+p.
“Proof”: Let n=qm and p=qm…”

Universal instantiation

It is a theorem that for any integer n, there is no integer strictly between nand n+1. So if you are given an arbitrary integer k, there is no integer strictly between k and k+1. There is no integer between 42 and 43. By itself, universal instantiation does not seem to cause problems, provided you pay attention to the types of your variables. (“There is no integer between πand π+1” is false.)

However, when you introduce variables using both universal and existential quantification, students can get confused.

Example

Consider the definition of limit:

Definition: limx→af(x)=L if and only if for every ϵ>0 there is a δ>0for which if |x−a|<δ then |f(x)−L|<ϵ.

A proof for a particular instance of this definition is given in detail in Rabbits out of a Hat. In this proof, you may not put constraints on ϵ except the given one that it is positive. On the other hand, you have to come up with a definition of δand prove that it works. The δ depends on what f, a and L are, but there are always infinitely many values of δ which fit the constraints, and you have to come up with only one. So in general, two people doing this proof will not get the same answer.

Reference

Susanna Epp’s paper Proof issues with existential quantification discusses the problems that students have with both existential and universal quantification with excellent examples. In particular, that paper gives examples of problems students have that are not hinted at here.

References

A nearly final version of The Handbook of Mathematical Discourse is available on the web with links, including all the citations. This version contains some broken links. I am unable to recompile it because TeX has evolved enough since 2003 that the source no longer compiles. The paperback version (without the citations) can be bought as a book here. (There are usually cheaper used versions on Amazon.)

Abstractmath.org is a website for beginning students in abstract mathematics. It includes most of the material in the Handbook, but not the citations. The Introduction gives you a clue as to what it is about.

Two languages

My take on the two languages of math are discussed in these articles: • Mathematical English • The Introduction to The Handbook of Mathematical Discourse. • The symbolic language of math.

The Language of Mathematics, by Mohan Ganesalingam, covers these two languages in more detail than any other book I know of. He says right away on page 18 that mathematical language consists of “textual sentences with symbolic material embedded like ‘islands’ in the text.” So for him, math language is one language.

I have envisioned two separate languages for math in abstractmath.org and in the Handbook, because in fact you can in principle translate any mathematical text into either English or logical notation (first order logic or type theory), although the result in either case would be impossible to understand for any sizeable text.

Topics in abstractmath.org

• Context-sensitive interpretation.
• “If” in definitions.
• Mathematical English.
• Parenthetic assertion.
• Scope
• Semantic contamination.
• Substitution.
• The symbolic language of math
• Variables.
• Zooming and Chunking.

Topics in the Handbook of mathematical discourse

These topics have a strong overlap with the topics with the same name in abstractmath.org. They are included here because the Handbook contains links to citations of the usage.

• Context-sensitive.
• “If” in definitions.
• Parenthetic assertion.
• Substitution.

Posts in Gyre&Gimble

Names / Naming mathematical objects / Rabbits out of a Hat. / Semantics of algebra I. / Syntactic and semantic thinkers Technical meanings clash with everyday meanings / Thinking without words. / Three kinds of mathematical thinkers Variations in meaning in math.

Other references

Contrapositive grammar, blog post by David Butler. / Proof issues with existential quantification, by Susanna Epp.

The role of logic in teaching proof, by Susanna Epp (2003). / The language of quantification in mathematics instruction, by Susanna Epp (1999). / The Language of Mathematics: A Linguistic and Philosophical Investigation by Mohan Ganesalingam, 2013. (Not available from the internet.) / On the communication of mathematical reasoning, by Atish Bagchi, and Charles Wells (1998a), PRIMUS, volume 8, pages 15–27. / Variables in Wikipedia.

Conjecture

Many beginning abstract math students have difficulty understanding a symbolic expression like the one above. Could this be cause by resistance to treating the expression as a structure to be studied?