NYCJUG/2012-12-11
open-source, Hilbert, Kindle, converting text to numeric, table, data input, maze generation, DHTMLX, grid, short-term memory, derive quadratic equation
Location:: ThomasNet
Meeting Agenda for NYCJUG 20121211
1. Beginner's regatta: an example of simple text manipulation - see "Open Source Hilbert for the Kindle.doc". An example of extracting a table of numbers from a text file: see "Example of Free-Form Text Wrangling.doc". 2. Show-and-tell: a new j-mode for emacs - see J-mode on Github. Generating mazes - see "one line BASIC maze generator.doc". 3. Advanced topics: using J on the web - see "JHS with the DHTMLX grid.doc". Memory and thought: see "Short-term memory ability may predict IQ.doc", "Study indicates how we maintain visual details in short-term memory.doc", "Greater working memory capacity benefits analytic but not creative memory.doc", "Four is magic number for memory.doc", and "Doing Math Unconsciously.doc". 4. Learning, teaching and promoting J, et al.: see "TeachingProgrammingToYoungsters.doc".
Beginner's regatta
We first looked at some code from John Baker for converting an open-source document into a form suitable for reading on a Kindle; he applies this to a File:Hilbert kindle.pdf to make it available in an easily-readable form on the Kindle.
Next, we examined in detail some code for converting a table of numbers from a text file into a TAB-delimited form suitable for loading into a database, as shown here.
Example of Free-Form Text Wrangling
There is a wealth of commodity-related data available in the "CRB Yearbook", of which I have the 2007 edition, on a data CD, albeit in text files. The following is an example of extracting some of this data from one of the files to put it in a format more suitable for loading into a database. We start by reading in the relevant text as LF-delimited items as shown here. After reading it in, based on our knowledge of the file, we remove all excess spaces so we can break the vector of vectors into table cells based on a space delimiter.
$rcraw=. <;._2 CR-.~fread 'c:/Data/Comdy/ReutersCRBIndex1956-2007.txt'
158
rcraw=. dsp&.>rcraw
>4{.rcraw
<<TOC3>> Monthly Reuters-CRB Index (CCI) (High, Low and Close 1967 = 100)
Year Jan. Feb. Mar. Apr. May June July Aug. Sept. Oct. Nov. Dec. Range
-------------------------------------------------------------------------...
$rct=. <;._1&>' ',&.>rcraw
158 15
Looking at what we have, we see that the initial column seems inordinately long because of a row of minus signs denoting a text-based separator line. We remove this specific line, then examine the start and end of the array to verify that it looks reasonable.
5 10{.rct
┌────────────────────────────────────────────────────────────────────────...
│<<TOC3>> ...
├────────────────────────────────────────────────────────────────────────...
│ ...
├────────────────────────────────────────────────────────────────────────...
│Year ...
├────────────────────────────────────────────────────────────────────────...
│------------------------------------------------------------------------...
├────────────────────────────────────────────────────────────────────────...
│1956 ...
└────────────────────────────────────────────────────────────────────────...
#&>5 10{.rct NB. Tally number of items in first 5 x 10 cells...
8 7 11 5 5 6 3 3 5 4
0 0 0 0 0 0 0 0 0 0
4 4 4 4 4 3 4 4 4 5
180 0 0 0 0 0 0 0 0 0
4 4 4 4 4 4 4 4 4 4
3{rct NB. This line gives us a display problem,
┌────────────────────────────────────────────────────────────────────────...
│------------------------------------------------------------------------...
└────────────────────────────────────────────────────────────────────────...
rct=. (<<<3){rct NB. so we'll remove it.
#&>5 10{.rct NB. Look at sizes of first few cells again.
8 7 11 5 5 6 3 3 5 4
0 0 0 0 0 0 0 0 0 0
4 4 4 4 4 3 4 4 4 5
4 4 4 4 4 4 4 4 4 4
3 4 4 4 4 4 4 4 4 6
5 10{.rct NB. Sizes are all small, so display some data at beginning.
┌────────┬───────┬───────────┬─────┬─────┬──────┬────┬────┬─────┬──────┐
│<<TOC3>>│Monthly│Reuters-CRB│Index│(CCI)│(High,│Low │and │Close│1967 │
├────────┼───────┼───────────┼─────┼─────┼──────┼────┼────┼─────┼──────┤
│ │ │ │ │ │ │ │ │ │ │
├────────┼───────┼───────────┼─────┼─────┼──────┼────┼────┼─────┼──────┤
│Year │Jan. │Feb. │Mar. │Apr. │May │June│July│Aug. │Sept. │
├────────┼───────┼───────────┼─────┼─────┼──────┼────┼────┼─────┼──────┤
│1956 │High │---- │---- │---- │---- │----│----│---- │---- │
├────────┼───────┼───────────┼─────┼─────┼──────┼────┼────┼─────┼──────┤
│Low │---- │---- │---- │---- │---- │----│----│---- │107.92│
└────────┴───────┴───────────┴─────┴─────┴──────┴────┴────┴─────┴──────┘
5 _10{.rct NB. Also look at some data at end.
┌──────┬────┬────┬─────┬──────┬──────┬──────┬──────┬──────┬──────┐
│(High,│Low │and │Close│1967 │= │100) │ │ │ │
├──────┼────┼────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┤
│ │ │ │ │ │ │ │ │ │ │
├──────┼────┼────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┤
│May │June│July│Aug. │Sept. │Oct. │Nov. │Dec. │Range │ │
├──────┼────┼────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┤
│---- │----│----│---- │---- │109.36│110.36│115.83│114.71│115.83│
├──────┼────┼────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┤
│---- │----│----│---- │107.92│107.15│109.15│110.95│107.15│ │
└──────┴────┴────┴─────┴──────┴──────┴──────┴──────┴──────┴──────┘
We replace the empty-cell indicators of ---- with a more J-like "infinity" (_), then further examine how the data lines up. We see that there are ragged ends every third row due to the formatting of the data: each year begins with the first column occupied by the year number but the following two rows of that year, with the "Low" and "Close" data rows, don't line up because of the differing number of items in a row.
rct=. rct rplc&.><'----';'_'
3}.5 _10{.rct
┌──────┬────┬────┬─────┬──────┬──────┬──────┬──────┬──────┬──────┐
│_ │_ │_ │_ │_ │109.36│110.36│115.83│114.71│115.83│
├──────┼────┼────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┤
│_ │_ │_ │_ │107.92│107.15│109.15│110.95│107.15│ │
└──────┴────┴────┴─────┴──────┴──────┴──────┴──────┴──────┴──────┘
_3 10{.rct
┌─────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┐
│Low │348.57│345.49│346.56│360.85│376.35│363.63│375.73│380.15│359.07│
├─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
│Close│363.30│353.27│361.91│379.53│379.80│385.63│391.49│390.95│370.10│
├─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
│2007 │High │396.93│ │ │ │ │ │ │ │
└─────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘
Using our knowledge of the mis-aligned row labels, we generate a boolean to preferentially shift the appropriate rows. Once we've verified that the result looks right, we apply it to our table.
(0{"1 rct) e. 'Low';'Close'
0 0 0 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0 1 ...
_4 10{.rct|."(0 1)~]-(0{"1 rct) e. 'Low';'Close'
┌────┬─────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┐
│2006│High │363.70│364.28│365.66│382.46│399.66│387.52│396.16│399.90│
├────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
│ │Low │348.57│345.49│346.56│360.85│376.35│363.63│375.73│380.15│
├────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
│ │Close│363.30│353.27│361.91│379.53│379.80│385.63│391.49│390.95│
├────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
│2007│High │396.93│ │ │ │ │ │ │ │
└────┴─────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘
3}.6 10{.rct|."(0 1)~]-(0{"1 rct) e. 'Low';'Close'
┌────────┬───────┬───────────┬─────┬─────┬──────┬────┬────┬─────┬─────┐
│1956 │High │_ │_ │_ │_ │_ │_ │_ │_ │
├────────┼───────┼───────────┼─────┼─────┼──────┼────┼────┼─────┼─────┤
│ │Low │_ │_ │_ │_ │_ │_ │_ │_ │
├────────┼───────┼───────────┼─────┼─────┼──────┼────┼────┼─────┼─────┤
│ │Close │_ │_ │_ │_ │_ │_ │_ │_ │
└────────┴───────┴───────────┴─────┴─────┴──────┴────┴────┴─────┴─────┘
Since this expression appears to fix the mis-alignment, apply it to the variable. This ability to test our solutions before applying them is a great benefit of the interactivity of J.
rct=. rct|."(0 1)~]-(0{"1 rct) e. 'Low';'Close'
rct=. (_1|.2{rct) 2}rct
Now look at filling in the implicit year values missing from these two trailing rows per year.
,0&".&>3}.0{"1 rct
1956 0 0 1957 0 0 1958 0 0 1959 0 0 1960 0 0 1961 0 0 1962 0 0 1963 0 0 ...
>./\,0&".&>3}.0{"1 rct
1956 1956 1956 1957 1957 1957 1958 1958 1958 1959 1959 1959 1960 1960 19...
yrs=. >./\,0&".&>3}.0{"1 rct
Ensure there are no surprises in the column denoting the row type.
3{.hlc=. 3}.1{"1 rct
┌────┬───┬─────┐
│High│Low│Close│
└────┴───┴─────┘
~.hlc
┌────┬───┬─────┐
│High│Low│Close│
└────┴───┴─────┘
Get prices as simple numeric matrix.
$pxs=. ".&>2}."1]_1}."1]3}.rct
154 12 1
$,"2 pxs
154 12
5 12{.,"2 pxs
_ _ _ _ _ _ _ _ 109.36 110.36 11...
_ _ _ _ _ _ _ _ 107.92 107.15 10...
_ _ _ _ _ _ _ _ 108.27 110.36 1...
114.09 112.4 110.86 110.54 108.82 107.58 109.4 108.25 107.11 105.23 1...
112.31 109.37 108.83 108.64 106.47 106.29 107.42 106.26 103.58 103.38 10...
_3 12{.,"2 pxs
348.57 345.49 346.56 360.85 376.35 363.63 375.73 380.15 359.07 361.19 38...
363.3 353.27 361.91 379.53 379.8 385.63 391.49 390.95 370.1 383.92 40...
396.93 0 0 0 0 0 0 0 0 0 ...
pxs=. ,"2 pxs
Replace these terminal zeros with the infinity we're using for missing values for uniformity's sake.
pxs=. (0=pxs)}pxs,:_
_3 12{.pxs
348.57 345.49 346.56 360.85 376.35 363.63 375.73 380.15 359.07 361.19 38...
363.3 353.27 361.91 379.53 379.8 385.63 391.49 390.95 370.1 383.92 40...
396.93 _ _ _ _ _ _ _ _ _ ...
Establish all the relevant end-of-month values we'll use for each data item. We cheat a little by using a pre-calculated file of valid end-of-month dates that accounts for non-business days (weekends).
$eoms=. <;._2 fread '\amisc\Clarifi\Data\EOMDates.txt'
756
3{.eoms
┌─────────┬─────────┬─────────┐
│1/31/1950│2/28/1950│3/31/1950│
└─────────┴─────────┴─────────┘
eoms i. <'1/31/1956'
72
3{.eoms}.~eoms i. <'1/31/2007'
┌─────────┬─────────┬─────────┐
│1/31/2007│2/28/2007│3/30/2007│
└─────────┴─────────┴─────────┘
684-72
612
moixs=. 72+i.613
({.,{:) moixs{eoms
┌─────────┬─────────┐
│1/31/1956│1/31/2007│
└─────────┴─────────┘
$eoms=. moixs{eoms
613
Now that we've got all the major data items neatened up, let's use the "Completion Backward Principle" to envision how we want the resulting array to end up.
eg=. 0 : 0 1/31/1956 High _ 1/31/1956 Low _ 1/31/1956 Close _ ... 1/31/1957 High 114.09 1/31/1957 Low 112.31 1/31/1957 Close 112.31 ... )
Since we now have all the pieces of this available, we can assemble them as needed to reach our target format but we won't give the details of that here.
Show-and-Tell
Zach went over code he's written in emacs lisp for a J major mode for Emacs. Major modes are the method by which emacs is configured into a more useful IDE for a specific language or task. j-mode taget="_blank" is hosted on Github, so look there for the most recent version of the code.
j-mode provides font locking ( which is emacs' version of syntax aware highlighting ). Direct REPL integration, and access to the J-VOC documentation.
We looked at a one-liner in BASIC for generating a random maze, then a slightly more sophisticated example in J. The J example is further developed to create a better-looking maze using sets of line-drawing characters that match up well with each other. Examples are shown of testing the connectivity of the maze by using flood-fill in a paint program.
Advanced Topics
We looked at using dynamic HTML under the J HTTP server (JHS).
JHS with the DHTMLX Grid
December 3, 2012 John Baker
Grids are the most important GUI user object. It’s hard to think of a user-friendly data munching application that doesn’t have a grid beating at its heart. Consequently, any serious GUI interface contender must support grids. My previous post showed how to use MathJax with JHS. MathJax is an impressive and important JavaScript library; it clearly demonstrates the potential of CHJ1 GUI interfaces but let’s face it, mathematical typesetting will not win many consulting contracts. Grids won’t seal the deal either but their absence is a huge “next” signal. To support serious business and technical applications JHS needs grids.
Fortunately, the JavaScript world is grid saturated. The difficulty is not finding a grid but choosing among dozens of candidates. For this demo I Googled around and found DHTMLX. According to this probably biased article the DHTMLX grid performs well on large inputs and, more importantly, there is an open source version.
You have to start somewhere so I opted to use DHTMLX to build a simple CSV file editor. The CSV files I am going to edit are TAB delimited text files. Each file has a fixed number of columns with column names in the first row. Here is an example TAB delimited file. The idea is to load the file data into the grid. Tweak a few rows and save the result. By increasing the size of the CSV file we can gauge the performance of the grid. Let’s get started.
Using the DHTMLX grid requires some preparation.
1. Create a local directory and edit J’s ~config/folders.cfg to reference the directory with the name GridDemo. jpath '~GridDemo' should return the full directory path.
2. Download the files in the GridDemo folder and copy them to ~GridDemo.
3. Download the Standard Edition (Version 3.5) of DHTMLX. The distribution file dhtmlxGrid.zip contains the grid source and supporting files.
4. Extract the /dhtmlxGrid/codebase/ directory from dhtmlxGrid.zip and copy the entire directory tree to ~GridDemo.
5. Also extract /dhtmlxGrid/samples/common from dhtmlxGrid.zip and copy the directory to ~GridDemo. When you’re finished the top-level of ~GridDemo will look like the following where names without extensions are directories.
calendar dhtmlxgrid.js GridDemo.ijs t100rows.txt
common dhtmlxgrid_skins.css imgs t5000rows.txt
dhtmlxcommon.js excells jodoval.png
dhtmlxgridcell.js ext skins
dhtmlxgrid.css favicon.ico t1000rows.txt
The main J script is ~GridDemo\GridDemo.ijs. Start JHS and load this file.
load '~GridDemo/GridDemo.ijs'
Then browse to this site.
. http://127.0.0.1:65001/GridDemo
If all goes well you will see the following GridDemo page after pressing the Edit Grid button.
To load and edit files enter their fully qualified names in the Input and Output boxes and press Edit Grid. To edit a cell double-click it. To save changes press Save Grid.2 There are more sophisticated ways to pick files on JavaScript pages. It’s easy to pop up standard host OS file dialogs but it’s not particularly easy to determine host directory paths. This post outlines the demons web programmers must slay to select host files. JHS circumvents these difficulties by asking the J server, which is a typically a local console process, to do the dirty work. JavaScript’s access to local files is limited for security reasons but J has no such restrictions. Use the force Luke!
Three test files t100rows.txt, t1000rows.txt, and t5000rows.txt are included with the demo. On my test machines load times vary from fractions of a second for the smaller files to nine seconds for the largest. This is competitive with the basic C# grid control and fast enough for serious work.
In subsequent posts I will explore JavaScript/JHS graphics options and start the process of integrating, grids, graphs and MathJax with JHS.
1. CSS, HTML and JavaScript.↩
2. The freebie version of DHTMLX does not support grid serialization. Here is how to roll your own.
Memory and Cognition
We looked at several articles on cognitive abilities and their relation to short-term memory as well as newly-circumscribed limits of short-term memory and the different use of short-term memory for analytic versus creative thinking. We finished off this section on cognition with an article on the counter-intuitive notion of unconscious aspects of thought.
These articles are as follows.
Short-term memory ability may predict IQ
[From http://phys.org/news103472118.html on July 12, 2007]
U.S. psychologists have found people with high IQs might be able to remember more than the four objects an average person can store in short-term memory.
That ability, said University of Oregon psychology professors Edward Awh and Edward Vogel, varies from person to person, but an individual's capacity of short-term memory is a strong predictor of their IQ and scholastic achievement.
One hypothesis psychologists have considered is that memory capacity might be influenced by the complexity of items being stored.
The researchers discovered that even when very complex objects had to be remembered, people were able to hold four items in active memory. However, Awh said, the clarity of those items was not perfect, and some people had much clearer memories than others.
"Knowing the number of things a person can remember tells you nothing about how clear a person's memory may be," Awh said. "So even though people with high IQs can think about more things at once, there are not guarantees about how good those memories might be." The research, which included recent graduate Brian Barton, appears in the July issue of the journal Psychological Science.
Copyright 2007 by United Press International
Study indicates how we maintain visual details in short-term memory
[From http://phys.org/news154349817.html on February 20, 2009 ]
Working memory (also known as short term memory) is our ability to keep a small amount of information active in our mind. This is useful for information we need to know on-the-fly, such as a phone number or the few items we need to pick up from the grocery store. We hang on to the information for a brief period of time, just long enough to make a phone call or get through the checkout line, and then we forget it forever.
We receive much of our information through our visual system, but it was unknown how much of this visual information is actively involved in short term memory. Psychologists John T. Serences from the University of California, San Diego, along with Edward F. Ester, Edward K. Vogel and Edward Awh from the University of Oregon wanted to examine which neural systems enable the maintenance of these visual details in short term memory.
While undergoing functional magnetic resonance imaging (fMRI) scans, volunteers were shown an image for one second and were instructed to remember either the color or the orientation of the image. Following the image, volunteers saw a blank screen for 10 seconds, then were shown another image and had to indicate if it was the identical color or orientation (depending on which they were told to remember) as the first image. The researchers analyzed brain activity during the 10 second delay (when short term memory is active) in the primary visual cortex, the main region of the brain which handles visual information.
The results, described in Psychological Science, a journal of the Association for Psychological Science, revealed that during the 10 second delay, there was very specific activation in the visual cortex, specifically in areas normally involved in processing color and orientation. Moreover, close examination of the patterns of activity in the visual cortex allowed the authors to "decode" the specific color or orientation value that volunteers were holding in mind. This suggests that during short term memory the visual area of the brain is actively "thinking" about a specific feature of an object (e.g. color or orientation), to ensure that the information will be maintained and not forgotten.
The authors note that these findings "suggest that observers have top-down control over which features are stored" because the activity in the visual cortex represented only the voluntarily stored aspects of the image. In this way, we can ensure that only the most relevant details of the world around us are maintained in this online mental workspace.
Source: Association for Psychological Science
Greater working memory capacity benefits analytic, but not creative, problem-solving
[From http://medicalxpress.com/news/2012-08-greater-memory-capacity-benefits-analytic.html on August 7, 2012; originally in Psychology & Psychiatry (Medical Xpress)]
Psychological scientists have long known that the amount of information we can actively hold in mind at any given time – known as working memory – is limited. Our working memory capacity reflects our ability to focus and control attention and strongly influences our ability to solve problems.
In a new article in the August issue of Current Directions in Psychological Science, a journal of the Association for Psychological Science, Jennifer Wiley and Andrew Jarosz of the University of Illinois at Chicago explore the role of working memory capacity in both mathematical and creative problem solving.
Converging evidence from many psychological science studies suggests that high working memory capacity is associated with better performance at mathematical problem-solving. In fact, decreased working memory capacity may be one reason why math anxiety leads to poor math performance. Overall, working memory capacity seems to help analytical problem-solvers focus their attention and resist distraction.
However, these very features of working memory capacity seem to impair creative problem-solving. With creative problems, reaching a solution may require an original approach or a novel combination of diverse pieces of information. As a result, too much focus may actually impair creative problem solving.
The authors note that, in the real world, problems are not always distinctly divided into analytic and creative types – successful problem solving depends on the needs of a given situation. Journal reference: Current Directions in Psychological Science search and more info website
Provided by Association for Psychological Science
Four is the 'magic' number for our mind coping with information
[From on November 28, 2012; article in Psychology & Psychiatry ]
Four is the “magic” number Enlarge (Medical Xpress)—According to psychological lore, when it comes to items of information the mind can cope with before confusion sets in, the "magic" number is seven. But a new analysis by a leading Australian psychiatrist challenges this long-held view, suggesting the number might actually be four.
In 1956, American psychologist George Miller published a paper in the influential journal Psychological Review arguing the mind could cope with a maximum of only seven chunks of information. The paper, "The Magical Number Seven, Plus or Minus Two. Some Limits on Our Capacity for Processing Information", has since become one of the most highly cited psychology articles and has been judged by the Psychological Review as its most influential paper of all time.
But UNSW professor of psychiatry Gordon Parker says a re-analysis of the experiments used by Miller shows he missed the correct number by a wide mark. Writing in the journal Acta Psychiatrica Scandinavica, Scientia Professor Parker says a closer look at the evidence shows the human mind copes with a maximum of four 'chunks' of information, not seven.
"So to remember a seven numeral phone number, say 6458937, we need to break it into four chunks: 64. 58. 93. 7. Basically four is the limit to our perception.
"That's a big difference for a paper that is one of the most highly referenced psychology articles ever – nearly a 100 percent discrepancy," he suggests.
Professor Parker says the success of the original paper lies "more in its multilayered title and Miller's evocative use of the word 'magic'," than in the science. Professor Parker says 50 years after Miller there is still uncertainty about the nature of the brain's storage capacity limits: "There may be no limit in storage capacity per se but only a limit to the duration in which items can remain active in short-term memory".
"Regardless, the consensus now is that humans can best store only four chunks in short-term memory tasks," he says.
The full discussion paper includes many exemplars of the magic of 'four'.
Provided by University of New South Wales search and more info website
Not What You Consciously Thought: How We Can Do Math Problems and Read Phrases Nonconsciously
[From http://www.sciencedaily.com/releases/2012/11/121114083930.htm: ScienceDaily (Nov. 14, 2012)]
Can we actually read words and phrases and solve multi-step mathematical problems without our having consciously been aware of them? A team in the Psychology Department at the Hebrew University has conducted a series of experiments that give a positive answer: people can read and do math nonconsciously.
The results constitute a challenge to existing theories of unconscious processes, that maintain that reading and solving math problems -- two prime examples of complex, rule-based operations -- require consciousness. The conclusions of the Hebrew University team were published this week in the Proceedings of the National Academy of Sciences. The research team, headed by Dr. Ran Hassin, included graduate students Asael Sklar, Ariel Goldstein, Nir Levy and Roi Mandel, as well as Dr. Anat Maril.
To present sentences and equations unconsciously, the researchers used a cutting-edge technique called Continuous Flash Suppression (CFS). In CFS, one eye is exposed to a series of rapidly changing images, while the other is simultaneously exposed to a constant image. The rapid changes in the one eye dominate consciousness, so that the image presented to the other eye is not experienced consciously. Using this technique, more than 270 students at the Hebrew University were exposed to sentences and arithmetic problems.
In one set of experiments using this technique, participants were asked to pronounce numbers that appeared on a computer's screen. These numbers were preceded with unconscious arithmetic equations. The results of the experiments showed that participants could more quickly pronounce the conscious number if it had been the result of the unconscious equation. For example, when 9-5-1 was shown nonconsciously, the participants were faster in pronouncing 3 than 4, even though they did not consciously see the equation.
In another set of experiments reported in the PNAS paper, participants were nonconsciously exposed to a number of short verbal expressions that remained on screen until participants could say that they saw them. (In the meantime, the other eye was exposed to the rapidly flashing images). The results showed that negative verbal expressions (e.g., human trafficking) or unusual phrases (e.g., the bench ate a zebra) became conscious to the viewers before more positive expressions (e.g., ironed shirt and more usual phrases (e.g., the lion ate a zebra), indicating a definite "pickup" by the unconscious of something negative and out of the ordinary.
"These results show that the humans can perform complex, rule-based operations unconsciously, contrary to existing models of consciousness and the unconscious," say the researchers.
"Therefore," said Dr. Hassin, "current theories of the unconscious processes and human consciousness need to be revised. These revisions would bring us closer to solving one of the biggest scientific mysteries of the 21st century: What are the functions of human consciousness."
Learning, teaching, and problem-solving
We finished off with some excerpts from the J Forum on teaching programming to youngsters, in particular, its use in elaborating mathematical concepts. One correspondent - Johann Hibschman - makes the case that J is not as good as "traditional" mathematical notation for doing proofs with a pencil on paper. He gives an example of deriving a proof of the quadratic equation. However, another correspondent - Raul Miller - counters with an example of doing this in J, using specific data to demonstrate the plausibility of his derivation.
Suggestions for Teaching Students Programming and Math
There was some discussion on the J-forums about teaching programming and mathematical concepts to (young) students.
from: Edward Mokurai Cherlin <mokurai@sugarlabs.org> to: Programming forum <programming@jsoftware.com> date: Mon, Dec 10, 2012 at 1:10 PM subject: Re: [Jprogramming] J v Python
If we are going to argue J v Python, we need to at least look at NumPy (based in part on APL) and SciPy. If anybody is interested, I can start those topics.
I also have another set of issues, regarding choice of languages in working with Sugar education software. I will give the background here, and again if anybody is interested, I can provide much more information. Our first question is how to present programming at the elementary school level. The options in Sugar are Turtle Art, with paths into Logo and Python, and Etoys, with a path into Smalltalk. Both are excellent for their purposes, and can easily be introduced in third grade. XOs also have Perl (not recommended for children), FORTH (which we can discuss), and GeoGebra built in. Most Sugar software is written in Python, and we have a textbook on how to do it.
http://en.flossmanuals.net/make-your-own-sugar-activities/
Turtle Art provides a tile set implementing stack-based programming which can be used as a path into FORTH. I am also working on math textbooks in J, starting with first-grade arithmetic, and would like to get J packaged for Linux generally and for Sugar in particular. I hold with Marvin Minsky:
. You don't understand anything until you learn it more than one way.
. (In Rebecca Herold, Managing an Information Security and Privacy Awareness and Training Program (2005), 101.)
In order to understand programming, one should have a solid grasp of options, including both the limitations and possibilities defined by Computer Science. In particular, one should understand the concept of Turing Completeness, including Church's Thesis on the wide range of methods for building Turing complete languages. I recommend at a minimum languages on the LISP model, including Logo for children; on the APL model, including J, used starting with first-grade arithmetic; tile-based programming in tree structures, as in Turtle Art and Etoys; Turing machines themselves (I built one in Turtle Art, and Alan Perlis created a wonderful set of examples of Turing Machine extensions in his Computer Science textbook using APL); Object-Oriented Programming, as in Smalltalk and Etoys; and math languages such as GeoGebra. I have also recommended Python for some purposes.
Another essential concept is called Syntactic Sugar. LISPers in particular know that the syntax of a language is not the language. This is in large part because LISP separates the syntax analyzer from the rest of the language and allows users to replace it, and also because writing a syntax analyzer in LISP is so simple and easy.
This goes with the concept of application-specific languages, which should be easy to create. This is well understood in the LISP and FORTH communities, but not very well elsewhere.
For practical programming, the criteria are quite different. Many organizations prefer compatibility with past practice over any considerations of efficiency and quality. I don't have the resources to argue with them. My father George tried to offer the Society of Actuaries an introductory actuarial textbook in APL, but they insisted on FORTRAN and wouldn't even consider the possibility of providing both.
The correct term in English for a person who speaks several languages is "polyglot".
The correct term in English for a person who speaks only one language is "American".
This is equally true for natural languages and computer programming languages.
Perspectives on Using J for Math
from: Johann Hibschman <jhibschman@gmail.com> to: chat@jsoftware.com date: Tue, Dec 11, 2012 at 9:23 AM subject: Re: [Jchat] Was: [Jprogramming] J v Python
I can't pretend to either be an educator, but for my own use, I've found that J makes for a great computational notation and a great notation for writing about mathematics on computers, it doesn't work well for me as a pen-and-paper notation for actually doing math.
If I'm writing something up, I'm naturally contrained to a single linear line of text, and I don't have to worry about how long it takes me to handwrite symbols. If I'm doing work on paper, on the other hand, I can work in two dimensions with fractions, and it matters that I find "*" to be a relatively slow symbol to draw.
Similarly, "traditional" notation naturally maps to simple algebraic manipulation in a way that J doesn't. If I have
e = a + 2*b + c + d (traditional)
I can write
e - 2*b = a + c + d
while the same tokens in J,
e = a+2*b+c+d
can only naturally be "split" rather than reordered.
(e-2*b+c+d) = a
Even, then, the "split" only works for an initial element, since it's not true that
e = a-b-c-d
is the same as
(e+c-d) = a-b
I'm not that satisfied with these examples. The point I'm trying to express is that since J expressions are strictly cumulative, most manipulations require an awareness of the entire expression, unlike algebraic notation, whch builds on the associativity and commutativity of addition.
As an example, I found it much easier to work out a basic derivation of the quadratic formula in traditional notation rather than in J notation. In J, it would be hopeless if I didn't use the polynomial verb.
0= (c,b,a) p. x
0= ((c%a),(b%a),1) p. x
(*:-:b%a) = (((c%a)+*:-:b%a),(b%a),1) p. x
((-c%a)+*:-:b%a) = *:((-:b%a),1) p. x
(*:-:%a)*((*:b)-4*a*c) = *:((-:b%a),1) p. x
((],-) (%:(*:b)-4*a*c))%2*a) = ((-:b%a),1) p. x
x = (2*a)%~b (+,-) %:(*:b)-4*a*c
vs.
0 = c + bx + ax^2
0 = c/a + (b/a) x + x^2
b^2/4a^2 = c/a + b^2/4a^2 + (b/a) x + x^2
b^2/4a^2 - c/a = (b/2a + x)^2
(b^2 - 4ac)/4a^2 = (b/2a + x)^2
+- sqrt(b^2 - 4ac)/2a = b/2a + x
x = (-b +- sqrt(b^2 - 4ac))/2a
Writing out the J on paper took forever, with lots of fiddly little colons and adding extra parentheses to make the expressions work. I often had to "go back" and insert parenthesis around expressions, which isn't a problem on a computers, but is a nuisance on paper. Lest you think this is a trivial example, it extends (I think) to bigger problems. If I try to imagine working my way through a typical (say) undergraduate physics E&M problem set using J notation, I think I'd grind to a halt.
J is optimized for feeding into a computer, not for writing on a whiteboard or on paper. I know APL was used as a non-computer notation, but I think the "funny symbols" actually help there, and even then it seems like such a linear way to work.
Now, I have no easy way to extrapolate to greater educational policy, but I can say that I'm glad to have both notations at my disposal now: traditional for working things out symbolically and J for implementing and experimenting numerically.
Regards,
Johann
from: [[User:Raul Miller|Raul Miller]] <rauldmiller@gmail.com> via srs.acm.org date: Tue, Dec 11, 2012 at 10:15 AM
Of course exceptions are all that we need to prove that a universal rule is not universal.
Also, a lack of volume (or focus) for a discussion does not mean that a subject of the discussion is not an issue.
But I think we are stumbling over the lack of definition for "standard mathematical notation". If it is defined as "existing practice" then (ironically) J is a part of "standard mathematical notation". If it is instead defined as "popular practice" then it must exclude most of mathematics. If there is some rigorous formal definition, no one here has identified it.
Based on my personal experience (as a student, and as a tutor) when a person is having a problem with mathematical notation they will typically think of it as a problem they are having with "math" as a subject -- if they were capable of isolating their problems they would typically be able to get past them and on to other issues.
That said, the lack of written materials using J to treat specific subjects is a huge problem -- for example, the amount of good materials cannot exceed the total amount of material. And a lack of good materials yields a problem with familiarity (among other things).
Anyways, I see more interesting posts in this thread and I want to move on to them.
from: [[User:Raul Miller|Raul Miller]] <rauldmiller@gmail.com> date: Tue, Dec 11, 2012 at 11:30 AM
First off, I agree with you that this is difficult.
When I was first presented with the derivation of the quadratic equation, I could not have done the derivation myself -- it's not a completely trivial task. And, doing it with unfamiliar notation does not make it easier. That said, when I work with J, I prefer working with concrete examples. This corresponds to the "check your work" admonition that my teachers had repeated to my classes over and over.
So, using trial and error, I picked a few examples:
C=: 2 2 6 6 4 B=: 3 3 9 9 9 A=: 1 1 3 3 5 X=: _1 _2 _1 _2 _1
And, these do work with the p. notation, producing zeros:
(C,.B,.A) p. X 0 0 0 0 0
But I think for this kind of work I would be more comfortable using addition and multiplication:
C+(X*B)+(X*X*A) 0 0 0 0 0
We know, by our choice of problem, that A is not zero, so:
(C%A)+(X*B%A)+(X*X*A%A) 0 0 0 0 0 (C%A)+(X*B%A)+(X*X) 0 0 0 0 0
At this point we want to break out the sub-expression involving X from the sub-expression which does not contain X:
(X*B%A)+X*X _2 _2 _2 _2 _0.8
-C%A _2 _2 _2 _2 _0.8
And we can add (B%2*A)*(B%2*A) to both expressions, and the consequent results will still be equal:
(X*B%A)+(X*X)+(B%2*A)*(B%2*A)
0.25 0.25 0.25 0.25 0.01
(-C%A)+(B%2*A)*(B%2*A)
0.25 0.25 0.25 0.25 0.01
Since we are using B%2*A a lot, let's introduce a new symbol for it (and of course the new symbol substitutes into the above equation, retaining the previous result):
D=:B%2*A (X^2)+(2*X*D)+D^2 0.25 0.25 0.25 0.25 0.01 (-C%A)+D^2 0.25 0.25 0.25 0.25 0.01
The first equation involving D can be simplified (presumably we are already familiar with the pattern from Pascal's triangle):
(X+D)^2 0.25 0.25 0.25 0.25 0.01
Now we can take the square roots:
%:(X+D)^2 0.5 0.5 0.5 0.5 0.1 X+D 0.5 _0.5 0.5 _0.5 _0.1 %:(-C%A)+D^2 0.5 0.5 0.5 0.5 0.1
Whoa... what happened here? When I use %: I only get positive roots, but I can see from my example that some results are negative. This gets into an issue which I feel is sometimes glossed over when using the concept of "free variables" -- X can take on a variety of values, so the concept of "equality" is ambiguous.
But we can find both the positive and negative roots in J by using (,:-)%:
(,:-)%:(-C%A)+D^2 0.5 0.5 0.5 0.5 0.1 _0.5 _0.5 _0.5 _0.5 _0.1
And the structure of this answer is I think an important issue -- there are going to be two square roots, and what we are looking for is the presence of our X values in one of the alternatives.
Anyways, living with this limitation, we subtract D from both of our equations:
X _1 _2 _1 _2 _1 (-D)+"1 (,:-)%:(-C%A)+D^2 _1 _1 _1 _1 _0.8 _2 _2 _2 _2 _1
And we can see that our X values are present in the computed result. But we should substitute back using our definition of D so that we are working with the original terms:
(-(B%2*A))+"1 (,:-)%:(-C%A)+(B%2*A)^2 _1 _1 _1 _1 _0.8 _2 _2 _2 _2 _1
And, at this point, we might want to simplify that equation -- we can put 2*A into the denominator:
((-B)+"1 (,:-)%:(-2*2*A*C)+B^2)(%"1)2*A _1 _1 _1 _1 _0.8 _2 _2 _2 _2 _1
Note that we need the "1 because of my (,:-) notation and my using a list of values for our examples, but further exploration of that topic is beyond the scope of usual presentations of the quadratic equation.
Anyways, some notes here:
1) using a computer to check my work helped me see when I am going off track. J is designed for this, and trying to use J without using the computer for this purpose is going to take more effort. I have left out the trials I made which were erroneous (either syntax errors or faulty thinking), and this is usual practice in mathematics (but discovering mistakes is I think an important part of learning to use math).
2) there are no books or materials that I know of which present the quadratic equation in this fashion. Deriving it by hand is going to involve more work than looking it up.
3) the difficulties presented here are just a small slice of the difficulties facing both the student and the educator when dealing with any variation on how concepts are presented.
4) the concept of equality in the context of free variables is, I think, a topic which does not get enough attention in many presentations. As a result, many students will invent bogus concept of "equality" which will cause difficulties for them later.
5) This quadratic equation derivation is just one example of the use of J to present a subject which involves mathematics. But a typical student will have a background involving many years of work involving whatever notations and a variety of mathematical concepts. You can't reasonably expect to equal that level of skill by working just one or two examples using J. FYI,
Materials
- File:Open Source Hilbert for the Kindle.pdf
- File:Hilbert kindle.pdf
- File:JHS with the DHTMLX grid.pdf
- File:Example of Free-Form Text Wrangling.pdf
- File:J-modeEL.pdf
- File:J-mode.txt
- File:One line BASIC maze generator.pdf
Suggestions for a "J" Image
Just in case we ever have an O'Reilly book for J, here's some ideas of animals (or other things) that could grace the cover. I'm fond of the Blue Jay but the Jaguar would also be good as long as no one confuses it with a Mac OS.
-- Devon McCormick <<DateTime(2013-01-14T03:08:28-0200)>>