Scripts/WeatherPlot
Notes
Harriett Neville and I were discussing the local weather. I said I thought we had sub-zero temperatures in Rochester NY fairly often. She was skeptical. I decided to find some data.
After some rummaging, I found the FEDERAL CLIMATE COMPLEX GLOBAL SURFACE SUMMARY OF DAY DATA site. It offers data sets from 1929 to yesterday for many locations around the world. The key links I found were these:
- A description of the site and data formats.
- ftp://ftp.ncdc.noaa.gov/pub/data/gsod/readme.txt
- A list of reporting location codes.
- ftp://ftp.ncdc.noaa.gov/pub/data/gsod/ish-history.csv
- The data is organized as one file per location per year:
- ftp://ftp.ncdc.noaa.gov/pub/data/gsod/1973/725290-14768-1973.op.gz
I downloaded the files I was interested in, from 1973-2009, to a directory and uncompressed them.
Here is a simple verb to extract the high and low temperatures and plot them. The temperatures are reported in Fahrenheit.
The main data processing is as follows:
1. Read all data files into a boxed list. 1. Cut the text files at LF. 1. Drop the 1st line, the title line, from each file. 1. Flatten the boxes to a list of boxed lines. 1. Remove duplicate blanks from each line. 1. Cut each line at blank. 1. Select the date, high temperature and low temperature elements. 1. Remove the '*' decorator from any data, if present. 1. Convert the all the data to numbers. 1. Unbox the data to a 3xN numeric matrix. 1. Remove any days with missing (9999.9) data. 1. Plot the data with a zero line and X labels for each year.
Finding the data, downloading it and plotting it in J took me about an hour. That's about the same length of time it took for me to create this wiki page.
Usage
Example usage:
roch73_09''
[{{#file: "plotweather.ijs"}} Download script: plotweather.ijs ]
NB. plot weather data require 'strings dir misc plot statfns' roch73_09=: 3 : 0 d=. 'C:\Users\me\Downloads\weather' f=. 1 dir d,'\725290*.op' data=.|:>>".&.>&.>(('*'~:])#])&.>&.>(2 17 18{])&.>cut &.>deb&.>;}.&.>LF cut &.>fread&.>f miss=.(9999.9~:1{data)*.9999.9~:2{data data=.miss#"1 data years=.0{"1 (10000 100 100 #: 0{data) mark=.0,}:>:bx -.years=(}.years),0 xmark=. ~.years pd 'reset' pd (}.data),0 pd 'xticpos ',":mark pd 'xlabel ',":xmark pd 'key high low zero' pd 'keypos lbi' pd 'title Rochester NY Daily High and Low Temperatures 1973-2009' pd 'isi 1280x 800x' pd 'save bmp ~temp/rochestertemp.bmp 1280x 800x' pd 'save png ~temp/rochestertemp.png' ) gn=: 3 : 0 ".@(('*'~:])#])y ) rmd=: 3 : 0 (*./"1 ]9999.9 ~: y)#y ) rochmin73_09=: 3 : 0 d=. 'C:\Users\me\Downloads\weather' f=. 1 dir d,'\725290*.op' datay=.>(<./) L: 0 rmd L: 0 (>@>) L: 2 gn L: 0 (2 17 18&{@cut@deb) L: 0 }. L: 1 LF cut L: 0 fread L: 0 f years=.0{"1 ]10000 100 100 #: 0{"1 datay mark=.0,}:>:bx -.years=(}.years),0 xmark=. ~.years xa=.i.#datay dats1=.(3 lsfit xa,:(1{"1 datay)) p. xa dats2=.(3 lsfit xa,:(2{"1 datay)) p. xa pd 'reset' pd (|:(1}."1 datay,.dats1,.dats2)),0 pd 'xticpos ',":mark pd 'xlabel ',":xmark pd 'key high low highfit lowfit zero ' pd 'keypos lbi' pd 'title Rochester NY Yearly Minimum High and Low Temperatures 1973-2009' pd 'isi 1280x 800x' pd 'save png ~temp/rochestermintemp.png' ) roch2005=: 3 : 0 d=. 'C:\Users\me\Downloads\weather' f=. 1 dir d,'\725290*-2005.op' data=.|:rmd L: 0 (>@>) L: 2 gn L: 0 (2 17 18{]) L: 1 cut L: 0 deb L: 0 ;}.L: 1 LF cut L: 0 fread L: 0 f xa=.i.}.$data dats1=.(5 lsfit xa,:(1{data)) p. xa dats2=.(5 lsfit xa,:(2{data)) p. xa years=.0{"1 ]10000 100 100 #: 0{ data mark=.0,}:>:bx -.years=(}.years),0 xmark=. ~.years pd 'reset' pd (1}. data,dats1,:dats2),0 pd 'xticpos ',":mark pd 'xlabel ',":xmark pd 'key high low highfit lowfit zero ' pd 'keypos lti' pd 'title Rochester NY Daily High and Low Temperatures 2005' pd 'isi 1280x 800x' pd 'save png ~temp/rochester2005temp.png' )
See Also
Footnotes
<<FootNote>>
Contributed by David Mitchell