sgram - Spectrogram displays for the WWW

A few initial caveats:  sgram now appears to be stable and I will soon be removing 
a number of flags and messages which were part of the debugging process.  

**** Jim Luetgert 18 Oct 1999 ***

sgram is designed to rapidly provide GIF images of spectrograms for display on the WWW.  
sgram periodically retrieves current data for a selected list of stations and updates 
GIF images which are transferred to webserver(s).  The required waveforms are obtained 
via periodic requests to waveserver(s).  Spectra are computed, converted to a color scale 
and plotted directly to GIF format image files.  I use Perl Scripts on the webserver(s) 
to dress up the data displays.  

How to set it up - 
The first task is to set up a directory as a holding tank for the GIF image 
and .html files to reside while they are waiting to be dispatched to the webserver(s).  
I have found it useful to set up a directory and sub-directory of form   .../gifs/sgram/.  
This provides the opportunity for other subdirectories of /gifs/ for other 
image creating modules.
If you are going to display a logo on your helicorder gifs, put a copy of the logo.gif
in this directory.  If you are using a Perl Script on the webserver(s) to generate
your html display, you need do nothing further.  

Examples of a Perl Script are included.

Make sure that your earthworm.d file contains a module id for sgram.  It should be MOD_SGRAM.
Insert an entry for sgram in your startstop.d file.
Now comes the fun part; setting up the configuration files.  

sgram.d -

****************************************************************************************
#
# This is the spectrogram parameter file. This module gets data gulps
# from the waveserver(s), and computes spectrograms


#  Basic Earthworm setup:
#
LogSwitch    1              # 0 to completely turn off disk log file
MyModuleId   MOD_SGRAM      # module id for this instance of report 
RingName     HYPO_RING      # ring to get input from
HeartBeatInt 15             # seconds between heartbeats

>>>>
>>>> The first four config variables are standard earthworm.
>>>>
>>>> We need to talk to the transport ring so that StartStop
>>>> can terminate this process.
>>>>

wsTimeout 12 # time limit (secs) for any one interaction with a wave server.

>>>>
>>>> We need to know a reasonable time limit for getting
>>>> a response from any wave server.
>>>>

# List of wave servers (ip port comment) to contact to retrieve trace data.
 WaveServer 130.118.49.67  16022      "wsv2 - ad1"
 WaveServer 130.118.49.67  16023      "wsv2 - ad2"
 WaveServer 130.118.49.67  16024      "wsv2 - ad3" 
 WaveServer 130.118.49.210 16022      "wsv1 - ad1"
 WaveServer 130.118.49.210 16023      "wsv1 - ad2"
#WaveServer 130.118.49.210 16024      "wsv1 - ad3" 
 WaveServer 130.118.49.210 16025      "wsv1 - 1 Nano"
 WaveServer 130.118.49.210 16026      "wsv1 - 1 DST"

>>>> 
>>>> This is the list of waveservers to try.  If the requested
>>>> data for a given SCN is on more than one waveserver, the
>>>> first listed will be preferentially used.  If a request
>>>> fails (gap in the data, broken connection) the alternate
>>>> server(s) will be tried.  The comment field is optional;
>>>> it makes error messages a bit more helpful.  The maximum
>>>> number of waveservers is specified by MAX_WAVESERVERS in
>>>> sgram.h.
>>>> 

# Filename prefix on target computer.  This is useful for identifying
# files for automated deletion via crontab.
Prefix nc

>>>> 
>>>> For each SCN/date, a (sometimes large) GIF file and an (optional) .html
>>>> wrapper are created.  Unless there is some mechanism in place
>>>> to blow away out-of-date files, the webserver will eventually
>>>> drown in old files.  To fix this problem, these two types of
>>>> file are assigned a prefix to readily identify them as volatile
>>>> and a crontab process is used to blow them away. e.g.
>>>> 15 1 * * 0-6 find /home/ehzweb/waveforms/sgram -name nc\* -mtime +7 -exec rm {} \;
>>>> will remove files starting with nc which are older than 7 days.
>>>> 

# Directory in which to store the temporary .gif, .link and .html files.
GifDir   /home/luetgert/gifs/sgram/  

>>>> 
>>>> This is the directory on your local machine which is used to
>>>> store the working versions of the GIF and .html files.  
>>>> For each SCN, there may also be a SCN.hist file which keeps 
>>>> track of which dates are available for display.  
>>>> This is used to construct the optional index.html file.  
>>>> If SaveDrifts (see below under optional commands) is set, 
>>>> a file SCN.drft is created for each SCN.  Each time a new 
>>>> spectrogram hour is started, the min and max spectral
>>>> values for the previous hour of data are recorded.
>>>> 

# List of target directories on this computer to place output.
#                      Directory
 LocalTarget   /home/picker/sendfiler/sgram1/

>>>> 
>>>> Here we specify the location of our web pages on the local machine.
>>>> This may be a directory for a local webserver to access or 
>>>> it may be a directory for sendfile to use as a source for 
>>>> sending files to a remote webserver.
>>>> When setting up a directory structure on your webserver,
>>>> I would recommend a directory dedicated to GIF displays.
>>>> with a sub-directory dedicated to spectrogram displays.
>>>> The maximum number of local targets is specified by 
>>>> MAX_TARGETS in sgram.h.
>>>> 

# Plot Parameters - sorry it's so complex, but that's the price of versatility
        # The following is designed such that each SCN creates it's own
        # spectrogram display; one per HoursPerPlot hours of data.
# S                  Site
# C                  Component
# N                  Network

>>>> 
>>>> These are the SCN identifiers as they appear in the 
>>>> waveserver menu.
>>>> 

# 04 HoursPerPlot    Total number of hours per gif image

>>>> 
>>>> I have been using 24 hours per image for overviews and  
>>>> 2 hours per image for detailed spectrograms.
>>>> 

# 05 Plot Previous   On startup, retrieve and plot n previous hours from tank.

>>>> 
>>>> This lets you fill in those little gaps left by system crashes,
>>>> power outages, silly mistakes, etc.  I usually leave this
>>>> set to 1 so that gaps get filled automatically if something
>>>> bad happens and statmgr has to restart sgram.  This will only
>>>> let you fill in past data from the current plot (i.e. <=HoursPerPlot).
>>>> If you need past plots use sgram2, the offline 
>>>> spectrogram gif generator.  [when&if I finish it]
>>>> 

# 06 Local Time Diff UTC - Local.  e.g. -7 -> PDT; -8 -> PST
# 07 Local Time Zone Three character time zone name.
# 08 Show UTC        UTC will be shown on one of the time axes.
# 09 Use Local       The pages will start relative to local time rather than UTC time.

>>>> 
>>>> These variables govern how the time axes are displayed on 
>>>> your plot and whether the plot starts at 00:00 UTC or local.
>>>> 

# 10 XSize           Overall x-size of plot in inches

>>>> 
>>>> This lets you tailor the width of your display.
>>>> Setting this > 100 will imply pixels
>>>> 

# 11 Pixels/Line     Vertical pixels per line of trace displayed

>>>>
>>>> I use 1 for the 24 hour displays and 4 or 5 for the 2 hour displays
>>>>

# 12 Minutes/Line    Number of minutes per line of trace displayed

>>>>
>>>> I use 1 minute/line
>>>>
>>>> The product of Minute/Line, Pixels/Line, and HoursPerPlot determines
>>>> the overall y-size of the plot in pixels.
>>>>

# 13 Seconds/Gulp    Number of seconds per fft

>>>>
>>>> I use 60 second ffts
>>>>

# 14 Freq Max        Maximum frequency

# 15 Freq Mute       Hz to mute at low end

>>>>
>>>> Some stations produce a lot of very low frequency energy which
>>>> is not of interest.  This variable applies a mute at the low end.
>>>>

# 16 nbw             Display scaling. 1:linear, 2:log [neg for greyscale]


# 17 amax            Clipping amplitude applied to spectrum [0 for none]

>>>>
>>>> The spectral amplitude is expressed as spectral colors ranging from
>>>> blue (low values) to red (high values).  If amax is 0, each time a 
>>>> spectrum is calculated (a single horizontal line on the plot), its 
>>>> maximum value is determined and assigned to the highest (red) color.
>>>> Each line will have a different relationship between spectral amplitude
>>>> and color, although the maximum will always be red and the minimum
>>>> will always be blue.  To make a spectrogram for which the colors 
>>>> are meaningful and consistent we need to establish a constant relationship
>>>> between spectral amplitude and the color spectrum.  To do this, we assign
>>>> a spectral amplitude of 0 to blue and assert a spectral amplitude value
>>>> (amax) to be assigned to red.  Any higher values will be plotted as red.
>>>> The assignment of amax is largely a matter of taste, but you can get
>>>> a start by setting the option SaveDrifts to log the values being calculated.
>>>> As a rule of thumb, low values of amax will make spectrograms sensitive
>>>> to the background noise levels; high values will be senstive to larger events.
>>>>

# 18 scale           A scaling factor for the wiggle-line trace.

>>>>
>>>> A data trace is plotted along the right-hand time axis for comparison
>>>> with the spectrogram.  The scaling factor for the amplitude of this
>>>> trace is arbitrarily assigned.  Make a plot and use the above scaling factor
>>>> to adjust the amplitude to taste.
>>>>

# Comment            A comment for the top of the display.
#                                      
#     S    C   N  04 05 06  07  08 09 10   11  12 13 14 15    16  17       18   Comment
#                   

 Plot MCM  VHZ NC 24 2  -7  PDT 1  0  600  1   1  60 10 0.2   2   40000    1.0 "Convict Moraine"
 Plot MCS  VHZ NC 24 2  -7  PDT 1  0  350  1   1  60 10 0.0   1   30000    1.0 "Casa Diablo Hot Springs"
 Plot MDP  VHZ NC 24 2  -7  PDT 1  0  350  1   1  60 10 0.0   2   30000    1.0 "Devil's Postpile"
 Plot MDP  DLI NC 24 2  -7  PDT 1  0  350  1   1  60 10 0.0   2   30000    1.0 "Devil's Postpile"

    # *** Optional Commands ***

# The file with all the station information.
 StationList     /home/earthworm/run/params/calsta.db

>>>>
>>>> This is the database file describing the characteristics
>>>> of each channel.  
>>>>

 Make_HTML          # Flag to construct html wrapper

>>>> 
>>>> If this flag is set, a html wrapper is constructed for display
>>>> of the spectrograms.
>>>> 

 Days2Save     7    # Number of days to display on web page; default=7

>>>> 
>>>> This variable governs how many dates are retained in the SCN.hist
>>>> file in your local gifs directory.
>>>> 

 UpdateInt    10    # Number of minutes between updates; default=2

>>>> 
>>>> Each UpdateInt minutes, sgram wakes up and updates all its files.
>>>> 

 RetryCount    2    # Number of attempts to get a trace from server; default=2

>>>> 
>>>> If you can't get it in two trys, there is probably something
>>>> seriously wrong.  Nonetheless, there may be reasons to try some more.
>>>> 

 Logo    smusgs.gif # Name of logo in GifDir to be plotted on each image

>>>> 
>>>> If a Logo is specified, the appropriate gif file should be 
>>>> placed in the /GIFs/sgram/ directory so the program can
>>>> find it.  If specified the Logo will be appear in the upper
>>>> left corner of each SCN/date spectrogram plot.
>>>> 


# We accept a command "SaveDrifts" which logs drifts to GIF directory.
  SaveDrifts

>>>> 
>>>> If SaveDrifts is set, a file SCN.drft is created for each SCN.  
>>>> Each time a new spectrogram line is started, the mean, rms, 
>>>> min and max values for the first minute of data are recorded.
>>>> 

# We accept a command "BuildOnRestart" to totally rebuild images on restart.
  BuildOnRestart

>>>> 
>>>> During the process of tuning the displays this option is useful.
>>>> For production, this should be disabled.
>>>> 


# We accept a command "Debug" which turns on a bunch of log messages
# Debug
# WSDebug
****************************************************************************************


Known Bug!
	The library functions which handle the process of reading an existing .gif file, 
modifying it, and writing it back out are not good at detecting damaged files.  Sometimes, 
if sgram stops or is halted in mid-stream, we are left with a damaged .gif file.  
Upon restart, the process will crash and burn without error messages when an attempt 
to read this file is made.  There are two solutions; 1) use the BuildOnRestart option above 
to force overwriting of the file or 2) go to the gif directory and delete the offending 
spectrogram .gif file.  


