heli1 - Helicorder displays of waveforms for the WWW

**** Jim Luetgert 18 Oct 1999   29 Jan 2003 ***

heli1 is designed to rapidly provide GIF images of waveforms for display on the WWW.  
heli1 periodically retrieves current data for a selected list of stations and updates 
GIF images which are transferred to webserver(s).  
The required waveforms are obtained via periodic requests to waveserver(s) and plotted 
directly to GIF format image files.  As the required GIF files are constructed, they 
are copied to a local directory.  

You can specify, using the command LocalTarget, a destination directory 
on the local machine for the completed helicorder plots.
This can be the directory where your webserver expects to find them or it can be 
a directory used by Will Kohler's module sendfile to transfer the files to a remote 
webserver via a secure socket connection.  A further alternative is to use sfilercp to 
transfer the files to a remote webserver.

I use Perl Scripts on the webserver(s) to dress up the data displays.  

How to set it up - 
The first task is to set up a directory as a holding tank for the GIF image 
files to reside while they are waiting to be dispatched to the webserver(s).  
I have found it useful to set up a directory and sub-directory of form   .../gifs/helicorder/.  
This provides the opportunity for other subdirectories of /gifs/ for other 
image creating modules.  
If you are going to display a logo on your helicorder gifs, put a copy of the logo.gif
in this directory.  If you are using a Perl Script on the webserver(s) to generate
your html display, you need do nothing further.    

Examples of a Perl Script are included.

One of the goals of looking at seismograms is the ability to relate the wiggles
back to ground motion.  To do this, we need to know several things about the 
sensors and electronics for each channel.  This information must be stuffed into 
a database file describing your network (or at least the channels you are using).
Constructing this file is a bit of a pain, but it is useful for other modules
where you might like to know the real ground motion, such as magnitude calculations.
Here is an example of the first part of the database file for northern California:

# S N C LatD LatM LonD LonM Elev Inst Inst_sens Gain Sensor Sens_unit Sens_sens SiteCorr Comment
#
# 1 - NCSN analog = 0.9954 microv/count
# 2 - NCSN DST    = 6.865  microv/count
# 3 - RefTek      = 1.907  microv/count
# 4 - Nanometrics = 1.0    microv/count
# 5 - Quanterra   = 2.5    microv/count
#
# 1 - L4c    = 1 v/cm/s
# 2 - FBA-23 = 1.25 v/g = 0.001278 v/cm/s/s
# 3 - Guralp = 8 v/cm/s
# 4 - HS1    = 0.448 v/cm/s
# 5 - Wilcoxin = 10 v/g = 0.01022 v/cm/s/s
# 6 - Dilatometer = 1 v/0.000001 strain
# 7 - Streckeisen STS-1 = 24 v/cm/s
# 8 - Streckeisen STS-2 = 15 v/cm/s
# 9 - Episensor = 5 v/g = 0.005112 v/cm/s/s
#    Sens_unit = 1 (displacement); 2 (velocity); 3 (acceleration)
# 
#                                   Inst Inst_sens Gain Sensor Sens_unit Sens_sens SiteCorr
#                                      \         \    \     \   |        /         /
#                                       \         \    \     \  |       /         /
#                                        \         \    \     | |      /         /
#                                         \         \    \    | |     /         /
#                                          \         \    \   | |    /         /
#                                           \         |    |  | |   |         /
#                                            \____    |    |  | |   |     ___/
# Nanometrics - 4                                 |   |    |  | |   |     |

KMPB  NC  HLZ  40 25.0506 124  7.2368  9310.0     4 1.00 1.00 9 3 5.00 1.00   "Mount Pierce"
KMPB  NC  HHZ  40 25.0506 124  7.2368  9310.0     4 1.00 1.00 3 2 8.00 1.00   "Mount Pierce"

CCH1  NC  DDI  37 44.5920 122  5.8020   145.0     4 1.00 1.00 6 1 1.00  1.00   "Chabot Dilatometer"
CCH1  NC  VDN  37 44.5920 122  5.8020   145.0     4 1.00 1.00 4 2 0.448 1.00   "Chabot N (Downhole)"
CCH1  NC  VDE  37 44.5920 122  5.8020   145.0     4 1.00 1.00 4 2 0.448 1.00   "Chabot E (Downhole)"
CCH1  NC  ADZ  37 44.5920 122  5.8020   145.0     4 1.00 1.00 5 3 10.0  1.00   "Chabot DIG FBA Z (Downhole)"
CCH1  NC  ADN  37 44.5920 122  5.8020   145.0     4 1.00 1.00 5 3 10.0  1.00   "Chabot DIG FBA N (Downhole)"
CCH1  NC  ADE  37 44.5920 122  5.8020   145.0     4 1.00 1.00 5 3 10.0  1.00   "Chabot DIG FBA E (Downhole)"

CGP1  NC  DDI  37 38.7236 122  0.6862   337.0     4 1.00 1.00 6 1 1.00  1.00   "Garin Park Dilatometer"
CGP1  NC  VDN  37 38.7236 122  0.6862   337.0     4 1.00 1.00 4 2 0.448 1.00   "Garin Park N Downhole"
CGP1  NC  VDE  37 38.7236 122  0.6862   337.0     4 1.00 1.00 4 2 0.448 1.00   "Garin Park E Downhole"
CGP1  NC  ADZ  37 38.7236 122  0.6862   337.0     4 1.00 1.00 5 3 10.0  1.00   "Garin Park DIG FBA Z (Downhole)"
CGP1  NC  ADN  37 38.7236 122  0.6862   337.0     4 1.00 1.00 5 3 10.0  1.00   "Garin Park DIG FBA N (Downhole)"
CGP1  NC  ADE  37 38.7236 122  0.6862   337.0     4 1.00 1.00 5 3 10.0  1.00   "Garin Park DIG FBA E (Downhole)"

Someday this file will be replaced by queries to the Database, but until that is available,
a flat file will suffice.

Make sure that your earthworm.d file contains a module id for heli1.  It should be MOD_HELI.
Insert an entry for heli1 in your startstop.d file.
Now comes the fun part; setting up the configuration file.  

heli1.d -

****************************************************************************************
#
# This is the heli1 parameter file. This module gets data gulps
# from the waveserver(s), and creates helicorder displays.


#  Basic Earthworm setup:
#
LogSwitch    1              # 0 to completely turn off disk log file
MyModuleId   MOD_HELI       # module id for this instance of report 
RingName     HYPO_RING      # ring to get input from
HeartBeatInt 15             # seconds between heartbeats

>>>>
>>>> The first four config variables are standard earthworm.
>>>>

# List of wave servers (ip port comment) to contact to retrieve trace data.
 WaveServer 130.118.49.67  16022      "wsv2 - ad1"
 WaveServer 130.118.49.67  16023      "wsv2 - ad2"
 WaveServer 130.118.49.67  16024      "wsv2 - ad3" 
 WaveServer 130.118.49.210 16022      "wsv1 - ad1"
 WaveServer 130.118.49.210 16023      "wsv1 - ad2"
#WaveServer 130.118.49.210 16024      "wsv1 - ad3" 
 WaveServer 130.118.49.210 16025      "wsv1 - 1 Nano"
 WaveServer 130.118.49.210 16026      "wsv1 - 1 DST"

>>>> 
>>>> This is the list of waveservers to try.  If the requested
>>>> data for a given SCN is on more than one waveserver, the
>>>> first listed will be preferentially used.  If a request
>>>> fails (gap in the data, broken connection) the alternate
>>>> server(s) will be tried.  The comment field is optional;
>>>> it makes error messages a bit more helpful.  The maximum
>>>> number of waveservers is specified by MAX_WAVESERVERS in
>>>> heli1.h.
>>>> 

# Directory in which to store the temporary .gif files.
GifDir   /home/earthworm/gifs/helicorder/  

>>>> 
>>>> This is the directory on your local machine which is used to
>>>> store the working versions of the GIF and .html files.  
>>>> For each SCN, there may also be a SCN.hist file which keeps 
>>>> track of which dates are available for display.  
>>>> This is used to construct the optional index.html file.  
>>>> If SaveDrifts (see below under optional commands) is set, 
>>>> a file SCN.drft is created for each SCN.  Each time a new 
>>>> helicorder line is started, the mean, rms, min and max 
>>>> values for the first minute of data are recorded.
>>>> If you are plotting a logo on your helicorders, put a copy
>>>> in this directory.
>>>> 

# Plot Display Parameters - 
		# The following is designed such that each SCN creates it's own
		# helicorder display; one per panel of data.
		
# 01 HoursPerPlot    Total number of hours per gif image

>>>> 
>>>> I have been using 24 hours per image because it corresponds 
>>>> to a standard helicorder record.
>>>> 

# 02 Minutes/Line    Number of minutes per line of trace

>>>> 
>>>> I usually make this 15 so that we get four lines per hour.
>>>> 

# 03 Plot Previous   On startup, retrieve and plot at least n previous hours from tank.

>>>> 
>>>> This lets you fill in those little gaps left by system crashes,
>>>> power outages, silly mistakes, etc.  I usually leave this
>>>> set to 1 so that gaps get filled automatically if something
>>>> bad happens and statmgr has to restart heli1.  This will only
>>>> let you fill in past data from today's plot (i.e. <=HoursPerPlot).
>>>> If you need past plots use the DaysAgo parameter below. 
>>>> 

# 04 XSize           Overall size of plot in inches
# 05 YSize           Setting these > 100 will imply pixels

>>>> 
>>>> These let you tailor the size of your display.
>>>> 

# 06 Show UTC        UTC will be shown on one of the time axes.
# 07 Use Local       The day page will be local day rather than UTC day.

>>>> 
>>>> These variables govern how the time axes are displayed on 
>>>> your plot and whether the plot starts at 00:00 UTC or local.
>>>> 

#                                      
#        01  02 03 04  05 06 07 

 Display 24  15  1 10  20  1  0  

>>>>
>>>> These are parameters which should be common to all channels.
>>>> 

# Channel Parameters - 
		# The following is designed such that each SCN creates it's own
		# helicorder display; one per panel of data.
# S                  Site
# C                  Component
# N                  Network

>>>> 
>>>> These are the SCN identifiers as they appear in the 
>>>> waveserver menu.
>>>> 

# 04 Local Time Diff UTC - Local.  e.g. -7 -> PDT; -8 -> PST
# 05 Local Time Zone Three character time zone name.

>>>> 
>>>> These variables govern how the local time axes are displayed on 
>>>> your plot.  If UseDST is enabled (see below) put the standard time here.
>>>> 

# 06 Plot Previous   On startup, retrieve and plot n previous hours from tank.

>>>> 
>>>> This lets you fill in those little gaps left by system crashes,
>>>> power outages, silly mistakes, etc.  I usually leave this
>>>> set to 1 so that gaps get filled automatically if something
>>>> bad happens and statmgr has to restart heli1.  This will only
>>>> let you fill in past data from today's plot (i.e. <=HoursPerPlot).
>>>> This value will be set to the greater of itself or the Plot Previous
>>>> value in the Display parameter above.
>>>> If you need past days plots use the DaysAgo parameter below. 
>>>> 

# 07 Scale Factor    Scale factor to dress up image.

>>>> 
>>>> This lets you turn down the amplitude of those really noisy sites.
>>>> 

# 08 Mean Removal    Mean of 1st minute of each line will be removed.

>>>> 
>>>> Unfortunately, some data sources are not zero-mean and others can
>>>> can have unacceptable drift rates.  If this flag is set, each time
>>>> a new line is started, the mean of the first minute in the line
>>>> is calculated and removed from the data for the rest of the line.
>>>> This is not bulletproof! It can fail if there is a noise burst
>>>> in that first minute.
>>>> 

# 09 Filter          Apply a lo-cut filter.

>>>> This sometimes helps with an unacceptable drift rate.

# Comment            A comment for the top of the display.
#                                      
#     S    C   N  04  05   06   07  08   09

 SCN  MDP  VHZ NC -7  PDT  1   1.0   1    0  "Devil's Postpile"
 SCN  MSL  VHZ NC -7  PDT  1   1.0   1    0  "Sherwin Lakes"
 SCN  MLC  VHZ NC -7  PDT  1   1.0   1    0  "Laurel Creek Canyon"
 SCN  MCM  VHZ NC -7  PDT  1   1.0   1    0  "Convict Moraine"
 SCN  MCS  VHZ NC -7  PDT  1   1.0   1    0  "Casa Diablo Hot Springs"
 SCN  MPR  VHZ NC -7  PDT  1   1.0   1    0  "Pilot Ridge"
 SCN  MEM  VHZ NC -7  PDT  1   1.0   1    0  "East Mammoth"
 SCN  MCV  VHZ NC -7  PDT  1   1.0   1    0  "Convict Lake"
 SCN  MGP  VHZ NC -7  PDT  1   1.0   1    0  "Gravel Pit"
 SCN  MMS  VHZ NC -7  PDT  1   1.0   1    0  "Mammoth Summit"
 SCN  MDR  VHZ NC -7  PDT  1   1.0   1    0  "Doe Ridge"



    # *** Optional Commands ***

# The file with all the station information.
 StationList     /home/earthworm/run/params/calsta.db

>>>>
>>>> This is the database file describing the characteristics
>>>> of each channel.  
>>>> heli1 will run without this, but the reported scaling will be incorrect.
>>>>

# List of target directories on this computer to place output.
#                      Directory
 LocalTarget /home/ppicker/waveforms/helicorder/ 

>>>> 
>>>> Here we specify the location of our web pages on the local machine.
>>>> This may be a directory for a local webserver to access or 
>>>> it may be a directory for sendfile to use as a source for 
>>>> sending files to a remote webserver.
>>>> When setting up a directory structure on your webserver,
>>>> I would recommend a directory dedicated to GIF displays.
>>>> with a sub-directory dedicated to helicorder displays.
>>>> The maximum number of local targets is specified by 
>>>> MAX_TARGETS in heli1.h.
>>>> 

# Filename prefix on target computer.  This is useful for identifying
# files for automated deletion via crontab.
 Prefix nc

>>>> 
>>>> For each SCN/date, a (sometimes large) GIF file is created.  
>>>> Unless there is some mechanism in place
>>>> to blow away out-of-date files, the webserver will eventually
>>>> drown in old files.  To fix this problem, these types of
>>>> file are assigned a prefix to readily identify them as volatile
>>>> and a crontab process is used to blow them away. e.g.
>>>> 15 1 * * 0-6 find /home/ehzweb/waveforms/helicorder -name nc\* -mtime +7 -exec rm {} \;
>>>> will remove files starting with nc which are older than 7 days.
>>>> 

 UpdateInt    10    # Number of minutes between updates; default=2

>>>> 
>>>> Each UpdateInt minutes, heli1 wakes up and updates all its files.
>>>> 

 RetryCount    2    # Number of attempts to get a trace from server; default=2

>>>> 
>>>> If you can't get it in two trys, there is probably something
>>>> seriously wrong.  Nonetheless, there may be reasons to try some more.
>>>> 

 Logo    smusgs.gif # Name of logo in GifDir to be plotted on each image

>>>> 
>>>> If a Logo is specified, the appropriate gif file should be 
>>>> placed in the /GIFs/helicorder/ directory so the program can
>>>> find it.  If specified the Logo will be appear in the upper
>>>> left corner of each SCN/date helicorder plot.
>>>> 

 wsTimeout 5   # time limit (secs) for any one interaction with a wave server.

>>>>
>>>> We need to know a reasonable time limit for getting
>>>> a response from any wave server.
>>>>


# We accept a command "SaveDrifts" which logs drifts to GIF directory.
  SaveDrifts

>>>> 
>>>> If SaveDrifts is set, a file SCN.drft is created for each SCN.  
>>>> Each time a new helicorder line is started, the mean, rms, 
>>>> min and max values for the first minute of data are recorded.
>>>> 

# We accept a command "BuildOnRestart" to totally rebuild images on restart.
# The argument, DaysAgo, if >0 will plot old data and exit.
# BuildOnRestart 0    

>>>> 
>>>> Normally, the images being accumulated in the GifDir are initialized 
>>>> only when a new panel of data is begun.
>>>> During the process of tuning the displays this option is useful.
>>>> For production, this should be disabled.
>>>> 

# Decimate 1

>>>>
>>>> Normally, data is decimated before plotting to be roughly compatible with a
>>>> 72 dpi pixel density.  The decimation factor is computed online based 
>>>> upon the sample rate and plot scaling. This may be factor of 50-200. 
>>>> If a true imitation of a helicorder plot is required, in which the
>>>> "pen" overwrites itself, you can force the decimation to a smaller value.
>>>>

# UseDST

>>>>
>>>> If this flag is set, local time is automatically changed to/from Daylight
>>>> Savings Time on the appropriate days for North America.
>>>> If you use this feature, make sure that you have specified standard 
>>>> time in SCN above.
>>>>

# StandAlone      # Run heli outside of earthworm.

>>>> 
>>>> This option allows heli1 to be run as a standalone module.
>>>> Shared memory is not accessed and heartbeats are not sent or received.
>>>>
 
# DaysAgo 2    # Start plotting the requested number of days ago.

>>>> 
>>>> The argument should be >= 0.  If > data in tank, will start with oldest data.
>>>> Will plot one day's data and exit.
>>>> This allows you to retrieve older data from the waveserver tank.
>>>> For production, this should be disabled.
>>>> Setting this variable will cause BuildOnRestart to be set.
>>>> 

# We accept a command "Clip" to clip the traces at N pixels.
# Clip  200     

# Gulp  300       # Specify number of seconds of data to acquire at a time.

# PlotUp          # Plot from bottom to top.
  PlotDown        # Plot from top to bottom. [default]

  LabelDC         # Label each trace line with its dc offset.
# NoLabelDC       # Don't label each trace line with its dc offset [default].

>>>>
>>>> 

# We accept a command "Debug" which turns on a bunch of log messages
# Debug
# WSDebug
****************************************************************************************



My personal preference is to use a PerlScript on the webserver to create and manipulate
the web page content.  The helicorder files have a considerable amount of information 
encoded in their file names.  This allows Perl to do a lot of sifting and sorting of 
data to make it easier on the user.  For example, we can easily look at all available 
data for one channel or all available channels for one day.  


****************************************************************************************



Known problems:
1)	If a crash occurs while the local copy of a gif file is being written, the file can 
become corrupted.  When the program is started again, it may go zombie when trying 
to re-read the file.  Either remove the .gif file from your local gif directory or 
set the BuildOnRestart flag.