- Date: May-2010 (process 1 of 16)
-
Once the navigation and raw data were assessed to be okay, the actual processing of the resistivity data could start. Of note, see the metadata for various jd***gps_bestdepth shapefiles (available at <http://pubs.usgs.gov/of/2011/1039/html/ofr2011-1039-catalog.html>) for a description of the work needed to extract valid bathymetry values for use in the data processing. The resistivity data were merged with the navigation data and linearized using AGI's Marine Log Manager (MLM) software. (Note that the Marine Log Manager version is different than the software version of the AGISSAdmin software of which it is a part - although shipped together, the software is developed separately). The GPS files used are the "newgps" files that have been modified to include best available bathymetry values. The version of Marine Log Manager used was AGI SSAdmin MLM v 1.3.4.217 The GPS offset was set to 0 meters since the offset (17.6 meters for some of the lines, 16 meters for the rest) between the navigation antenna and the first electrode of the resistivity streamer was accounted for in an acquisition offset. Lateral offset is not accounted for. The lines processed represent lines from all three days of data acquisition. From April 13 - L1F7 and L4F1. From April 14 - L8F1, L9F3, and L19F2. From April 15 - L20F2 and L38F1. These line names are what the * refers to in the source used and source produced citations. This process step and all subsequent process steps were performed by the same person - VeeAnn A. Cross.
Person who carried out this activity:
VeeAnn A. Cross
U.S. Geological Survey
Marine Geologist
Woods Hole Coastal and Marine Science Center
Woods Hole, MA 02543-1598
(508) 548-8700 x2251 (voice)
(508) 457-2310 (FAX)
vatnipp@usgs.gov
Data sources produced in this process:
- Date: May-2010 (process 2 of 16)
-
Each DEP file was checked for anomalous bathymetry values, or duplicated distance along values, and those lines in the file were deleted.
Data sources used in this process:
Data sources produced in this process:
- Date: May-2010 (process 3 of 16)
-
Instead of using a single, averaged water resistivity value to process the resistivity data, I want to use a water conductivity file that supplies a value that can vary along the survey line. The water conductivity file is a comma separated ASCII text file that contains an offset value - which is the distance (in meters) between the conductivity meter and the first electrode. Then the remainder of the file has 3 columns of information: date, time, water resistivity (ohm-m).
Here's an example:
;This is a water conductivity file, a comma separated ASCII text file
;A semicolon starts a command line
;Offset is the distance between the conductivity meter and first ;electrode in meters
Offset=17.0
;date, time, WaterRes(ohm-m)
2005-03-04, 18:00:18, 0.315822053
2005-03-04, 18:00:48, 0.315059762
2005-03-04, 18:01:18, 0.314602132
2005-03-04, 18:01:48, 0.315970342
The process of preparing the YSI 600 XLM data for use with the resistivity processing was rather lengthy. The YSI data needed to be related to the resistivity data based on time. First, the raw YSI data were separated by day of acquisition into separate worksheets. Then a new column was added to reflect UTC time by adding 4 hours to the YSI time (local time).
- Date: May-2010 (process 4 of 16)
-
The GPS HYPACK navigation for each day of data collection is already in a comma-delimited text file from other processing (see metadata for the various jd***bestdepth shapefiles available at <http://pubs.usgs.gov/of/2011/1039/html/ofr2011-1039-catalog.html>). These files simply need to be reformatted for use with the YSI data so that the Excel command VLOOKUP can be used with the lookup column being time. The idea is to carry over the latitude and longitude values from the GPS file when the YSI time is the same as the GPS time. HYPACK navigation usually recorded a fix every second. There were times when HYPACK was off, and that will be discussed later. VLOOKUP has two methods of operation: one when the match is exact, and the other is approximate match. For some reason, using the exact match did not work well. Times that were exact matches still produced NA values. Therefore, the approximate match method was used. The approximate match method find the largest value less than or equal to the match value. Manual editing was done to remove the latitude longitude values that carried over from HYPACK navigation gaps.
- Date: Jun-2010 (process 5 of 16)
-
Once the VLOOKUP and editing were complete, the values were copied to a new worksheet to be exported from Excel. In order to determine valid YSI values, John Bratton wanted to not only look at the values in the spreadsheet, but also look at the spatial distribution of the data point and their values. The worksheet was exported from Excel as a comma-delimited text file and added to ArcMap 9.2 as an event theme by Tools - AddXY data, projection as geographic, WGS84. The event theme was then converted to a shapefile by right mouse click - data - export. A shapefile for each day of data collection was generated.
- Date: Jul-2010 (process 6 of 16)
-
Based on the spatial distribution of the YSI data values, and visual inspection of the data files in the Excel spreadsheet, John Bratton made a final worksheet containing the YSI values he considered valid. Once these values were selected, new shapefiles were generated containing only the accepted YSI values. This meant a repeat of the VLOOKUP procedures marrying the HYPACK navigation fixes with the valid YSI values, export from Excel, added as an event theme to ArcMap, and the subsequent creation of the shapefiles.
- Date: Jul-2010 (process 7 of 16)
-
The filtered (culled) YSI shapefiles do not have the measurements in the units I need. So within ArcMap 9.2, using VACExtras v. 2.1 (an extension written by VeeAnn Cross in Woods Hole), the resistivity values in ohm-m were calculated. VACExtras - Resistivity - Calc Resistivity. The calculation requires a temperature field in degrees Celsius and a salinity field - both of which are present in the YSI culled shapefile. The results are put in a new attribute called resval.
Data sources produced in this process:
- cull_4_13_exp.shp
- cull_4_14_exp.shp
- cull_4_15_exp.shp
- Date: Jul-2010 (process 8 of 16)
-
To use the water conductivity data in the resistivity processing software EarthImager, the data have to be in the format EarthImager expects. Although the documentation is a bit sparse, I believe that the times in the water conductivity file also have to match times in the resistivity file. In order to make certain that the times used match resistivity time, I did a spatial join of the culled YSI data with the point shapefile of the resistivity GPS data. Prior to the spatial join, the resistivity shapefile was projected to UTM, Zone 18, WGS84 using ArcToolbox 9.2 in ArcMap - Data Management Tools - Projections and Transformations - Feature - Project. The original shapefiles were projected from geographic, WGS84 to UTM, Zone 18, WGS84 - no transformation necessary. By doing this, the distance attribute generated when using a spatial join will be in meters instead of decimal degrees. Prior to the join, two fields were added to each resistivity gps shapefile: resval (as double) and resutctime (as text width 15). Then right mouse click on the shapefile - Joins and Relates - Join. Join data based on spatial location, bringing over all the attributes. This join was performed for 3 days of data collection - April 13, April 14, and April 15.
Data sources used in this process:
- jd103gps_mod_utm18_4ysi.shp
- jd104gps_mod_utm18_4ysi.shp
- jd105gps_mod_utm18_4ysi.shp
- cull_4_13_exp.shp
- cull_4_14_exp.shp
- cull_4_15_exp.shp
Data sources produced in this process:
- Join_Output_jd103utm
- Join_Output_jd104utm
- Join_Output_jd105utm
- Date: Jul-2010 (process 9 of 16)
-
All records in the join shapefile where the distance was <= 10 were selected and field calculator was used to copy the values from Resval_1 (from the YSI data) to resval. Much of this work is manual. Although distance is a close approximation, because these are basically salinity values, time proximity can be more important than spatial proximity in a tidal situation. Once a combination of spatial and temporal criteria were evaluated, each resval was populated with either a reasonable YSI resval or left at 0.
- Date: Jul-2010 (process 10 of 16)
-
Before the data can be exported for use with EarthImager, the shapefiles need to be sorted to return them to their acquisition order. The spatial join function changes the order of the records. Within ArcMap 9.2 using VAC Extras 2.1 the shapefiles were sorted. VAC Extras - FeatConv - Table Sort with the primary sort field FID_1 (the FID from the YSI data) ascending order. No secondary sort field was used.
Data sources used in this process:
- Join_Output_jd103utm
- Join_Output_jd104utm
- Join_Output_jd105utm
Data sources produced in this process:
- Join_Output_jd103utm_sort.shp
- Join_Output_jd104utm_sort.shp
- Join_Output_jd105utm_sort.shp
- Date: Aug-2010 (process 11 of 16)
-
Once sorted, the necessary attributes needed to be exported from each shapefile. Within ArcMap 9.2 using XTools Pro version 5.2: XToolsPro - Table Operations - Export Table to Text. Fields to export: gpstime, gpsdate, line, resval. This exports an ANSI comma-delimited text file.
Data sources used in this process:
- Join_Output_jd103utm_sort.shp
- Join_Output_jd104utm_sort.shp
- Join_Output_jd105utm_sort.shp
Data sources produced in this process:
- jd103_4con.txt
- jd104_4con.txt
- jd105_4con.txt
- Date: Aug-2010 (process 12 of 16)
-
An AWK script was run to read the exported file and then create individual water conductivity files based on the line name within the file. The AWK script awk_waterconnew:
BEGIN {
FS=","
}
{
FS=","
if ((NR != 1) && ($4 != 0))
{
resyear=2010
resmonth=substr($2,3,2)
resday=substr($2,1,2)
restime=$1
reshr=substr($1,1,2)
resmin=substr($1,3,2)
ressec=substr($1,5,2)
ohms= $4
outfile=$3 "_water.confmt"
printf("%s-%02d-%02d, %02d:%02d:%02d, %s\n",resyear, resmonth,resday, reshr, >resmin, ressec, ohms) >> outfile
}
}
Data sources used in this process:
- jd103_4con.txt
- jd104_4con.txt
- jd105_4con.txt
Data sources produced in this process:
- Date: Aug-2010 (process 13 of 16)
-
Another set of scripts was run to add the offset value to the water conductivity file. The offset is the layback distance between the YSI sensor and the first electrode in the continuous resistivity streamer. For April 13 and 14, the layback value was 17.6 meters. For April 15, the layback was 16 meters. A shell script was run to execute the AWK script in order to process every file. The shell script dooffset:
files=`ls *.confmt | cut -d. -f1`
for file in $files
do
awk -f addoffset $file.confmt > $file.con
done
The AWK script addoffset:
{
if (NR==1)
{
printf("Offset=17.6\n")
}
print $0
}
For April 15 data, the offset value was changed form 17.6 to 16.0.
Data sources used in this process:
Data sources produced in this process:
- Date: Aug-2010 (process 14 of 16)
-
Initial tests with using the CON files did not return satisfactory results. It's possible that the water conductivity is handled like the depth files, you need at least one value to appear in each section of the EarthImager roll along processing - otherwise it can't interpolate the values. However, the documentation doesn't specify this, and technical support from AGI surmised that my assumption is correct. Visual inspection of the CONFMT files indicates large temporal gaps in resistivity values. In these cases, the resultant CON files were copied to *mod.con and manually edited by copying over rows from the CONFMT file and editing the zero value to reflect one of the measured values already in the CON file.
Data sources used in this process:
Data sources produced in this process:
- Date: Aug-2010 (process 15 of 16)
-
Now that I have viable water conductivity files, these can be used with EarthImager. EarthImager version 2.2.8 build 562 was then used to process the data files. The process settings were set to CRP - Saltwater. The *.ini file accompanying the results contains the parameters used during processing. These parameters include: minimum voltage: 0.02; minimum abs(V/I): 2E-5; max repeat error: 3%; min apparent res: 0.03; max apparent res: 1000; max reciprocal error: 5%; remove negative resistivity, smooth model inversion; finite element method; Cholesky decomposition; Dirichlet boundary condition; thickness incremental factor: 1.1; depth factor: 1.1; max number of CG iterations: 100; stop criteria: number of iterations 8; max RMS 3%; error reduction 5%; L2Norm; CRP processing using a 65% overlap. These INI files can be loaded in EarthImager to help maintain consistent processing parameters for other datasets. When the files are processed, numerous files are generated. Because of the "roll-along" nature of the processing, each line takes several iterations of processing which are then combined into a single output. The output consists of numerous files including JPEG images and text files representing the XYZ position of each resistivity value. There are four JPEG image generated with each process when possible - a long version with the x-axis labeled with distance along line (in meters) and a corresponding short version of the same information. Additionally, there is a long version with the x-axis labeled with latitudes and longitudes and the corresponding short version with the same information. The JPEG files produced uses a color scale for the resistivity that is based on the data extent from that particular file. The JPEG images also include a plot of temperature along the line. In addition to the JPEG images, there are text files with the extensions of *.llt, and *.xyz. Each of these is a text file. The LLT file has four columns of information: longitude in decimal degrees, latitude in decimal degrees, depth in meters, and resistivity value in ohm-m. The XYZ file has three columns of information: distance along line in meters, depth in meters, and resistivity value in ohm-m. There is also a file created with a UTM extension which has the same format as the LLT file except it contains the coordinates in UTM eastings and northings in the zone appropriate for the survey line (software determined). This file was not checked for accuracy as software versions in the past gave erroneous UTM fixes. An example of the file naming convention is as follows: For input files of L3F1_lin.stg and L3F1_lin_wres.dep the resulting series of output files are: L3F1_lin1_trial1.ini; L3F1_lin_AllInvRes.llt; L3F1_lin_AllInvRes.xyz; L3F1_lin_trial1_InvResLong.jpg; L3F1_lin_trial1_InvResShort.jpg. The JPEG images with distance along lines along the X-axis are L3F1_lin_trial1_InvResLong_linear.jpg and L3F1_lin_trial1_InvResShort_linear.jpg. You can process an individual line as many times as you want and the software places the results in incrementing folder names starting with trial1. These data represent trial2 (L20F2 is trial3), which is the processing using a DEP file (without an average water resistivity value) and a CON file with water conductivity values along the line.
Data sources used in this process:
Data sources produced in this process:
- *.ini
- *.llt
- *.utm
- *.xyz
- *.jpg
- Date: Feb-2011 (process 16 of 16)
-
The XYZ output file was then loaded into MATLAB version 7.5.0.342 (R2007b), along with the depth information from the DEP file, to create a new JPEG image with the same color scale for all the data files. In this manner, the JPEG images can be compared directly. Care was taken to try to get the vertical and horizontal scales uniform as well, although this was not always possible due to MATLAB limitations. These images reside in the "matlabimages" folder. These JPEG images include a black line within the resistivity profile which represents the sediment water interface based on the depth values from the DEP file. The local MATLAB script used to load the data was cp_ir_50m_conproc.m, while the local MATLAB script used to export the JPEG image was exportfig.m. The MATLAB script generating the image indicates on the x-axis legend that the data used water conductivity files during processing.
Data sources used in this process:
Data sources produced in this process: