|
Historically, instruments logged large amounts of data internally, encoded in proprietary binary formats, so special software to read files in those formats was required. With the advent of more storage in smaller packages, instrument developers have started to output the data in American Standard Code for Information Interchange (ASCII) files. The software we use has evolved over time to accommodate these changes. The section below follows the same order as the instrument description section.
Disclaimer
The following instrument-specific processing descriptions represent instruments and workflows that were in place from before the original version of this publication (2008) through version 2.0 (2016). Since then, the Coastal and Marine Hazards and Resources Program (CMHRP) has begun evolving these data-processing workflows and making them more consistent throughout the CMHRP. Presently this effort is in the transitional phase. Links to latest processing code and workflow descriptions can be found at the Oceanographic Time-Series Data Collection overview page.
Acoustic Doppler Current Meter (ADCP) Data
Data from ADCPs are processed using the ADCP Toolbox, a collection of MATLAB® routines written at the USGS Woods Hole Coastal and Marine Science Center (WHCMSC), described in Coté and others (2005). We compare data from the four beams in pairs for symmetry and check for high-intensity reflection indicative of fish or other beam obstructions. "Bad" values are masked both manually and automatically and, where possible, a three-beam solution is used to fill in gaps caused by one compromised beam. Averages replace short-duration spikes in the data, with pad values replacing bad data points to maintain timebase consistency. Profiles are trimmed at the surface (6 percent of depth) using software provided by the manufacturer. However, bins that are out of the water at low tide are retained during higher tides to retain the full tidal range. We remove data collected by the instrument before and after the deployment using the instrument's tilt and compass data to define when the tripod was stably settled on the ocean floor. Finally, we rotate the data to compensate for magnetic variation from true north at the site and convert from the beam coordinates recorded by the ADCP into earth coordinates. Processed versions of these data files are stored in EPIC-compliant netCDF files.
The WHCMSC commonly configures ADCPs to collect wave data simultaneously with current data. The software provided by manufacturer is used to split the raw file and compute the wave statistics and spectra. MATLAB® programs written at WHSC are used subsequently to create EPIC-compliant netCDF files from the individual waves file. The waves processing software is described in Sullivan and others (2006).
Acoustic Doppler Current Profiler (ADCP) Sentinel V Data
The V is the new version of the ADCP, and it can output processed, calibrated data in ASCII or MATLAB .mat format, so a different form of processing and conversion software is currently under development. The manufacturer also provides processing software that is being evaluated as a replacement for our current (2016) processing method.
Nortek Aquadopp, Aquadopp HR, and Acoustic Wave and Current Profiler (AWAC) Data
Depending on configuration, the data may be logged as a single or multiple binary files. After the data are offloaded, processing software is employed to read the instrument files and convert the raw data to east, north, and up in EPIC-compliant netCDF files. Because of the variety of possible settings, there are several versions of the code: one to handle burst data, one to handle continuous sampling, and one for profiling. The manufacturer’s waves software (Quickwave) output can be converted to the format of the waves data from other instruments.
Nortek Vector
The software to read the raw burst files from the Vector and output the data in a form similar to ADV data is under development. Since the orientation sensors are in the housing, and because the head can be mounted in any orientation, the transformation from head x, y, z to Cartesian coordinates may not be possible simply by rotating. The principal component method is being evaluated as an alternate method of analysis.
Acoustic Doppler Velocimeter (ADV) and Pulse Coherent Acoustic Doppler Profiler (PCADP) Data
Binary files logged by ADVs and PCADPs are processed using a series of MATLAB® routines in the Hydratools toolbox developed at the USGS WHSC (Martini and others, 2007). The ADV and PCADP both employ burst sampling at high frequency to characterize the water flow, so these data types have one time series for each burst logged, and each file contains as many bursts as were logged during the deployment. Statistics, including mean and standard deviation, maxima, minima, and medians are computed on the bursts as part of the initial conversion from binary to EPIC-compliant netCDF and stored in a separate file.
In some ADVs, the instrument firmware from SonTek had a faulty Digital Signal Processing (DSP) chip that corrupted part, but not all, of the data from many bursts. Other sources of data degradation included fouling and flow obstacles such as seaweed and animals. To handle data flaws and retain as much viable data as possible, a system to vary the cutoff for minimum beam correlation and maximum standard deviation across each burst was developed to use with software for flagging bad sections. Small data gaps are interpolated over and larger sections of unusable data are replaced with pad values to maintain the timebase. Spikes in pressure and range to seabed data and obviously faulty records are replaced with the fill value. After questionable data records are treated, the raw velocities are rotated into east-north-up coordinates, and the mean and standard deviation statistics are recalculated and data are low-pass filtered. Separate files are written for raw burst data and for time series of the averaged statistics of the bursts. The mean current velocities in the statistics files are comparable to the current data from mechanical current meters.
SonTek Argonaut
The data are offloaded in several binary files. Processing software was developed and employed to read the instrument files and convert the raw data to east, north and up in EPIC-compliant netCDF files.
SonTek IQ
The instrument outputs data in a proprietary form. Software from the manufacturer allows conversion to .mat format output files that are used as input to the conversion programs. Software to generate time series of flow and profile outputs as EPIC-compliant netCDF was developed and used. The variables that have EPIC equivalents use the EPIC name; the other variable names closely match the names used by the manufacturer for convenience in correlation with the manufacturer’s documentation.
MicroCAT (MC), SeaCAT (SC)
Software provided by Sea-Bird Electronics (SeaSoft) is used to validate the raw data and compute salinity from time series of temperature and conductivity measurements recorded by Sea-Bird sensors. Water density is ordinarily computed from the temperature, salinity, and pressure measurements. Transmissometers and fluorometers are commonly logged by the SeaCATs and MicroCATs, so other variables may be present. Data processed with SeaSoft are stored in ASCII files so the final step is conversion to EPIC-compliant netCDF using a program called asc2epic.
YSI 6600 and EXO
The data from these instruments are stored in column-delimited ASCII files in scientific units. Software to read the raw data and write EPIC-compliant netCDF output files was developed and is employed. Calibrations are applied internally. Custom programs to read the ASCII files and write EPIC-compliant netCDF were developed.
Satlantic ISUS
The data are offloaded in several binary files. Processing software to read the instrument files and convert the raw data to EPIC-compliant netCDF files was developed and employed.
Optical Backscatter Sensor (OBS) Data
The SonTek Hydra datalogger converts raw OBS voltage measurements to counts with a 14-bit analog-digital converter. A linear conversion is used to convert counts back to volts so the data can be calibrated to sediment concentration using laboratory-derived calibration coefficients. After the point at which biological fouling significantly degrades the returns, data are replaced with the fill value. The OBS data are stored in the file with other data from the instrument that logged it (either an ADV or PCADP).
WET Labs ECO sensors (PAR, NTU, FLNTU)
The data from these instruments are stored in column-delimited ASCII files in counts. Software to read the raw data, apply the calibrations for that sensor, convert to scientific units, and then write EPIC-compliant netCDF output files was developed and is employed.
Transmissometer Data
The Sontek Hydra dataloggers also log transmissometer data in counts, so a linear conversion is necessary to convert counts to volts. The SeaCAT, however, logs the raw voltage, so no conversion is required. The transmissometer data can also be presented either as percent light transmission (from 0 to 100, where 0 indicates complete occlusion). Beam attenuation coefficients (units of m-1) were computed from the light transmission observations as -4(ln(T/100)), where T is percent light transmission over a beam length of 0.25 meters. The beam attenuation coefficient is linearly proportional to the concentration of suspended material in the water if the particles are of uniform size and composition (Moody and others, 1987). However, the size of the particles in the water changes with time, especially during resuspension events; therefore, the beam attenuation measurements must be interpreted with care. The processed data are usually stored in the file with other data from the logger to which the transmissometer was connected (MIDAS, ADV, PCADP, or SeaCAT).
Acoustic Backscatter Sensor (ABS) Data
Aquatec ABS raw data are logged in a set of binary files and processed with a MATLAB®-based toolbox that truncates files to include in-water times only, flags questionable points, and transforms the data to netCDF format. The ABS data are not subject to the normal conversion to scientific units and quality-control procedures because calibration protocols are not yet in place. When in doubt, we prefer to preserve all samples rather than remove potentially useful data. These data are distributed as provisional.
Laser In Situ Suspended Scattering and Transmissometry (LISST) Data
LISST raw data are extracted from the logger and stored in the native format. Sequoia Scientific Inc., the manufacturer, supplies processing software. Calibration methods and additional software for working with the data and converting to netCDF are under development at WHCMSC; and until they are completed, the raw data may be distributed as provisional.
Sea-Bird Seagauge
The Seagauge uses a pressure measurement to compute wave parameters and is typically programmed to sample in bursts. The output must be converted with the manufacturer’s software to make the needed computations. It is expected that a local source of atmospheric pressure is available to apply to the raw pressure measurements. Once the conversion is complete, a program is executed to read the data and output EPIC-compliant netCDF files. The burst files match the structure of the ADV and PCADP burst data, so one tool can read all burst files.
RBR D|wave
The D|wave pressure sensor data are also used to compute wave parameters. The D|wave may be programed to sample continuously or in bursts; the data output by the instrument is ASCII. Programs were written to treat both kinds of data and write EPIC-compliant netCDF output. The burst files match the structure of the ADV and PCADP burst data, so one tool can read all burst files.
Onset HOBO U_20 Wave Logger
Software from the manufacturer is initially used to read the data, apply calibrations, and convert to scientific units. The data may be output in comma-delimited ASCII files or in MATLAB .mat format files. Software to read either form of data and write EPIC-compliant netCDF output files was developed and is employed.
Down East Buoy-Mounted Meteorology Sensors
This system is designed to telemeter data to shore every hour via satellite and to log internally, so the data may be obtained after recovery if telemetry fails. A suite of programs was written to automate calling the buoy and issuing commands to download the data for processing. The data are column-delimited ASCII files with calibrations and conversion to scientific units done internally. A program to read the data and write EPIC-compliant netCDF output files was developed and is employed.
Onset HOBO Weather Station
Software from the manufacturer is initially used to read the binary data, apply calibrations, and convert to scientific units. The data may be output in comma-delimited ASCII files or in MATLAB .mat format files. Software to read either form of data and write EPIC-compliant netCDF output files was developed and is employed.
Sonar Images
The original logger for Imagenex sonar data stores each scan in a separate file, so after a 10-day deployment with hourly sampling, 240 files should exist. Those files are decoded to determine the signal amplitude for a series of pings or sweeps.
In 2012 we brought ASL IRIS loggers into use to record data from the Imagenex sonars. They store data using Imagenex proprietary formats, so a new front end was written to interface with the IRIS output files. In both cases the individual sonar files are combined to create a time series for the deployment. All the files from imaging sonar are aggregated into one netCDF file for the deployment. However, the data from the profiling sonar when used with the rotating head are too large for all the data from a deployment to be stored in one file. Usually 60 profiles are made in a “sample,” each profile rotated 3 degrees from the previous position. The samples made in one day are aggregated and stored in an azimuth (az) file, named by the sample date. The azimuth files are not published but are available by request.
The data may then be converted for display in a polar view, with scaling for the radius of the sweep or ping, but are not typically published. The sonar data are considered provisional, since processing and quality-control methods are still under development.
|
U.S. Department of the Interior |
U.S. Geological Survey
