USGS Home
Contact USGS
Search USGS

Data Processing

Home | Abbreviations | Data Collection | Data Processing | Data Access | Metadata | Appendix 1

Raw Data Files and DAPS Processing

The Data Acquisition, Processing and Storage (DAPS) group maintains an online field activity catalog for the CMGP known as InfoBank. This catalog is a structured information storage scheme of databases and software that provide access to USGS-CMGP data and metadata. Infobank can be accessed at http://walrus.wr.usgs.gov/infobank.

The raw and processed digital data files are stored in InfoBank by cruise identifier (ID) and data type in ASCII text format. The DAPS group uses a numbered system of filename extensions to distinguish between data types and processing levels. For example, bathymetry files have a .3xx filename suffix and navigation files have a .0xx, or .6xx suffix. More information about the data type numbering (.xxx) conventions used within InfoBank is provided in appendix 1. InfoBank also denotes the most processed file as the "best file" (see fig. 2).

Figure 2. Screenshot of the InfoBank website for the bathymetry data of cruise A-1-00-SC. The red box highlights the "best file" designation and the yellow box illustrates the data type numbering conventions used within InfoBank. Appendix 1 provides more details about InfoBank file naming conventions.

These "best" digital data files have been processed through various USGS developed software packages depending on data type (bathymetry or navigation) and data quality. Details about these programs can be accessed at the following Websites.

Bathymetry:

http://walrus.wr.usgs.gov/infobank/programs/html/programclassification/bat.html

Navigation:

http://walrus.wr.usgs.gov/infobank/programs/html/programclassification/nav.html

The bathymetry datasets were commonly processed using three programs. (1) batred, which calculates depth from travel times; (2) nosamebathy, which eliminates identical adjacent bathymetry values; and (3) nozerobathy, which removes records with a depth value of "0.0". Not all of the datasets included in this report were processed with one or more of these programs, however. Details about which processing steps were taken by the DAPS group are provided in the header information of the bathymetry and navigation ASCII files downloaded from InfoBank.

Editing in Excel in Preparation for MGD77T Reformatting

Prior to reformatting the data into MGD77T format the "best" bathymetry and navigation files from InfoBank were edited to remove the header information and reformatted using Excel 2010 to separate the data into tab-delimited columns. Next the bathymetry and navigation files were then merged. Lastly, the combined navigation and bathymetry was saved as a .txt file.

Processing Data Files into MGD77T Format Using NGDC's GEODAS Program

The NGDC created a program to assist in the formatting of geophysical data into the MDG77T format. This program is called GEODAS ReFormat to MDG77T and can be downloaded for use at http://www.ngdc.noaa.gov/mgg/gdas/gx_announce.Html.

The steps used for processing our single beam bathymetry datasets were as follows:

1. Opened the program.
2. Clicked on the "GO" button.
3. Chose Input format type : "Fields are delimited by tab, comma or spaces".
4. Chose Survey ID : Used "enter for each input file(s)".
5. Chose Line/Track/Segment ID : "No Line/Track/Segment Ids".
6. Selected input file, this was the .txt file saved previously from the combined navigation and bathymetry datasets.
7. Entered the character "!" in the next window when asked to "enter header records to skip in file".
8. Left field blank on the next window when asked for no-value string for data fields.
9. Chose data fields in input format based on what the columns represent in the original InfoBank data headers.

   For step number 9 above, the first four input fields selected were always the same input choices. These were:

   1. Date-Time YYYYJJJHHMMSSs
   2. Latitude Decimal Degrees
   3. Longitude Decimal Degrees
   4. Time in seconds to nav fix SSSS.s

   Additional data inputs depended on the type of bathymetry data retrieved from InfoBank, which was indicated in the original data file header information.

   Choices included (corrections are for sound velocity to depth):

   1. Un-corrected Depth pos meters
   2. Un-corrected Depth neg meters
   3. Corrected Depth pos meters
   4. Corrected Depth neg meters
10. Once the input fields were chosen, clicked OK.
11. Entered the Survey ID information. The program then created an output file.
12. Opened the output file in Excel 2010.
13. Added zeros to the time-zone correction field column, indicating that no time zone correction was needed.
14. Visually scanned the file verify that it was processed correctly.
15. Saved file in MGD77T (.m77t) format.

Creation of MGD77T Header Files

To create the accompanying header files (.h77t), an Excel spreadsheet was supplied from the NGDC with already populated required field names.

Information about the NGDC header field requirements and definitions is available at http://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.pdf (74KB PDF).

When creating a header file for each cruise, fields were populated in the header spreadsheet based on InfoBank metadata records, digital data file headers, and accompanying publications (often cruise reports) for the survey. If a field was unknown or did not apply to the dataset it was intentionally left blank.

Data Validation

Processed bathymetric depth values are then spot checked based on position against the Global Multi-Resolution Topography (GMRT) grid within GeoMapApp. This ensures that data depths are reasonable; datasets with depth values significantly different from the global grid bathymetric depths were excluded from this archive and not submitted to the NGDC. They were also reported to the DAPS group as possibly erroneous or poor-quality data.

Home | Abbreviations | Data Collection | Data Processing | Data Access | Metadata | Appendix 1