GDA (Geologic Data Assistant), an ArcPad Extension for Geologic Mapping: Code, Prerequisites, and Instructions
by Evan E.
Thoms and Ralph A. Haugerud
Use of product or trade names in this report is for descriptive purposes only and does not constitute endorsement by the Geological Survey
Table of Contents
GDA (Geologic Data Assistant) is an extension to ArcPad, a mobile mapping software program by Environmental Systems Research Institute (ESRI) designed to run on personal digital assistant (PDA) computers. GDA and ArcPad allow a PDA to replace the paper notebook and field map traditionally used for geologic mapping. GDA allows easy collection of field data that include:
This report provides a link to downloadable code, describes the required hardware and software necessary to use GDA, describes the GDA data structure, gives setup instructions for GDA, describes moving data from GDA into a desktop GIS, and discusses customization of GDA. To use GDA efficiently will require an intermediate knowledge of computer systems and, most importantly, good file management. GDA does not have to be used in conjunction with ESRI's ArcMap, but knowledge of that program is useful for customizing certain GDA files and viewing shapefiles. Haugerud and Thoms (2004) discuss rationales behind some of the choices made in building GDA.
Note that the GDA code includes HTML help files that discuss setup and use of the software. Much of this report is copied from the GDA help files.
There are 2 downloadable GDA packages from which to choose depending on the intended host system.
To use GDA, you should have some
familiarity with your PDA, with ArcPad, and with some basic
principles of GIS.
ArcPad, and thus GDA, runs on the Windows CE 3.0 (or greater) operating system, also known as the PocketPC operating system. It can also be installed on Windows-based desktop, laptop, or tablet computers. It does not work on Palm or Linux devices. Most WinCE computers are adequate. When selecting a PDA, pay particular attention to display readability, communications capabilities, battery life, the quality of the digitizer, memory capacity, and speed. We are currently using Hewlett-Packard iPAQs with 400MHz processors, 64MB memory, and 240 x 320 pixel 16-bit color daylight-readable displays.
An extended-capacity battery, or an expansion pack with an additional battery, is desirable. You would like to have enough power for continuous operation throughout a 14-hour field day. When using a GPS it is especially useful to set your PDA to never shut off. One reason for this is simply efficiency, but the other is that turning a PDA off without first disconnecting the com port often causes the PDA to lock up so that it can only be reset with a soft reboot. The largest drains on the battery are the intensity of the backlight and the length of time it is on, so set your backlight to dim to nothing after a short period of inactivity.
Use a Compact Flash card, Secure Digital card, or other non-volatile memory for mapping project workspace. This provides storage for large raster images and guards against data loss when, inevitably, you forget and let the PDA battery run down. We use expansion packs that provide both a Compact Flash slot and an additional battery. Be aware that unless you store programs in a non-volatile space on your PDA, ArcPad will evaporate from memory if the battery dies. As a backup in the field, keep a PC nearby from which you can re-install ArcPad.
Bluetooth (a short distance device-to-device radio protocol) is excellent for communication between the PDA and a GPS unit. Cable connections between GPS and PDA are fragile, inconvenient, and hard to make waterproof.
Protect the PDA screen from abrasion with a self-adhesive replaceable plastic screen protector (check Office Depot or a similar supplier.)
The PDA probably won't survive a 4-ft drop onto bedrock. Get a rugged case that provides significant shock-proofing. Consider third-party cases (such as Otterbox) as well as cases available through your PDA manufacturer. Such cases can also be waterproof, with a clear vinyl window over the PDA screen--but if you retain this window, omit any additional screen protector.
A third-party stylus with a pocket clip or lanyard is handy. Multifunction instruments that combine a stylus with a pen or pencil (or both) are available for under fifteen dollars.
Standard recreational-grade GPS provides nominal XY accuracy of 10 meters (with averaging) at a cost of $100-300. We find this sufficient. For increased accuracy use a WAAS-capable GPS unit, or a differential GPS unit, at substantially greater cost. Note that ArcPad provides an excellent GPS display: all one needs is a screenless GPS receiver that that can send standard NMEA 0183 signals to another device.
GPS satellites broadcast at a wavelength that does not pass through the human body, so a pocket is not a good place from which to use a GPS unit. Instead, put it in the lid of a backpack, on the top of a staff protruding from a pack, or fastened it to your hat. Another option is to use a small external antenna on a hat (google on GPS Pathfinder Pocket Antenna Cap) while keeping the receiver in a pocket or on a belt clip (if you can tolerate the necessary connecting cables).
Communications between the GPS unit and PDA are a potential problem. If a cable connection is to be used you may have to purchase appropriate cables for both GPS unit and PDA separately, and you may have to join them with a male-to-male adapter and null modem connecter as well. See the ArcPad user manual for good instructions on following this route. Duct tape will transform the cables and adapter into a single, stronger, unit. The data-cable fitting on many PDAs is weak. One can mount the PDA in a rigid windowed box with a short data cable that leads from the PDA to a more substantial fitting anchored to the box and worry less about breaking the connector, but you will still be trussed up like Gulliver among the Lilliputians. A wireless connection (e.g. Bluetooth, see above) is better.
Compact flash card GPS units arguably provide the most robust connection. There are no cables, no risk of losing the signal due to distance from the receiver (as there is with Bluetooth devices), and since the GPS unit draws current from the PDA battery, there is no worry of having the unit die before your PDA does. They can also be swapped in and out of the compact flash slot with fewer com port hassles than the Bluetooth units. The main disadvantage with this method is that you are likely to lose the GPS signal as you hold the PDA near your body while entering data.
We use credit-card sized Bluetooth GPS receivers such as that sold by Socket Communications (http://www.socketstore.com/product.asp?sku=2542622). These 16-channel receivers can be bought for less than $230. They weigh 2 ounces and the replaceable and rechargeable battery lasts 9 hours, more than adequate for working out of a vehicle with a cigarette-lighter recharger available. For a long day on foot carry an extra battery or mobile recharger, such as the iPowerPak by Data Nation (http://www.data-nation.com) or Mobile Power Pack by Socket.
In recent tests with a Bluetooth GPS unit and a Bluetooth-ready iPAQ h2200, we have found the Bluetooth Manager to be much improved over older versions, causing us to lean even more strongly toward that system.
ArcPad is a simple GIS program for the PocketPC, optimized for collection and display of map information while mobile. It has fewer symbolization and database capabilities than the average desktop GIS, can be customized to a limited extent using XML and VBScript, and has an excellent GPS interface.
ArcPad can be obtained from ESRI (http://www.esri.com/software/arcpad) for about $500. A free evaluation copy that times out after 20 minutes may be downloaded from the web. GDA will not work with versions of ArcPad earlier than 6.0.1 We are using ArcPad 6.0.3. We have not experimented with this version of GDA on ArcPad 7.
Third party file browser
The file browsing program that comes with the Windows CE, File Explorer, does not, by default, show file extensions. Editing the registry to force file extensions to show is an option available to the advanced user, but it is risky. The safest remedy is to install a third party program that shows extensions. PEFile, by Vieka, is a cheap and reliable solution. Seeing file extensions is important because there are many files associated with GDA which have names that differ only by their extensions. Many can be edited in a text editor on the PDA by temporarily changing the extension to ‘.txt’.
ArcGIS or ArcView will make it easier to define symbolization, such as the colors of dots that correspond to different map units. See the ArcPad manual for advice on this. (Note that you can also do this with a text editor, carefully copying, pasting, and editing the .xml file for the layer. Colors are in RGB, 0..255 gamut for each component. Search for "VALUEMAPRENDERER".)
ArcGIS provides tools for reformatting data for digital base maps. You may need to reproject rasters to the projection of your field project. ArcGIS will export an entire map project to ArcPad. Note that you may have to tile large raster images to avoid the 50MB input-image size limit on the MrSID converter that is shipped with ArcGIS. See the ArcPad documentation.
You will almost certainly need ArcGIS or ArcView to make a completed geologic map from the data you collect with GDA.
Excel (or OpenOffice) or Access
You will need to edit .dbf files that constitute some of the picklists. This is conveniently done on a PC with Microsoft's Excel. The Calc program in OpenOffice.org works equally well. CDBF (google for it) works well. Another solution is to create an Access database containing all of the picklist tables and export them as needed in .dbf format. Let us know if you find a DBF editor that runs on a PDA!
A text editor that works gracefully with plain ASCII files is handy for minor modifications of layer definition (.apl) and VBScript (.vbs) files. On a Windows platform, WordPad and Notepad are fine. Many other editors are available. Avoid using Microsoft Word because it may add hidden and extraneous formatting tags that are not recognized by GDA.
GDA consists of 4 sets of files:
1) The GDA toolbar and events that occur when buttons on the toolbar are tapped are defined by XML and VBScript files GDA_1.0.apa and GDA_1.0.vbs. From left to right, icons when tapped effect the following:
2) GDA modifies certain aspects of ArcPad's behavior with XML and VBScript files ArcPad.apx and ArcPad.vbs.
3) GDA help is provided by HTML help files in folder GDAHelp.
4) GDA data for a mapping project reside in a single user-designated project directory. Multiple mapping projects require multiple project directories. Each project directory contains files for the 4 data layers described below, a collection of .dbf files that define picklists, per-station files of notes and sketches that may be created while using GDA, and an ArcPad.apm file (written by ArcPad each time work is saved) that defines active layers, pathnames, current map extent, and controls some symbolization. Empty copies of the data files are stored in a Templates directory that resides at the same directory level as the different project directories. A mapping project directory will also commonly contain a projection file, a subdirectory with base map layers, a subdirectory with compiled geologic data layers, and archive subdirectories.
GDA adds 4 editable data layers to a project: stations, structure, geolines, and mylar. Most data layers have 5 files: 3 files with extensions shp, dbf, and shx that constitute a shapefile, an ArcPad XML layer-definition file with extension .apl, and--except for the mylar layer--a .vbs file that contains VBScript code called by events defined in the layer definition file.
Stations is the primary point data layer. Within it are recorded all station locations and much of the data associated with them. Adding a new station brings up an edit form that controls almost all point attributes and allows editing of associated tables for Samples, Photos, and the structure layer. Attributes of points in this layer are:
Other pages of the Stations form allow editing of subsidiary tables (dbf files) for Samples and Photos. Fields in these tables and the associated entries in the Stations edit form are shown in the tables below.
With properly configured symbolization (see Symbolization), points in the Stations layer can be shown as dots whose color corresponds to MAPUNIT.
Structure is a point layer for storing and plotting orientation data (bedding, foliation, lineation, etc.) Such data are stored in a layer separate from the Stations layer so that they can be plotted. A point must first be added to the Stations layer before orientation measurements can be recorded. Structure data are added from the Str page of the Stations form (see image).
As each measurement is added to the Structure layer, GDA programmatically copies the coordinates of the station point to the Structure layer. Multiple structure measurements may be added at a single station.
Features of the Structure layer are described in the following table:
The table below shows the contents of StrucTypes_1.0.dbf as it is distributed in GDA. The Text field shows the value that will be stored within the STYPE field. Note that Structure_1.0.apl contains instructions for STYPEs in addition to those listed below; these can be added to StrucTypes_1.0.dbf and will be symbolized. Addition of other values to StrucTypes_1.0.dbf will require that you update the symbology defined in Structure_1.0.apl (see Symbolization).
Geolines is a line layer for contacts, faults, shorelines, and other linear features. Attributes of lines in this layer and their characteristics are:
The table below shows the contents of LineTypes_1.0.dbf as they are distributed GDA. The LINETYPE field shows the value that will be stored in the LTYPE field. If these values are changed or more are added and you wish to have new values symbolized properly, you will have to update the symbology (see Symbolization).
Mylar is a layer of unattributed lines, included to provide flexibility. Use Mylar for map ornamentation, text written directly on the map, or other annotation. Note that in most cases any information in this layer must later be translated into a form that can be queried by machine.
ArcPad allows GDA to use a wide variety of base materials, including previous geologic data. The primary limitation is symbolization. Only one raster layer may be used at a time, as ArcPad does not support transparency.
Establish an ActiveSync connection between the PC and the
Use the default installation folder suggested by the setup wizard, i.e. ‘ C:\Program Files\Arcpad’
On the PC,
Start>Programs>ArcPad>Install ArcPad application for Windows
Both the .bat and .cab files assume you installed ArcPad in the default directory!
Download the appropriate GDA package
for your system and unzip the file.
Choose ‘No’ to any dialogs asking you to retry installation of a font file.
The default locations after installation are shown below. If you have chosen to install ArcPad in a non-default folder, you will have to move the files from the unzipped GDA_PC_Install.zip file manually.
After installation on a PC, move the My
Documents\GDAProjects\Templates folder to another drive and
folder if you wish; the other folders must remain within the ArcPad
After installation on a PDA, move the My
Documents\GDATemplates folder to a non-volatile storage card and make a
copy of it for each project.
Configure PDA to communicate with Bluetooth GPS unit.
These are general instructions for making contact with a Socket Bluetooth GPS unit on an internally enabled bluetooth PDA. If you use a different GPS unit these instructions may not apply.
Make sure the GPS unit is fully
Open ArcPad Options (crossed wrench and hammer: top toolbar, 2nd icon from right, select Options...) and configure Protocol and GPS pages.
With the Socket GPS communicating via internal Bluetooth on an IPAQ H3900, appropriate settings are:
Protocol: NMEA 0183
IMPORTANT: On the Capture page enable averaging. 10 points is a good number. This is necessary for certain VB script routines to be fired properly, but it is also good GPS practice.
Configure other GPS quality parameters and warnings as needed on the Quality, Capture, and Alerts pages. For instance, you can tell ArcPad to reject any GPS fixes with a PDOP (position dilution of precision) greater than 6 (4 is considered excellent and you may routinely see PDOP values better than 3. Never accept a PDOP greater than 8).
IMPORTANT: On the Paths page, set the Default Maps & Data Path to the project folder your data are in.
Save the map as ArcPad.apm within the default data folder. This map will now be opened by default when ArcPad is started.
Check that ArcPad is receiving valid
NMEA strings by going to the GPS dropdown menu and choosing GPS
Debug. If you see nothing, then something is configured
A mapping project has a common geographic area, a common geologic vocabulary, and a single map projection and datum. Many geologists will work on only one project. Others might have several, perhaps including a project mapping glacial deposits in western Washington in the Washington State Plane North feet-NAD83 coordinate system and a project mapping late Cenozoic rocks and deposits in northwest Oregon in the UTM10 meters-NAD27 coordinate system.
For each new project (new area, new projection, and (or) new theme), create a directory on your storage card by making a copy of Templates and renaming it to <NewProj>, where <NewProj> is a relatively short version of the project name. Within <NewProj>, create subdirectories for base layers and compilation geology as you see fit.. With two projects named BremE and Wen, the Storage Card directory tree would be :
ArcPad maintains a default data path (that is, a default
directory). If a map-description file named ArcPad.apm is present in
the default data path, that map will be opened when ArcPad is
For convenience, keep all base materials in a subdirectory <NewProj>/Base. Possible base materials include:
Note that only one raster file can be shown at a time, as ArcPad does not support transparent rasters. If you want to display a raster composite, for example colored unit polygons over shaded relief, you must explicitly create this image in ArcGIS and export it to a suitable image-file format.
Raster images must all be in the same projection. Convert vector layers to the same projection as well; ArcPad won't reproject on the fly.
MrSID compression is desirable, both to create image pyramids and to reduce overall file size. Experiment to find the appropriate degree of compression. We find 10-15x works well. The 50Mb limit for single files input to the MrSID converter in ArcGIS may require that some input files be chopped into smaller tiles.
ArcPad needs to know the project map projection and the GPS cannot be enabled without one.
1) If you export the project from ArcGIS, a projection file (*.prj) should be created. Install this in the project directory.
2) You can put any prj.adf file (from within a coverage or grid – rename it to .prj) or .prj file (associated with a shapefile or from the ArcGIS\Coordinate Systems folder) into the project workspace. When you add a new layer that does not have an explicitly defined projection, you will be asked if you wish to use this projection.
3) You can assign a projection to each map layer with the Map Projection button (top right icon on the Layers dialog) in ArcPad.
ArcPad will not reproject data to a common projection for a
map project. All layers must have the same projection,.
Your choice of projection will usually be forced by the projection of the raster layers (DRGs, shaded-relief images, DOQQs) used for a map base. In a given mapping project, such rasters should all be in the same projection.
You will need to edit Units_1.0.dbf to establish an initial unit vocabulary for a new project. Open a copy of this file in Excel, OpenOffice Calc, Access, or similar, select and delete existing values, add new values, and re-arrange values. Note that "LITH" need not be specified. Save and move the modified copy of the file to the project workspace.
If symbolization is based on any vocabulary values you create or edit, you will need to change the symbology element within the shapefile’s .apl file. You may do this manually from within a text editor if you wish, but it is much easier to use the method outlined below.
You can create symbology for a new project with ArcGIS or ArcView:
Included in the ArcPad install folder are two folders called apTools3 and apTools8. In these folders are extensions with export tools for use in either ArcView 3.x or ArcGIS 8.x (and 9.x).
To install apTools8.dll for ArcGIS:
Add the shapefile for which you want
to create symbology to an ArcMap or ArcView map
Points can be symbolized with any true type font character, lines can be symbolized with different thicknesses and color only, and polygons can be symbolized with solid opaque colors only.
Click on the Export Symbology tool you
To rotate point symbols according to an attribute value (eg Azimuth from the Structure shapefile):
Open the .apl
file for the shapefile you are dealing with and search for
Each time you start ArcPad you will have to re-start communication between the PDA and GPS unit. With WinCE 3.0 and the Socket Bluetooth GPS,
Power on the GPS unit
Once GDA is installed and the first map is saved as \Storage Card\<ProjName>\ArcPad.apm, ArcPad will restart with GDA up and running in the most recent project and map extent.
When we start GDA we routinely make GPS Active, enable GPS Tracklog, and--using the Layers form--make the GPS Tracklog invisible.
Whenever you tap OK on the Station edit form, or tap ADD on the
Structure, Photos, and Samples pages, data are written to memory. You
need not tap the Save icon on the top toolbar.
Data not written to files
We have had problems with data not being written to dbf files. We think these problems have occurred because of mixed versions (e.g., an old .apl file and a newer .dbf format) or corrupt XML in a layer definition (.apl) file.
If GDA is properly configured and you pay attention, failure to
record data will be evident immediately. (A) Set up Station point
symbolization to reflect MAPUNIT. Are the dots the right color?
(B) Using the layer dialog, label each station point with MAPUNIT or
the like. If you don't see MAPUNIT written beside a new station, data
were not recorded. (C) Can you see structure symbols? If not, why?
(D) Label each structure point with its INC.
At intervals, copy all data files to an archive folder, copy this folder to a PC, and burn it to CD-ROM for safekeeping. Data files include
It may be useful to put a copy of Units_1.0.dbf in the archive folder, to better understand the MAPUNIT and LITH vocabularies that were suggested to the mapper. It will also be useful to put copies of all digital photographs into the archive folder. We suggest naming the archive folder with the geologists' initials and date of archiving, (a truncated version of a station ID), such as 06_0322ET. The directory structure of the PDA storage card may then look like this:
Here are two strategies for archiving, each with its advantages
and disadvantages: (1) You can COPY data files to the new
archive folder. Grow a single, evolving set of data files for the
project. Archive folders grow steadily larger and each archive folder
supplants its predecessor. At any time, a single set of shapefiles
encompasses all of the your data for the project. An advantage of
this approach is that earlier observations are readily accessible.
Turn GDA databases into a narrative
The Python script gda2html.py (located in the Utilities folder) will turn a GDA project folder, or an archive folder, into a narrative in HTML format that is easily read, edited, and printed. Most computers with ArcGIS have Python available as part of the ArcGIS installation. To use this script you will have to locate and install modules dbf and image; see instructions within file gda2html.py.
To move data into a desktop GIS, we have found it simplest to archive a project, then copy the entire archive directory to the computer that hosts the GIS. The archive directory on the PDA may then be deleted, or left for reference during further mapping. If the working directory on the PDA is on a storage card, you can temporarily mount the storage card on the GIS host for fast drag-and-drop transfer.
How GDA data are incorporated into a compilation project in the desktop GIS will depend strongly on the data structure of the desktop project. We suggest these guidelines:
We wrote GDA in the course of our employment with the U.S. Geological Survey. It is not subject to US copyright. You have an unrestricted right to modify and redistribute GDA code. However, we request that, as a matter of courtesy:
Customize with a DBF editor
During use--perhaps on a daily basis--you may find that Units_1.0.dbf should be edited for clarity and performance. We particularly find it useful to edit values in the LITH field, reduce the number of rows that correspond to differing values of LITH, and re-order rows. StrucTypes_1.0.dbf, SampType.dbf, and Geologists_1.0.dbf can be similarly edited.
Customize with ArcView or ArcGIS`
For new projects, and as projects evolve, symbolization will have to be created and modified. This can be done on a PC with ArcView or ArcGIS, and new symbology exported to layer definition (.apl) files that are then copied to the PDA. See Symbolization, above.
Customize by editing XML and VBS files with a text editor
All of GDA's behavior can be modified by editing the appropriate XML and VBS files. Use a text editor that works gracefully with plain ASCII, not Microsoft Word! Modifying VBS files requires knowledge of VBS and a clear understanding of the internals of GDA. We don't recommend this for most GDA users.
Modifying XML files to change picklists, to change vocabularies from open to closed, and to modify some symbolization is relatively straightforward. Use existing XML as a guide to format. Copy, paste, and modify. Experiment! Be sure to keep an unmodified copy. To test for corrupt XML, open the modified file with a browser such as Firefox (Microsoft's Internet Explorer 6 doesn't display XML). If the file will not display, it probably won't work.
Note that if you modify symbolization by directly editing the layer definition (.apl) file, the EXACT statements within VALUEMAPRENDERER must be in strict ASCII order, followed by a single OTHER statement. If the EXACT statements are not properly ordered, the symbolization will not work as you expect.
Customize using ArcPad Application Builder
If you wish to significantly modify forms, change or add attributes, or add new tables, we suggest working with ArcPad Application Builder. This is expensive. We found that the $1,500 cost of Application Builder is cheaper than the time required to learn VBScript and the innards of ArcPad. Expect to spend a lot of time reading ESRI's User Forums.
Haugerud, R.A., and Thoms, E.E., 2004, Geologic Data Assistant (GDA): an ArcPad extension for geologic mapping , in Soller, D.R., editor, Digital Mapping Techniques '04: U.S. Geological Survey Open-File Report 04-1451, http://pubs.usgs.gov/of/2004/1451/haugerud/index.html.
For questions about the content of this report, contact Evan ThomsVersion history
Download a current version of Adobe Reader for free
This report is available only on the Web