Background

3DEP is a partnership program with State, local, and Federal partners that is managed by the USGS to respond to growing needs for high-quality topographic data and for a wide range of other 3D representations of the natural and constructed features in the United States. 3DEP informs critical decisions that are made across the Nation every day, which depend on elevation data ranging from immediate safety of life, property, and the environment to long-term planning for infrastructure projects.

NAIP acquires aerial imagery during the agricultural growing seasons in the continental United States. This “leaf-on” imagery is used as a base layer for geographic information system programs in the Farm Services Agency’s County Service Centers and is used to maintain the Common Land Unit boundaries. Because the airborne lidar sensor system analyzed is a hybrid sensor that collects imagery and lidar data simultaneously, it has the potential to collect data that satisfies USGS 3DEP and USDA NAIP requirements in a single collection.

This report presents accuracy assessment results of the light detection and ranging (lidar) data collected in Colorado during a pilot project in fall 2019 and supplements the work published in Kim and others (2020). The focus of this report is on the geometric quality and characteristics of the lidar data.

Data Acquisition

Hexagon acquired data for this project using the CountryMapper sensor system operating at an altitude of 3.6 kilometers (km), a speed of 180 knots, a pulse repetition frequency of 750 kilohertz (KHz) and a scan rate of 112 Hz. The resulting imagery product was 20 centimeters (cm) ground sample distance ortho imagery and a pulse density of 2.9 points per square meter (ppsm). The elliptical scan pattern of the airborne lidar sensor system allowed for the separation of the point cloud into forward- and backward-looking scanned groups, which enabled intraswath difference analysis.

Real time kinematic (RTK) global navigational satellite system (GNSS), total station, ground based lidar, unoccupied aerial system (UAS) lidar, and UAS imagery data were collected to compare to the data collected by Hexagon using the airborne lidar sensor system. USGS collected these data using survey-grade GNSS and total station instruments along with a terrestrial laser scanner (TLS). In the RTK method, a fixed base station (base station) occupying either a known location, or a random location that will be post processed to known coordinates, transmits corrections to a moving global positioning system (GPS) receiver (rover). The rover uses these corrections to greatly enhance its positional accuracy, achieving centimeter-level precision (Van Sickle, 2008). The base station data and the rover data were post processed after 16 days, when precise ephemeris of the GNSS satellites were available (Soler and others, 2011). The ground-based lidar is operated by strategically positioning the lidar scanner to ensure full coverage of the target area. These scans from multiple locations are necessary for complete coverage of most targets or those with obstructions. The lidar systematically emits laser pulses and records the return time and intensity of the reflected light, capturing millions of data points. Reflective targets are placed in the region of data collection, and their positions are surveyed using a total station that utilized control established via RTK methods. These targets are used to georeference the lidar data scans and improve the accuracy of merging multiple scans (Bethel and others, 2005). The GNSS and total station data include points collected on ground surfaces, roof planes, infrastructure features, TLS georeferencing spheres, and individual trees. The total station is especially useful in acquiring ground points under tree canopy, where GNSS solutions are difficult to establish. The TLS data include scans of buildings, parking lots, and groups of trees. USGS also collected UAS data. All survey data, including the UAS data, are available at https://doi.org/10.5066/P9CPDWUU (Irwin and others, 2020).

Intraswath Analysis and Lidar Density Quantification

The density and the intraswath errors were computed by manually delineating eight polygons (using ArcGIS version 10.8.1 and LP360 software version 2021.1.47.0) in the data. The number of points that fall within the polygon and the area of the polygon are used to compute the density. These polygonal regions were carefully chosen to be as planar as possible (for example, roads known to be smooth, building roofs, parking lots, and so on; fig. 3).

The observed density in the data is shown in figure 4. The regions were selected across the dataset, and the density ranges from 3.1 to 3.7 ppsm, which is better than the requirement for QL2 data as mentioned in table 1 (fig. 4). A couple of regions were chosen in areas of overlapping swaths and where the point density is proportionally (to the number of swaths) higher.

For intraswath analysis, the precision was measured by fitting a plane through the points inside the manually delineated polygons and then calculating the root mean square error (RMSE) of the planar fit. Generally, the intraswath precision values were very high, better than 1 cm in most cases (fig. 5).

Interswath Analysis

Figure 6 shows a profile of a hypothetical surface that falls in the overlapping region of two adjacent swaths. The surface as defined by the swaths is shown in lighter dotted lines, whereas the solid profile represents the actual surface. A poorly calibrated system leads to at least two kinds of errors in lidar data: (1) the same surface is defined in two (slightly) different ways (relative or internal error) by different swaths, and (2) the deviation from actual surface (absolute error). The calibration procedures are of less concern than the final quality of data; however, a process to test the quality of instrument calibration is needed, because a well-calibrated instrument is a necessary condition for high-quality data. Therefore, the interswath error (along with the intraswath error) is an important direct measure of the quality of instrument calibration.

The interswath analysis was done by using the American Society for Photogrammetry and Remote Sensing (ASPRS) Data Quality Metrics (DQM; Stensaas and others, 2018) tool to analyze the swath-to-swath overlapping regions. In this method, data in each tile is split into their component swaths. The ASPRS DQM method implements the concept of point-to-plane data quality measures over natural surfaces. The prototype works on ASPRS’s LASER file exchange format (LAS) files containing swath data. If the swaths are termed Swath 1 and Swath 2 (fig. 7), the software uniformly samples single return points in Swath 1 and then chooses “n” (user input) points. The neighbors of these “n” points (single return points) in Swath 2 are determined.

A least squares plane is fit through the neighboring points using eigenvalue/eigenvector analysis in a manner like principal component analysis. The equation of the planes is the same as the component corresponding to the least of the principal components. The eigenvalue/eigenvector analysis provides the planar equations as well as the RMSE of the plane fit. The use of single return points in conjunction with a low threshold for RMSE is used to eliminate sample measurements from nonhard surfaces (such as trees). The DQM software calculates the offset of the point (say “p”) in Swath 1 to the least squares plane. The output includes the offset distance as well as the slope and aspect of the surface as implied in the planar parameters.

The software was used to calculate the interswath differences on a per tile basis in the western AOI and the eastern AOI. The results are graphically shown in figure 8 and summarized in table 2. A total of 780 tiles of lidar data in the western AOI and 188 tiles of lidar data in the eastern AOI were tested for this analysis.

In most tiles (99 percent), the root mean square difference values measured were 0.04 m or less for the western AOI and 0.026 cm or less for the eastern AOI. These values are well within the QL2 requirements of the LBS as stated in table 1 and, together with the analysis in the previous section (and figs. 2 and 3), indicate that the data are well calibrated.

Absolute Geometric Accuracy Assessment

The absolute accuracy of airborne lidar data is divided into quantitative and semiquantitative sections. The measurements shown in semiquantitative sections (such as comparison of tree heights and comparison of locations of lane markings) have subjective components to them; however, we consider them to be important metrics towards understanding of the accuracy of the data.

Semiquantitative Measurements and Analysis: Tree Heights

The assessment survey involved collecting total station data as well as TLS data for trees to compare the tree heights obtained from total station and TLS to the airborne data. A UAS-based lidar sensor was also flown to collect lidar data that were compared to the airborne lidar data (fig. 9).

Individual trees were selected in the field, and total station points were collected along its trunk up to the highest observed leaf or branch. It is acknowledged that the ability of the total station (and TLS) to accurately obtain tree heights is dependent on the location of total station (or TLS) instrument as well as obstructions; therefore, care was taken to select the instrument locations such that the best view of the selected trees was available in areas of high-density trees. It may not be possible to collect data on the top of trees using a TLS or total station if the line of site is obstructed (fig. 10).

Three elevation measurements for each tree were noted: the highest elevation from the total station measurements, the highest elevation from the TLS points, and the highest elevation point of the airborne lidar sensor system. The differences among the airborne lidar data and measurements from the total station as well as TLS were noted. When compared to the elevation of trees obtained from total station data, the airborne lidar data generally were lower by 0.33 to 1.25 m in the BLM plots (table 3). This result has been documented in Andersen and others, 2006. Other studies have determined that the consistency between tree heights is higher if the systems are optimized for capturing vegetation as opposed to bare earth (Wang and others, 2019). The difference between total station- and TLS-based measurements is high in the USFS plot for two possible reasons: (1) the higher density of trees (in USFS plots as compared to BLM plots) can make it difficult for total station-based measurements to capture the top of trees, and (2) the TLS measurements were made from multiple stations, which can increase the chances of capturing top of trees.

Quantitative Measurements and Analysis: Vertical Accuracy and 3D Accuracy

Assessing absolute accuracy in rural areas with few well-defined features that can provide reliable horizontal coordinate measurements is difficult and this assessment is therefore limited to only a vertical accuracy assessment. In this survey, check points were surveyed in clear and open areas as well as in vegetated areas such as in tall grass and shrubs and below tree canopy. The vertical accuracy measurements presented here are measured against only those points classified as ground in the final airborne lidar data. The LP360 software as well as the DQM software were used to derive the measurements. The results presented here are from the LP360, although the DQM software measurements were comparable. Figure 11 shows examples from two locations in the western AOI.

Table 4 lists the summary of measurements from western AOI survey locations. Only the Windy Gap wildlife area measurements can be considered ideal for nonvegetated vertical accuracy assessment because those were made over concrete or asphalt surfaces in clear and open areas. Most other locations had substantial cover of shrubs or trees (in the case of USFS plot). The measurements indicate the airborne lidar data are within the specifications for QL2. Table 5 lists the vertical measurements of site surveyed in the eastern AOI. The RMSE ranges from 0.028 to 0.035 m. Tables 4 and 5 indicate that in clear and open areas, the airborne lidar data are well within the requirements specified in the LBS (less than 0.10 m RMSE from table 1) for open areas as well as in vegetated areas. The USFS locations were not considered clear and open as they had substantial tree cover.

Horizontal Accuracy Assessment

The horizontal component of accuracy for 3DEP lidar data is not explicitly verified in most 3DEP projects collected under Geospatial Products and Services Contracts. The LBS does not provide quality thresholds or methods for verification of data; therefore, the methods presented in this section are forward-looking research efforts to develop operational methods for verifying 3D accuracy (horizontal and vertical) in future versions of the LBS.

In this report, we present three methods to verify the horizontal component of the airborne lidar data accuracy:

The three-dimensional image from the TLS scan is colorized using the red, green, and blue values obtained from the camera collocated with the TLS. This TLS method used manual building roof extraction and comparison of plane intersections using methods described in Kim and others (2020). The total station method used automatic estimation of shift in data (between reference total station and airborne lidar) using methods described in Stensaas and others (2018). The background image for the lidar intensity method was derived from the airborne lidar’s intensity data collected over a parking lot.

Terrestrial Laser Scanner Method

The first method, which is described in Kim and others (2019), uses TLS lidar data from preselected building roof planes. The roof planes were manually delineated and mathematically modeled. The intersection points of these roof planes were calculated. The same process (of manually extracting and mathematically modeling the roof planes) was repeated on conjugate roof planes from the airborne lidar data, and the intersection points of the various roof facets were calculated. The 3D accuracy of the points was assessed by comparing the two sets of coordinates. This process was repeated for multiple buildings across the eastern AOI. A detailed report of the procedures, mathematical modeling, and analysis is in Kim and others (2020).

The results from the analysis indicate that the airborne lidar is translated by 0.22 m (standard deviation [s.d.]: 0.033 m) in easting, 0.074 m (s.d.: 0.096 m) in northing and 0.034 m (s.d.: 0.016 m) in elevation (Kim and others, 2020).

Total Station Method

The second method explored was using total station points obtained by carefully pointing the total station viewer to the planes of roofs and measuring points (coordinates) on the surface in reflectorless mode. These points are considered reference check points. This method explores an automated solution to determine the discrepancy in geometric positioning between reference data and airborne lidar data. In figure 13, the check points measured by the total station are notionally illustrated as red dots, and the airborne lidar points illustrated on roof planes are in blue. A two-dimensional neighborhood of points is selected around the total station check point. Since these points are carefully curated during data collection, there is high confidence that the two-dimensional neighborhood of points lie on the roof plane. The equation of the plane is determined by fitting a plane to this neighborhood of points, and the distance of the plane from the check points are calculated. This process is repeated for all the check points collected. The method assumes that the geometric discrepancy between the airborne lidar and the total station points, at the scale of a building or small neighborhood, can be estimated by a single shift (∆; for example, ∆X, ∆Y, ∆Z). The step-by-step process is below:

Table 6 contains the results of four sites using this method, and appendix 1 table 1.1 shows the measurements in more detail. The measurements shift is estimated by taking the pseudo inverse of the matrix formed by the columns (Nx, Ny, Nz, which represent the equations of the plane) and multiplying it with the corresponding point-to-plane distance vector column.

The average 3D shift between the airborne lidar data and the total station data were determined to be (0.19 m, 0.078 m, 0.057 m) with this method. Two measurements involved roof planes from multiple houses or structures.

Lidar Intensity Method

The third method (fig. 14) for observing horizontal errors in the data involved using check points collected on visible parking lot or road markings on freshly paved surfaces. This method is the most common method used in the industry (when possible) and is analogous to the data quality assessment methods used for aerial and satellite imagery. Because the LBS do not require horizontal accuracy to be reported with the 3DEP data delivery, these measurements are usually made by the data vendors for their internal processes and not externally reported.

The airborne lidar intensity data were used to identify parking lot line intersections. These check points were selected before the survey in the intensity airborne lidar data, and GNSS (real-time kinematic) methods were used to measure the same check points. The measured differences were noted in only one parking lot and were on an average (0.15 m, 0.06 m, and 0.04 m) in easting, northing, and elevation, respectively. The lower density of QL2 (2 ppsm) precludes a precise location of such intersections. They also require manual delineation; however, with increasing point density (QL1 and QL0 or higher), such methods could offer the least expensive mode of precision testing the 3D accuracy of data, and established image processing toolsets may be able to extract more precise locations of such check points from denser lidar datasets.