Joint Agency Commercial Imagery Evaluation History

The Joint Agency Commercial Imagery Evaluation (JACIE) partnership was established in 2001 to better understand the characterization and calibration of space-borne remote sensing. JACIE member agencies include the National Aeronautics and Space Administration (NASA), National Geospatial-Intelligence Agency, National Oceanic and Atmospheric Administration (NOAA), U.S. Department of Agriculture, National Reconnaissance Office, and U.S. Geological Survey (USGS).

JACIE represents the U.S. Government’s commitment to using remote sensing technology to provide information for critical science needs and other applications. The USGS website for JACIE includes information on past, current, and future JACIE efforts and can be accessed at https://www.usgs.gov/calval/jacie.

JACIE fulfills its mission through a unique interagency team of scientists and engineers who evaluate the quality and utility of commercial and civil government remote sensing data. JACIE’s role has evolved in response to changes in technologies, methods, and needs. As a result, team members define metrics, prioritize requirements, and assess and assign quality measures for image datasets acquired by the U.S. Government.

In addition, extended relationships are established via JACIE that span civil and military organizations and include national and international government, academia, and industry. The acronym JACIE can refer specifically to the Government partnerships or it can refer to these extended relationships. The acronym JACIE used in the remainder of this document is the latter.

JACIE sponsors an annual workshop to exchange information on the characterization and application of commercial and civil government imagery to all resolutions used by the Government. The event is a face-to-face opportunity for exchanging information.

This report describes common methods JACIE members use to assess data quality. It serves as a model for documenting best practices for how the Government can characterize and assess the quality of remote sensing datasets acquired in support of its needs.

Importance of Data Quality

The importance of calibration and validation of space and aerial remote sensing systems lies in ensuring the quality and integrity of the acquired data and derived products. Methods of calibration and validation are well-documented as a requirement for data acquisition by government and industry. Sensors must be calibrated to provide measurements with the accuracy and precision to meet scientific and other needs.

In the remote sensing world, this need of calibration continues to grow. The calibration of analog film cameras and the resulting “Camera Calibration Report” (Boland and others, 2000) has been known as a requirement for any mapping, engineering, and science work based on the photography acquired by these cameras. From the 1950s until 2019, the National Bureau of Standards, followed by the USGS, operated and provided calibration services and issued these standard reports for the United States and other countries. Although providing a laboratory calibration service for satellites is not feasible, a similar means of ensuring data quality is needed for Earth observation (EO) data from orbit. This need is increased by the speed at which remote sensing, as a technology as well as a practice, is advancing. More data and data products are being produced because it has become easier to build and operate EO satellites, as well as produce data products and perform large-scale analyses.

This data proliferation is spurred by the convergence of technological and economic factors, which include lower cost launch vehicles, higher speed data downloads, high-speed processing at low cost, practically unlimited data storage and archiving capacities, and ease of access to acquire data. These changes are best represented by two substantial outcomes: (1) the rise of low-priced and prolific small satellites (also known as “smallsats”) providing high-resolution and around-the-clock remote sensing data, and (2) the emergence of commercial remote sensing satellite corporations that operate these satellites and provide the data to Government users. The importance of JACIE and a better understanding of remote sensing systems is driven by the sheer number of remote sensing systems, amount of associated data, development of interoperable/integrated products, and many enhanced science-use opportunities.

To enable the use of datasets offered by the growing number of commercial and civil data providers, the products of sensing systems must be calibrated and validated, and their measurement accuracy must be characterized and documented. Such documentation is a critical first step in ensuring products meet user requirements of acceptable data quality. The need and ability to establish a remote sensing baseline that provides calibration to international reference standards, information documentation and continuity, and product validation and monitoring is common for all remote sensing systems. However, the key steps necessary for science quality use are not performed for many sensors because of a lack of knowledge about what is needed and (or) the associated cost.

The push to combine remote sensing inputs to support science has been happening for many years. That effort is now combined with an information revolution that enhances the ability to store, access, and evaluate metadata; and import, use, or create near-real time processing algorithms and high-speed processing. The result is that needed science information can be integrated with the appropriate accuracy and uncertainty levels to meet fit-for-purpose science usability requirements. This push drives the need for JACIE’s best practices report and for their continued enhancement to provide uncertainty information in support of information interoperability.

This document is informed by many years of knowledge from Government civil systems, including Landsat and the Moderate Resolution Imaging Spectroradiometer (MODIS), and their quality baselines and processes. Subsequent sections provide processes to utilize and compare with these well-calibrated reference systems and international standards, as well as provide a reference to enhance data and product quality with an eventual goal of establishing interoperability quality levels for various data streams to meet higher level science product development and application use.

Plan for Component Performance Reassessments

One of the lessons learned at NIST in previous interactions with NASA and NOAA is that some of the sensor data problems on-orbit could not be isolated fully because no duplicates or even samples of components were available for re-examination. Duplicates of filters, apertures, mirror samples, diffusers, and so forth, are valuable to have for re-examination at metrology laboratories where high-accuracy data can be obtained by simulating on-orbit conditions to sort out data discrepancies.

For example, the band edge wavelength of filter transmission is temperature-dependent and it could be re-measured to understand on-orbit data. At NOAA, in the case of Geostationary Operational Environmental Satellite (GOES) sounder on GOES–N and high-resolution infrared sounder on polar operational environmental satellites, a large discrepancy—as high as 6 degrees Kelvin—was observed between the measured radiance of an on-orbit blackbody and that radiance calculated using the pre-launch vendor-supplied spectral response function (SRF) of the sensor. This discrepancy affected the on-orbit product retrieval and assimilation of numerical weather prediction models because the atmospheric quantity of interest is determined by varying it to make calculated radiances match with observed atmospheric radiances.

The calculated radiance is essentially a convolution of the SRF with the monochromatic radiances from radiative transfer computation. Therefore, as a first step, NOAA employed NIST to make independent measurements of SRFs of witness samples of filters of on-orbit GOES sounders. In the affected channels of GOES–8 and GOES–10 sounders, NIST measurements done at the on-orbit operational temperature conditions disagreed with SRFs in use by NOAA and were determined to be more consistent with on-orbit radiance observations at known blackbody temperatures, thus explaining the possible discrepancy (American Society for Photogrammetry and Remote Sensing, 2014).

A similar investigation was completed on high-resolution infrared sounder filters to compare vendor measurements and NIST measurements (Cao and Ciren, 2004). Again, there were noticeable discrepancies, and the NOAA analysis showed such discrepancies affect product retrievals and their inferences on weather prediction models. As a lesson learned from this interaction, it is essential to have SRFs measured at simulated on-orbit operating conditions and they should be independently verified with authentic witness samples.

One simple best practice based on lessons learned in NASA missions is that each satellite mission should have duplicates of critical components of their radiometric instruments for future on-orbit data reassessments. Furthermore, this best practice guideline could be extended to the manufacture of key instrument subsystems within specific filters and to have instrument representatives in the manufacturing facility to monitor component testing and acquire authentic duplicates of space hardware.

Post-Launch and Pre-Launch Validation and Traceability

Post-launch. Part of the overarching principles advocated by the workshop report (Datla and others, 2011) for high-quality climate observations is to arrange for production and analysis of each climate data record independently by at least two sources. It goes on to state, “Not only instruments, but also analysis algorithms and code require validation and independent confirmation” (Datla and others, 2011, p. 630). This requirement is because confidence in the quantitative value of a geophysical parameter will be achieved only when different systems and different techniques produce the same value (within their combined measurement uncertainties). Comparisons of the results by different sources are expected to reveal the origin of the flaw to be corrected by their advocates to improve the confidence of their systems and techniques to produce high-quality data.

This process is broadly called “validation” and is defined by the Working Group on Calibration and Validation (WGCV) of the Committee on Earth Observation Satellites (CEOS). Traditionally the process is completed post-launch through ground “truth” campaigns where different satellite systems compare their measurements on ground “truth” sites. This validation process requires that the ground “truth” system has proved capable of credible datasets. Post-launch validation should be planned using land sites of known radiometric characterization.

For instruments such as the Advanced Very High-Resolution Radiometer, the ground “truth” sites essentially provide what is called “vicarious calibration.” For satellite sensors such as SeaWiFS, the instrument calibration that began in the laboratory is continuing through vicarious calibration by comparison of data retrievals to in-water, ship, and airborne sensors to adjust instrument gains. The WGCV of CEOS is identifying suitable sites and their characteristics for on-orbit sensor validations and vicarious calibrations for sensors worldwide (Committee on Earth Observation Satellites, undated). CEOS WGCV members are working with NMIs such as NIST in the United States and the National Physical Laboratory in the United Kingdom in this effort. One site selected is the moon, as an on-orbit stability monitor for the visible and near infrared spectral region up to 2.5 micrometers. The SeaWiFS and MODIS sensors currently on-orbit have been successfully viewing the moon as a stability monitor. One of the recommendations of the Achieving Satellite Instrument Calibration for Climate Change workshop is that necessary lunar observations be carried out to make the moon an SI traceable absolute source for on-orbit satellite calibrations.

Programs at NASA and NOAA provide high-altitude aerial platforms with radiometrically calibrated sensors for validation of satellite sensor data by simultaneously observing the satellite sensor footprint of Earth’s atmosphere (for example, the University of Wisconsin Scanning Hyperspectral Imaging Spectrometer; Federal Geographic Data Committee, 1998).

Pre-launch. In following the various steps of best practice in figure 1, new research methods and other advancements in technology can be investigated to help improve the uncertainties.

Although not yet proven for satellite sensor pre-launch validation activity, it is becoming possible in the laboratory to project special scenes that are radiometrically calibrated, which is achieved by using a light source and a digital micromirror device to project a scene of interest (Datla and others, 2011). The spectral engine and the spatial engine are two digital micromirror device projectors independently illuminated by light sources and controlled to project the appropriate combination of the spectral features and the spatial features. Algorithms are used to develop appropriate combination of basis spectra to project the real scene of interest.

High-quality image data could be projected to the sensor, and thus preflight validation could be emulated with on-orbit sensor data samples. As the accuracy of this validation equipment improves, such an exercise could help evaluate the sensor performance more realistically.

On-Orbit Intercomparisons and International System of Units Traceability

It is best to have intercomparisons of similar sensors on-orbit to assess consistency in data products and sensor performance. Such intercomparisons are possible when both sensors being intercompared are SI traceable on-orbit. Intercomparison of on-orbit sensors has become possible with the technique of simultaneous nadir overpass (SNO) (Gil and others, 2020): when both satellites observe the same footprint at nearly the same time, they cross each other in their orbits.

To bring self-consistency and intercalibration among sensors, a group called Global Space-based Inter-Calibration System formed and is actively pursuing SI traceability for intercalibrations working with NMIs such as NIST. Intercomparisons can find agreement or disagreement between sensors in their radiance measurements. In either case, lessons may be learned on possible systematic effects that are currently ignored or neglected. Because the true value of the measurement on-orbit will always be an unknown quantity, the accuracy of the measurement can best be assessed by combining the results from different sensors and calculating the uncertainty of that combined reference value based on the individual sensor data. The combined reference value and the estimate of its uncertainty in the time series would allow scientists to investigate methods to minimize uncertainty.

Basic Satellite Specifications and Operating Information

Basic information about a remote sensing satellite provides an essential understanding of the system’s capabilities and expected performance. This information provides the general operating capabilities of a system and the data it generates.

During development, this basic information is often referred to as the system specifications, or the intended operational capabilities of a system. Following launch and the commencement of operations, these specifications should be updated by actual performance specifications. It is important that planned performance is updated with actual on-orbit performance because these are the performance specifications of the system that develop the remotely sensed measurements provided to users.

Basic System Specifications

A description of the core systems specifications, as well as the expected measurement units, are described in tables 1 and 2.

Internal Relative Band Registration

Internal band registration measures how accurately the various spectral bands are aligned to each other. The band registration assessment is typically accomplished by selecting a reference band and measuring the alignment of other bands to this reference band. Alternately, each band can be registered to every other band, if computationally feasible. The assessment provides a numerical evaluation of the accuracy of the band registration within an image. That action is carried out by using automated cross-correlation techniques between the bands to be assessed.

Automated Grayscale Cross-Correlation

When applying automated cross-correlation during band assessment, the reference image is taken to be the image data from one of the bands in the multispectral image product; the rest of the bands are considered search images. This process is repeated cyclically by considering image data from a different band to be the reference image. In this manner, data from each band are analyzed against data from every other band.

In cases where the bands within an image have different resolutions, the reference or the search image is resampled to match the lowest resolution of the two bands. Typically, cubic convolution resampling method is used to resample the data. The band characterization analysis measures the relative alignment of the bands to each other by fitting the residuals from all the band combinations using a least squares method.

External Geometry Assessment

External geometry assessment evaluates the absolute positional accuracy of the image products with respect to ground (geometric) reference. The geodetic ground reference is chosen to have known positional accuracy and internal geometric consistencies, which are typically better than the images being evaluated. Measurement and reporting methods are described in the following subsections.

Automated Cross-Correlation

Automated methods, which can be used for coarse to high resolution images, use statistical techniques for error analysis, whereby conjugate points in the reference and search images are identified automatically and refined using similarity measures, such as normalized cross-correlation metrics. Small subimages are then extracted around the conjugate point locations, and image-based correlation methods are used for comparisons.

Ground coordinates of features from the reference image are compared with the corresponding coordinates obtained from the search image. This process measures the relative accuracy of the search image with respect to the reference image. Statistical summaries, such as mean and standard deviation of the residuals from all the conjugate points, provide a measure of the relative geometric offset between the reference and search images. As shown in figure 3, plotting the points measured between the two images also helps to assess any systematic bias or higher order distortion within the search image.

Geometric Accuracy Reporting Practices

There are many reporting methods for geometric data quality. According to the Federal Geographic Data Committee (FGDC), reporting should indicate the characteristics listed in table 3.

The American Society for Photogrammetry and Remote Sensing has built on FGDC reporting standards (Federal Geographic Data Committee, 2010) to introduce geometric reporting standards (American Society for Photogrammetry and Remote Sensing, 2014). These standards tend to be used by the mapping industry, but the methods remain generic and applicable to JACIE reporting practices as well.

The FGDC developed and approved the National Standards for Spatial Data Accuracy (NSSDA) in 1998. The NSSDA describes a way to measure and report positional accuracy of features in a geographic dataset. Generally, the following steps are taken while evaluating a geographic dataset based on the NSSDA (Federal Geographic Data Committee, 1998) and are detailed in Bruegge and others (1999):

In general, the USGS Earth Resources Observation and Science Calibration and Validation Center of Excellence (ECCOE) has reported the circular RMSEr statistic, along with mean errors. The NSSDA statistic follows trivially from the RMSEr statistic as a single multiplication factor. A link to sample worksheets to perform these calculations for three-dimensional data is provided in appendix 1. For image data assessment that does not include elevation information, the columns regarding elevation or “Z” statistics can be ignored. When evaluating digital elevation models where only the elevation values are of chief concern, the X and Y statistic columns can be ignored. In that case, the NSSDA statistic is different when compared to the horizontal accuracy NSSDA statistic (appendix 1)

Spatial Domain Measures

As mentioned previously, image sharpness can be defined using spatial data. When spatial data are used, the assessment is said to be in the spatial domain. Measurement and reporting methods are described in the following subsections.

Edge Spread Function

Edges are present in many scenes in urban and rural areas and may be used as targets. The spatial quality using edges can be quantified by measuring the rate of transition of intensity across an edge. This can be accomplished by generating the edge spread function (ESF), as shown in figure 4, or by normalized edge response where the intensity has been normalized between 0 and 1. USGS maintains a list of spatial calibration and validation sites at the following location: https://calval.cr.usgs.gov/apps/spatialsites_catalog.

An edge target, which can be automatically or manually selected, is shown in figure 4. The ESF is generated by aligning the transects and normalizing the values to one.

Ideally, if the goal is to understand the quality of the imaging system, the process should be applied to raw unprocessed images. However, our analysis is done on processed and even resampled products because these products are most preferred by the user community.

• • Ideally, edges are at least 10 pixels long, although results can sometimes be gathered on edges as small as five pixels in width. The extent of homogenous areas on each side of the edge should also be five or more pixels deep from the edge.

• • Edges used to measure relative edge response (RER) should be inclined, but not at 45 degrees; vertical or horizontal edges are not useful because they do not provide enough range of edge intersections to create a good understanding of edge response.

• • MTF targets as shown in table 4, if sufficiently wide, can be used to estimate SNR if their homogeneity is known (refer to “Signal-to-Noise Ratio” section).

Using the normalized edge response, three quantities are further calculated: RER, the full width half maximum (FWHM), and the MTF. The RER is defined as the slope of the ESF between plus or minus 0.5 pixel from the center of the edge. The FWHM is defined from the line spread function (LSF).

Image Quality Estimation Tool

The image quality estimation (IQE) tool (Innovative Imaging and Research, 2018) is a spatial analysis tool that was developed for the USGS under contract. IQE can estimate the spatial resolution performance of an image product by finding, selecting, and using uniform, high-contrast edges that normally occur within many acquired scenes such as field boundaries and straight roads. Large numbers of images containing many edges can be evaluated rapidly by using this tool. By using IQE, benefits include trending analysis of image performance, automated image quality assessment, and quantification of image sharpness parameters.

A graphical user interface is used for IQE to ease use and provide performance consistency. This interface provides an easy-to-understand method to process large quantities of data and to visualize spatial quality summary statistics. Instead of using traditional edge targets, IQE uses features present within aerial and satellite images. Therefore, for the IQE to be successful, the imagery must contain features consistent with the GSD of the imager being assessed. As an example, if Landsat 8 or 9 data (U.S. Geological Survey, 2021) are being processed, features large enough to create a near horizontal or near vertical edge that is approximately 360 meters need to be present in the imagery (at least 12 pixels x 30 meters GSD). For this case, large uniform agricultural fields, such as those frequently found in the midwestern United States, would be appropriate.

Note that the IQE tool requires some inclination to the edges that are used, and that perfectly north to south or east to west edges will not work for analysis.

Frequency Domain Measures

When spatial frequency data are used, the assessment is said to be in the frequency domain. Frequency domain measures are typically used in assembly and integration to measure system and subsystem performance. These measures are more difficult to use on-orbit.

Linear Regression

The radiometric characterization requires accurate geometric alignment between two images from different sensors. Usually the georeferencing of different satellite sensors is performed independently; thus, it is not rare for the relative georeferencing error to be substantial. Effectively, the comparison results in the juxtaposition of different pixels, which results in worse than true comparison. Each plot in figure 5 is a scatter plot between two sensors for each band (typical blue, green, red, and near infrared band). For georeferencing to be accurate, a pixel located in a fixed location on the Earth’s surface within one image must match the same pixel in the other image. However, if there is mismatch in georeferencing between two sensors, the scatter plot indicates much wider distribution as shown in figure 5. Accordingly, the regression line is substantially different compared to the geo-corrected scatter plot (fig. 6), which demonstrates the importance of relative georeferencing correction.

One of the most common surface types that has been extensively used to perform trending and cross-comparison of satellite sensors is pseudo-invariant calibration sites (PICS) (Helder and others, 2010). PICS are locations on the Earth’s surface that are considered temporally, spatially, and spectrally stable with time. PICS provide a measure of stability or change present in the sensor’s radiometric response and are used for trending of sensor calibration gains and biases with time, sensor cross-calibration, and development of absolute sensor calibration models. The CEOS worked with partners around the world to establish a set of CEOS-endorsed, globally distributed, reference standard PICS for the post-launch calibration of space-based optical imaging sensors. These sites are listed in the USGS calibration and validation test sites catalog (U.S. Geological Survey, 2024b).

Signal-to-Noise Ratio

The measurement of SNR determines how much of the reported radiometric response is from the signal reflected/emitted from the target (signal) and how much is from spurious signals developed within the imager electronics, detectors, compression, transmission, and so forth. Larger SNR values result in clearer imagery. Regardless of the cause of the noise, it affects image radiometric accuracy and, if severe, degrades spatial resolution.

Measuring SNR is difficult unless the base radiance/signal is well known. SNR is often measured at the sensor/detector level. In this document, the SNR is broadly validated using measurements generally over special targets engineered for homogeneity or dark targets, such as the empty ocean at night, that are considered homogenous. An area of interest is identified over such regions, and the mean and standard deviation of the pixel values over this area of interest are determined. The ratio of mean to the standard deviation is used and reported as the SNR. Ideally, this task would be performed and reported for all spectral bands. SNR, as adopted by the USGS ECCOE, is defined in appendix 2.

Radiance/Reflectance

Radiometric correction is a prerequisite for generating high-quality scientific data, making it possible to discriminate between product artifacts and real changes in Earth processes, as well as to accurately produce land cover maps and detect changes. This work contributes to the automatic generation of surface reflectance products for the Landsat satellite series. Surface reflectance products are generated by a new approach developed from a previous simplified radiometric (atmospheric plus topographic) correction model. The proposed model keeps the core of the old model (incidence angles and cast-shadows through a digital elevation model, Earth to Sun distance, and so forth) and adds new characteristics to enhance and automatize ground reflectance retrieval.

Satellite sensor data needs to be converted to physical units such as radiance and reflectance to be useful for the scientific community. The digital numbers recorded by the satellites are converted to at-sensor spectral radiance and TOA reflectance using information usually provided in the metadata files. For the standard radiance products, the calibrated digital numbers often can be converted to at-sensor spectral radiance using the radiometric scale factor, M_λ, and a bias parameter, A_λ (both parameters are usually present in the metadata). The at-sensor spectral radiance is calculated using the following equation:

Top of Atmosphere Reflectance

Radiance data conversion to TOA reflectance is the fundamental step in putting image data from multiple sensors and platforms onto a common radiometric scale. A reduction in image-to-image variability can be achieved through normalization for solar irradiance by converting the at-sensor spectral radiance to a planetary or exoatmospheric reflectance. The TOA reflectance of the Earth is computed according to the following equation:

The TOA reflectance, ρ_λ, is proportional to the spectral radiance at the sensor’s aperture (L_λ) and the square of the Earth to Sun distance (d) in astronomical units divided by the mean exoatmospheric solar irradiance (ESUN_λ) and cosine of the solar zenith angle (θ_s). The ESUN_λ values are dependent on wavelength and solar spectrum model used in the calculation of TOA reflectance.

When comparing images from different sensors, using TOA reflectance instead of at-sensor spectral radiance has three advantages. First, TOA removes the effect of solar zenith angle owing to the time difference between data acquisitions. Second, TOA reflectance compensates for different values of the exoatmospheric solar irradiance arising from spectral band differences. Third, the TOA reflectance corrects for the variation in the Earth to Sun distance between different data acquisition dates. These variations can differ geographically and temporally.

Several factors can be addressed while comparing reflectance from two or more sensors. A consistent solar exoatmospheric model spectrum is vital because different solar exoatmospheric models are slightly different and can increase the discrepancy while comparing reflectance of two sensors. Relative radiometric sensor comparison is performed in the TOA reflectance space to eliminate the differences originated by solar zenith angle and Earth to Sun distance. The atmospheric difference between the acquisitions and the effect of BRDF owing to forward scattering or backscattering would need to be determined, if desired for higher accuracy. A compensation is made for spectral band differences, if required, using the spectral band adjustment factors.

After all conversions and corrections applied to image data from compared sensors, several regions of interest (fig. 7A) are selected from one or more near-simultaneously acquired image pairs. Corresponding regions from the reference and search sensors are usually registered and selected using cross-correlation. Average reflectance values for regions of interest from reference and assessed sensor images are calculated and plotted (as shown in fig. 7B).

In figure 7B, the comparison between reflectance in the search image (Y-axis) and reference image (OLI, X-axis). For a perfectly aligned pair of sensors, the plot should have values as close to the 1:1 line as possible. However, a linear distribution of points indicates that the sensors are aligned in a linear manner, and it is possible to extract the same information from one of the sensors by adjusting the other with the slope from the regression fit. The deviations from the line-fit are possibly a result of differences in the relative spectral response (RSR), sensor characteristics, and so forth.