In this work, Gaussian (Normal) and Uniform Probability Density Functions (Distributions) are used to describe random variables.

For a single (scalar) value, the Gaussian distribution isNθ|μ,σ2=12πσ2e−12θ−μ2σ2 (1)where

For a vector of multiple values, the multivariate Gaussian distribution isNθ|μ,Σ=12πkΣe−12θ−μTΣ−1θ−μ(2)where

For a single (scalar) value, the Uniform Distribution isfθ=1b−a for a≤θ≤b0 for xθ or θb(3)where

PEST is based on the Gauss-Levenberg-Marquardt technique for damped least-squares regression. Fienen and others (2024) provide a derivation and interpretation of the algorithm. This is a gradient-based algorithm that depends on parameter sensitivity to supply the gradients. The parameter sensitivity of the modeled output m(θ) (collocated in space and time with observation z) to model input parameter θ is defined as

This sensitivity is commonly approximated using finite differences by perturbing a parameter by a small increment Δθ as∂z∂θ≈mθ+Δθ−mθΔθ(5)where

The Jacobian matrix (J) is constructed of the local sensitivity of all pairs of observations with parameters forming a matrix:J=∂zi∂θj(6)where

At its root, the Jacobian matrix can be interpreted as the basis for an approximation for a mapping from ℝn→ℝm used in the change of variables for integration in calculus (the determinant of the Jacobian matrix provides the actual mapping if the Jacobian is a square matrix; Simmons, 1985, p. 673). In the context of parameter estimation, this matrix not only provides the gradients needed to find a solution to the inverse problem, but also serves as an approximate mapping from observation space to parameter space (for example, ℝobs→ℝpars), thus projecting information from a set of observation data to a set of parameter data. If the system were totally linear, then this mapping would be complete and a unique mapping would result in a calibrated model in one step. However, this is not the case, as discussed briefly below in the Parameter Estimation—Gauss-Levenberg-Marquardt Algorithm section (and in more detail in Fienen and others, 2024), so the mapping of observation information to parameters is approximate. That approximation, when adjusted based on assumed uncertainty of observation data, is a powerful one and can inform the likely quality of a parameter-estimation effort. This approximation can map the uncertainty of observation data onto parameter data and, similarly, project that out to model forecasts.

A similar calculation can also be made to determine the sensitivity of a model forecast (s) to the model and parameters (θ) also using the finite-difference approximationy≜∂s∂θ≈sθ+Δθ−sθΔθ(7)where, in this case,

Similar to local sensitivity of model observations, this calculation is made entirely from adjusting parameters and calculating model outputs. As a result, there is no need for an independent estimate of the forecast value to calculate this sensitivity.

The parameter-estimation process entails the finding of the set of parameter values that minimizes an objective function. The objective function (Φ) is a metric of weighted, squared differences between observations (z) and modeled equivalents to them Jθ:Φ=z−JθTΣϵz−Jθ (8)where

The diagonal variance values σ² constituting Σ_ϵ are informed to the PEST and PyEmu programs as weights, defined as w=1σ, where σ is an estimate of the standard deviation of the observed value. The matrix Σ_ϵ has σ² for each observation on the diagonal. This value of σ corresponds with the epistemic error that encapsulates measurement error and assumed errors in the model (for example, Doherty and Welter, 2010; White and others, 2014). As a result, the covariance matrices are metrics of error or uncertainty.

To find the parameters that minimize Φ (eq. 8) for a linear case, the best estimate of parameters θ^ is available using the Gauss-Newton method as

For nonlinear cases, a Taylor expansion is repeatedly iterated about an initial estimate of parameters (x₀). For an incremental step to new parameterszθ+δ≈zθ+Jδ(10)where

Rearranging this to iteratively solve for parameters θ, each subsequent iteration takes the form ofθ^i=θ^i−1+JTΣϵJ+λdiagJTJ−1JTΣϵz−Jθ^i−1(11)where

This iterative procedure accounts for the nonlinearity of the problem. For a linear problem, J would not change with changing parameter values. The inverse problem is solved making use of just these few pieces of information: J, Σ_ϵ, and z. As a result, much information regarding potential parameter estimation results can be gleaned from these values.

At the heart of the propagation of variance we wish to explore uncertainty cascading from observations to parameters and ultimately to forecasts. We can accomplish this propagation of variance in a Bayesian context. Bayes’ theorem states thatph|d=pd|hphpd∝pd|hph(12)where

In plain language, the posterior distribution of the hypothesis given the data p(h|d) is proportional to the product of the a priori knowledge about the hypothesis independent from the data ph and the information that the data provides pd|h. This is a form of updating information through new observations or experiments. In the parameter-estimation context, one can think of h as a set of parameters to be estimated, and d as a set of observations used in the parameter-estimation process.

From a Bayesian context, we can formally estimate the change in variance when updating the prior variance with new information (in this case, through parameter estimation with new evidence). A full derivation of these calculations is presented in Fienen and others (2010). The cascade of information from observations to parameters is “Notional Calibration” (Doherty, 2010b) as it reflects the reduction of uncertainty to parameters that would result from calibration to the dataset for which J indicates the sensitivity, and assuming the weights in Σ_ϵ and the prior parameter covariance in Σ_θ, if the relationships were all linear.

Given a prior covariance for the parameters (Σ_θ), the posterior covariance after updating with new observations Σ¯θ can be calculated using the Schur Complement (White and others, 2016) as

This equation shows that the prior covariance (Σ_θ) is either unchanged or reduced by the parameter-estimation process represented as the second term on the right-hand side. The actual observation values do not feature in this equation—only the sensitivity to observations as expressed in the Jacobian matrix J. This allows the value of potential observations to be evaluated as long as the sensitivity of a model output collocated in time and space with a potential observation location can be evaluated through perturbations, as shown in the “Local Sensitivity and the Jacobian Matrix” section.

The prior variance of a forecast based on prior parameter covariance or uncertainty (Σ_θ) can be calculated using the forecast sensitivity outlined in equation 7:

Similarly, the posterior variance of a forecast, based on an estimate of reduced parameter uncertainty through notional calibration, can be calculated as

The Monte Carlo technique is a rejection sampling technique in which an ensemble of parameter values is generated and the underlying process model is run using each member of the ensemble as a parameter set. For nonlinear Monte Carlo techniques, a Gaussian Distribution (eqs. 1 and 2) or a Uniform Distribution (eq. 3) is used to generate parameter or observation realizations by drawing random samples from the distributions. These realizations can use information from the PEST control file to construct variance and covariance values or use geostatistical information about the parameter values in the Gaussian case or to construct the bounds for the Uniform case. For each member, the objective function (Φ, eq. 8) is calculated, and for values of Φ lower than an acceptable threshold, all model outputs are aggregated, resulting in distributions of outputs and accompanying metrics of their covariance or uncertainty.

The first analysis result shows the contribution of various model parameters to uncertainty. This result can be summarized by group to help indicate which broad categories of parameter information contribute most significantly to the posterior uncertainty. As shown in figure 3, recharge pilot points in aggregate (rpp) are the most important, followed by specific yield (sy), anisotropy (aniso), and horizontal hydraulic conductivity pilot points in aggregate (hkpp). The remaining parameter groups include zones of hydraulic conductivity (horizontal and vertical), streamflow routing (sfr), and specific storage (ss).

The worth of existing observations can be calculated using equation 18. This has the dual purpose of potentially helping indicate which observations may be redundant in a monitoring network and helping users to understand the importance of specific observations in the calibration dataset as they impact forecast uncertainty. Figure 4 shows the locations of the observations scaled by their removed observation importance, aggregated across all stress periods. The “x” marks on figure 4 indicate all locations of head observations that were available for at least part of the history-matching time period. Two forecasts were evaluated for this analysis—the mean and median drawdown in the composite hydrograph area. In figure 4A and 4B showing aggregated data worth for both forecasts, the colored circle symbols indicate aggregated normalized data worth for all existing observation locations with a value greater than 0.001. The aggregated normalized data worth was calculated by summing the calculated data worth across all forecast and history-matching stress periods for which an observation was simulated at each specific location. This quantity was then normalized by dividing by the maximum aggregated value at each point. Therefore, the actual magnitude of the quantity does not have an interpretable value, but the relative rank of the values can help inform which locations have relatively more or less value. As figure 4A and 4B indicates, the patterns are not well connected with hydrologic features such as the streamflow routing (sfr) cells. Indeed, although the highest value in each case is near the composite hydrograph area, it is not within the area, which is suspicious. The density of existing data confounds the ability of this analysis to identify a single most important or valuable data point. The level of redundancy of existing data indicates that, in this case, the analysis does not deeply inform decisions to remove data from future consideration.

The final analysis using linear uncertainty methods evaluated the contributions to uncertainty reduction made by data obtained from new observations. For this evaluation, a network of potential head observations was simulated in every model cell within a buffer of about 10 miles around the composite hydrograph area. Despite good spatial coverage of head observations used for the calibration of the model, this exercise was performed to evaluate the potential to fill in gaps of unobserved areas and to focus on data collected at the onset of the forecast period. Forecast uncertainty was evaluated at each stress period (for example, 6-month intervals) for the entire 50-year forecast period. As discussed in the “First-Order Second-Moment (FOSM) Propagation of Variance” section, data worth must be evaluated with respect to a specific forecast of interest. As a result, potential data worth values exist for every simulated potential observation location for all 100 forecast-period stress periods. An aggregated representation of data worth, normalized to the maximum aggregated data worth over all forecast time, is shown in figure 5A. In other words, the scale from 0.0 to 1.0 is intended to convey the spatial location of maximum forecast uncertainty reduction achieved if a new head observation is made at the beginning of the forecast period (stress period 17). A heatmap of the typical location of the maximum drawdown within the composite hydrograph area, to provide some context regarding where most additional information is likely to be found if adding a new observation, is shown in figure 5B. The actual decrease of forecast variance for the mean drawdown is shown in figure 5C and generally decreases throughout the forecast time. This trend highlights that, as time continues in the forecast period, the value of monitoring new data decreases because more of the actual forecast time has elapsed and monitoring data at the beginning of the forecast period is eclipsed by actual observations throughout that time period. These results show that potential new data are most valuable close to the center of mass of where the maximum drawdown forecast is located. This conclusion is logical, as pumping greater than recharge is the key driver of drawdown, so collecting more data in the region where the forecast is occurring should be informative about the forecast. Observations at the beginning of the forecast period are also more informative than subsequent observations as the forecast period elapses (fig. 5C). The evolution of the actual forecast over time and that, once the trajectory is underway, the value of observations at the beginning of the forecast period diminishes, are shown in figure 5D.

The mathematical framework depends on the Jacobian matrix, a prior covariance structure for the parameters, and a meaningful observation weighting scheme. All of these elements depend on a model that is stable, predictable in its behavior, and a reasonable simulator of the system. Instabilities and large misrepresentations of the system dynamics can negatively impact the value of data-worth calculations.

In the example we evaluated, the existence of a large spatially and temporally distributed observation network masks the value of the analysis. For example, the identification of a single observation among hundreds of existing observations as less valuable implies that the information such an observation provides is redundant or disconnected from the forecast of interest. However, with such uniform coverage, one could argue that almost all the observations are redundant and picking one or more to drop is difficult either qualitatively or using the quantitative methods examined here. Similarly, when there are few spatial gaps where observations do not exist, it is difficult for the method to evaluate where additional information could be added. Finally, the temporal aspect of this modeling effort required summarizing observation acquisition and forecast times in ways that make the base units of the data worth quantities difficult to interpret, but instead focus on the rank of data worth from one location to another. The data worth methods presented are perhaps most valuable when fewer observations already exist. Nonetheless, the insights gained are valuable and the low incremental computational cost when a parameter estimation framework is in place make this a valuable analysis tool to use.