Research Questions Used To Test Changes From the Rainbow Colormap

As mentioned in the “Introduction,” we used the WaterWatch map as a case study in redesigning legacy geovisualization products. A multipart process began with (1) conducting interviews with the scientific producers of the Watches products (Kenney and others, 2024) and (2) using existing static visualization guidelines to identify potential design issues. In the work described in this report, we continued to the third step in the process (3) by selecting a well-defined problem common to phases (1) and (2) to do A/B testing through general public surveys. Using the following six research questions (RQ1 through RQ6), we tested whether changing a familiar feature, the rainbow colormap, improves product usability and the user’s subjective experience for the general public.

The results are used to provide both more context-based evidence on the efficacy of colormap choices and to provide guidance on user-centered design methods for legacy products.

U.S. Geological Survey Watches

The USGS is the primary Federal operator of hydrologic monitoring networks that are used to understand and predict drought, flood, and water availability conditions in the United States. The data are used by Federal agencies, scientists, and members of the public to make decisions about local, regional, and national water conditions.

The Watches provide maps, tables, and plots to describe real-time and historical hydrologic conditions. Information is updated as new data become available at the hourly to daily frequency, depending on the sampling frequency and data transmission rates. The map-based tools were designed and developed around internal audience requests and user feedback, and although they share some visual standards, they have been developed independently and without consistent approaches to the depiction of hydrologic conditions. In addition, the underlying technology bases are becoming more complicated to both maintain and upgrade, due to reliance on unsupported software tools. Therefore, extant visualizations have not been able to benefit from more recent advances in data visualization science, and there is an opportunity for future versions of the visualizations to improve the understandability, aesthetics, consistency, and potential utility across the Watches. To keep the study size tractable, we assessed only the streamflow anomaly maps from WaterWatch.

Respondent Recruitment and Initial Filtering of Responses

For Surveys 1 and 2, University of Maryland and University of Minnesota researchers [MDG and MAK] recruited respondents through the online survey company ROI Rocket, which maintains a verified database of potential respondents, and respondents were paid for their time. Sample sizes were designed to detect a point change of 10–15 percent in understandability. For Survey 1, which occurred during October 12–22, 2020, 556 questionnaires were completed. For Survey 2, which occurred during November 22–24, 2020, 511 questionnaires were completed.

A two-stage filter was applied to drop entries from respondents who might have completed the survey too quickly to provide adequate data. For the first filter, we examined the subset of respondents who correctly answered all questions and used the fastest respondent’s overall time and time spent on the eight search-and-click questions as a minimum threshold. For both surveys, the cutoff time for filtering out the search-and-click questions that were answered too fast was set at 1.5 minutes by the survey creators. Overall times for Surveys 1 and 2 are 5.5 and 4.5 minutes, respectively. We note that the software used to design the surveys, Qualtrics, estimated the average survey completion time to be around 7.5 minutes. Filtering out entries that did not meet both thresholds dropped 87 of 556 entries from Survey 1 and 46 of 511 entries from Survey 2.

During examination of the response data, we noted that a nontrivial number of respondents did not correctly understand one of the search-and-click question types, which required a respondent to click on a specific gage inside of a boxed region on the map. As a result, we applied a second filter to drop any entries by respondents who did not click inside the box. Doing so dropped an additional 57 and 28 entries from Surveys 1 and 2, respectively. Applying the two-stage filter led to overall drop rates of 26 percent and 14 percent and final sample sizes of 412 and 437.

Experimental Protocol and Materials

Respondents completed Surveys 1 and 2 online through a browser of their choice. The survey questionnaires were designed in Qualtrics and are provided in the dataset that supports this report (Gerst and others, 2025). The questionnaire for Survey 1 consisted of five parts: (1) warmup, (2) colormap sorting task, (3) search-and-click tasks to identify clusters of high, low, and transition (normal to above-normal) streamflows, (4) user preference questions, and (5) followup questions. The questionnaire for Survey 2 consisted of four parts: (1) warmup, (2) legend questions, (3) search-and-click tasks to identify clusters of high, low, and transition (normal to above-normal) gage locations, and (4) user preference questions. Parts 3 and 4 were the same across both surveys, whereas parts 1 and 2 varied in that Survey 2 primed respondents with some information about the legend prior to asking them to rank colors by hydrologic condition, whereas Survey 1 did no such priming.

For Survey 1, the warmup (part 1) started with a description of streamflow and the conventions of measuring streamflow with respect to a baseline normal. Next, respondents were presented with five precipitation events in randomized order: hurricane, afternoon thunderstorm, morning fog, one month without rain, and drought. In order to test understanding, they were then asked to rank the effect of the events on streamflow from highest to lowest. After ranking the events, respondents were shown the correct order.

Part 2 of Survey 1 was designed to test the intuitiveness of each colormap for representing streamflow. Respondents were presented with a scenario in which they were advising a map designer on how to use color to represent streamflow. The task randomly showed the respondent either a randomized rainbow colormap or a brown-blue colormap and then asked the respondent to rearrange the colors from highest to lowest streamflow. Once the arrangement was completed, the respondent was then shown the other colormap and asked to complete the same task.

For Survey 2, the warmup (part 1) also began with a description of streamflow and its measurement conventions. However, instead of respondents being asked to complete a practice ranking task, they were given more information about how the map legends related to color, percentiles, and the qualitative descriptors used on the maps (such as “much below normal”). Part 1 then presented a task where the respondent was randomly presented with either a rainbow legend or a brown-blue legend. While the legend was still visible, they were shown three multiple-choice questions, which asked the respondent to identify the colors representing normal, high, and low streamflows. Once the task was completed, the respondent was shown the legend with the other colormap and were asked the same multiple-choice questions.

Parts 3 and 4 were the same for Surveys 1 and 2. Part 3 consisted of two types of search-and-click tasks. One task type was designed to test how accurately and efficiently map users could identify clusters of gages recording high and low streamflow extremes, whereas the other task type was designed to test how accurately and efficiently map users could distinguish among clusters of gages recording normal, below-normal, or above-normal streamflow. At the beginning of part 3, respondents were given stylized (that is, non-map-based) versions of these tasks to check for understanding and were then shown the correct answers. For each task, respondents were sequentially shown maps using rainbow colormaps or brown-blue colormaps. To control for learning effects, the order of task types as well as the order of colormaps was randomized.

Part 4 first showed respondents a list of five statements, such as “looks more trustworthy” and “colors are more visually appealing,” and they were asked to choose whether each statement applied to the rainbow colormap, the brown-blue colormap, or equally to both. Following this list, respondents were then asked which map would make them more likely to keep using a website. Part 5, which was used only in the first survey, asked a few followup questions to evaluate respondents’ experience with maps and water data.

Map Design Modifications

Prior to understandability experiments being conducted, minor modifications for both control and treatment maps were applied: gray fill was used for surface waters, dark-gray lines depicted State boundaries, and gray circles were shown at the national scale for “unranked” sites that are without sufficient data for analysis (fig. 2). In addition, the two-letter State abbreviations were removed from the Mid-Atlantic Coast, but labels were maintained for map insets: “UNITED STATES” was the label for the inset maps of Alaska and Hawaii, and “PUERTO RICO” was the label for that inset map. Control and treatment map displays maintained title and legend placement similar to the web-based product, and gray lines for stream and river flowlines were also maintained.

A rainbow colormap was used for the control map that was identical to the web-based map display, with red depicting the low-percentile class and black used for the high-percentile class (fig. 2). A white, divergent colormap was used for treatments, with brown and blue used to depict low- and high-percentile classes, respectively (fig. 3).

Data Analytic Plan

All analysis was performed in R statistical software (R Core Team, 2024). Within each survey, the effects of colormaps were analyzed using regression analysis of each task, where the treatments were coded as binary dummy variables (rainbow baseline). Understandability was investigated using logistic regression, where the dependent variable was coded as a binary variable: completing a task correctly was a one and incorrectly completing a task was a zero. Efficiency was measured by the log of the time taken to complete the survey and analyzed using linear regression.

In comparing results of Surveys 1 and 2, responses were pooled and estimated with a single regression model that contained a colormap*survey interaction term in addition to main effect terms for colormap and survey. The overall effect across surveys of changing the colormap and priming with a legend was estimated by using an analysis of variance (ANOVA) test and Wald’s chi-square test (Wald, 1943), with the size of effects estimated by average marginal effects (margins package in R software; Leeper, 2024).

Because survey respondents responded to both colormaps, coefficient standard errors were adjusted for possible within-group correlation of residual errors. Consequently, in all inferences, robust standard errors were used, which were estimated by using the vcovCL function from the sandwich package in R software (Zeileis and others, 2020).

For analyzing subjective responses, such as trust, the significance of the difference among preferences (for example, rainbow versus same, rainbow versus brown-blue) was assessed using a pairwise binomial test. This test was used to determine whether one preference was more prevalent than others. Comparison of the distribution of preferences between surveys was done using an exact Fisher test (Fisher, 1925) to determine whether priming with a legend changed colormap preferences.

Mapping Colors to Streamflow Levels—Research Question 1

The results related to Research Question 1 (Do users have preconceived beliefs about how colors map to streamflow levels?) indicate that survey respondents’ sense of how color intuitively maps to streamflow levels was highly dependent on the colormap used. In Survey 1, respondents were asked to rank colors in order of highest to lowest streamflow. For the rainbow colormap, less than 1 percent of respondents ranked all colors of the rainbow colormap correctly and only 14 percent ranked three or more colors correctly (fig. 4A). In contrast, for the brown-blue colormap, 40 percent of respondents ranked all colors correctly and 81 percent ranked three or more colors correctly (fig. 4B).

For the rainbow colormap, much of the confusion came from the placement of black and green (fig. 5). Sixty-four percent of respondents incorrectly ranked black as representing the least streamflow, while only 17 percent correctly ranked black as representing the most streamflow. Green was most identified as a higher-than-normal streamflow color, as opposed to normal streamflow. In its place, orange was the most frequently chosen color for normal streamflow. For the brown-blue colormap (fig. 5), 47 percent of respondents correctly ranked white as representing normal streamflow, whereas 39 percent incorrectly ranked white as representing the least streamflow.

Survey 2 did not include the colormap ranking questions. Instead, it showed respondents the map legend with colors and streamflow-level labels (for example, normal, above normal, much above normal) and asked them to pick the colors that matched to low, normal, and high. For both colormaps, about 93 percent of respondents were able to identify the correct colors. This result indicates that while the brown-blue colormap more intuitively represented streamflow anomalies, differences in intuition about colormapping did not necessarily affect the ability to read the streamflow anomaly legend.

Effect on Understandability of Changing Colormaps—Research Question 2

The results related to Research Question 2 (Does switching from the rainbow colormap to a brown-blue colormap affect the understandability of WaterWatch maps?) indicated that brown-blue colormaps resulted in higher respondent performance in identifying thresholds between percentile classes. For the original rainbow colormap, respondent performance in Survey 1 varied across tasks (fig. 6), with the higher-than-normal cluster identification task being most understood (79 percent) and the above-normal-transition-to-normal task being least understood (51 percent). The effect of replacing the rainbow colormap with the brown-blue colormap also varied across tasks. All tasks showed some improvement, with the largest improvements in the higher-than-normal cluster and above-normal-to-normal cluster transition tasks: odds increased by a factor of 1.61 (with a confidence interval [CI]=1.22–2.13 percent and an attained significance level [p value]<0.001) and a factor of 1.31 (CI=1.10–1.56 percent and p value=0.002), respectively (table 1). The increases in correct response rate due to the use of the blue-brown colormap indicated that using black as the highest percentile and a normal-to-above-normal transition of green to turquoise causes significant confusion.

Effect on Understandability of Priming With a Legend—Research Question 3

The results related to Research Question 3 (Is understandability enhanced by introducing users to the colormap legend?) provided information about the usefulness of introducing respondents to map legends before requiring tasks. Survey 2 had the same search-and-click tasks as Survey 1, and in addition, it primed respondents with questions about the legend used for each colormap. For the original rainbow colormap, adding questions about the legend in Survey 2 did not change the tasks’ ranking in terms of understandability: the high-percentile cluster task was most understood, and the above-normal transition task was least understood. Also similar to Survey 1, all tasks showed some improvement from replacing the rainbow colormap (table 1). The largest improvements were in the high-percentile cluster and above-normal-percentile transition tasks: odds increased by a factor of 1.33 (CI=1.01–1.75, p value=0.04) and 1.42 (CI=1.21–1.68, p value<0.001), respectively.

Examination of the pooled model ANOVA results for the comparison of Surveys 1 and 2 showed that none of the interaction terms were significant (table 2). The ANOVA results also showed that for the low-percentile cluster test, priming with the legend had a significant effect on understandability (Wald’s [1943] chi-square test statistic [chi-sq]=8.58, p value=0.004), whereas changing the colormap in either survey had a minimal effect.

Effect on Efficiency of Changing Colormaps—Research Question 4

Results related to Research Question 4 (Does switching from the rainbow colormap to a brown-blue colormap affect the efficiency of WaterWatch maps?) showed that the most efficiency was gained for the task of identifying above-normal streamflow conditions. Sample sizes (numbers of respondents) for these results include the following: high/no legend=305, high/legend=333, low/no legend=207, low/legend=253, above/no legend=179, above/legend=201, below/no=91, and below/yes=212. Variations in sample size occurred because only respondents who completed both colormaps of a task correctly were included. For the original rainbow colormap, respondent efficiency in Survey 1 varied across tasks (fig. 7), with the high-percentile cluster identification task taking the least time (20.5 seconds [s]) and the above-normal to normal cluster transition task without legend priming taking the most time (28.3 s). This variation in efficiency is notable because the high and above-normal transition tasks were the most and least understood, implying that increases in understanding are correlated with increases in efficiency. The effect of replacing the rainbow colormap with the brown-blue colormap also varied across tasks. The above-normal to normal cluster transition task showed the largest improvement, with a reduction in time of 5.3 s, while the high-percentile cluster and low-percentile cluster tasks’ reductions in time were marginal and not significant (table 3). The increase in efficiency due to the use of the brown-blue colormap in the above-normal task indicates that the transition between light blue and white was more discernable than the transition between turquoise and green.

Effect on Efficiency of Priming With a Legend—Research Question 5

Results related to Research Question 5 (Is efficiency enhanced by introducing users to the colormap legend?) indicated that legend priming in Survey 2 decreased response time in three out of four tasks—all tasks except the above-normal tasks. For the original rainbow colormap, adding questions about the legend in Survey 2 (legend priming) did not change the tasks’ ranking in terms of understandability: the high-percentile cluster identification task was completed most quickly (17.9 s), and the above-normal transition task took the most time (25.8 s). In contrast to the improvements on understandability, the legend’s improvements on efficiency were more pronounced across tasks (table 4), with the exception of the above-normal transition task. For the above-normal transition task, the effect of changing the colormap was significant (F=14.2, p value<0.001), while the effect of priming with the legend was not (F=2.24, p value=0.14). The average marginal effect of replacing the colormap is a decrease in completion time of 18.6 percent (p value<0.001).

For the high-percentile cluster identification task, the colormap had no effects on either surveys (F=0.0003, p value=0.98), but priming with a legend had a significant effect (F=10.7, p value=0.0012). The average marginal effect of priming with a legend yielded a decrease in completion time of 11.1 percent (p value=0.005). In contrast, for the low-percentile cluster identification task, both replacing the rainbow colormap (F=0.410, p value=0.52) and priming with a legend (F=5.12, p value=0.03) improved efficiency. The average marginal effect of priming with a legend yielded a decrease in completion time of 9.8 percent (p value=0.02), and within Survey 2 with legend priming, the brown-blue colormap reduced completion time compared to the rainbow colormap by 2.8 percent (p value=0.05). Similarly, for the below-normal transition task, both replacing the rainbow colormap (F=0.154, p value=0.7) and priming with a legend (F=10.9, p value<0.001) affected efficiency, but only priming with a legend led to improvements. The average marginal effect of priming with a legend decreased completion time by 16.2 percent (p value< 0.001), while with Survey 2, the brown-blue colormap increased completion compared to the rainbow colormap by 1.9 percent (p value=0.03).

User Preferences—Research Question 6

Results related to Research Question 6 (Do user preferences for the rainbow colormap or a brown-blue colormap align with objective measures of performance?) indicated that with or without legend priming, respondents had a strong subjective preference for the rainbow colormap. In Survey 1, where respondents were not primed with the legend, respondents indicated that the rainbow colormap was better at identifying extremes, more appealing, a better signal of wetter versus drier, and preferred for a graphic on the landing page of a website (fig. 8). For identifying normal streamflow and for overall trust, respondents indicated that they prefer the colormaps about the same. The brown-blue colormap was identified as best for none of the subjective outcomes. When compared to Survey 2 (849 respondents across two surveys), the distributions mostly remained the same, with the exception of overall appeal (p value=0.06) and signaling wetter versus drier (p value<0.001). In particular, priming with a legend led to an increase in the number of respondents who thought the colormaps were equally good at signaling wetter versus drier (p value<0.001) and a decrease in the number of respondents who thought the brown-blue colormap was better at signaling wetter versus drier (p value=0.001).