Allow Space for Adaptation and Learning

Allowing space for adaptation and learning ensures that the design team can stay flexible and open to changes regardless of when they occur in a project’s lifetime (Conboy, Fitzgerald, and Golden, 2005). However, any development process is inherently confined by resource allocation, such as funding and staff availability. To accommodate for unforeseen changes, a project manager would ideally strike a balance between creating time allocations for completing each project outcome and keeping planning to a minimum (Barnhart and others, 2018; Meso and Jain, 2006). Ideally, managers would not try to control the development process too rigidly in order to be more responsive to changes (Cusumano and Yoffie, 1999). This part of Principle 1 is about staying flexible and embracing uncertainty in a proactive and organized manner.

Interviewees described how decision-support projects at USGS often focus on creating a tool the same way many scientists think about publishing a scientific paper—as a finite goal to be completed as quickly as possible. However, many of the larger, more successful tools at USGS were created and improved over the span of multiple years. While some interviewees revealed that this process could be frustrating early on when there is messy exploration and ambiguity, respondents also explained how allowing adequate space for the design and development process would eventually result in better tools. One interviewee stated:

Although essential to improving the tool, the iterative process described in this quote is time and resource intensive. Creating the space and time necessary for trial and error is essential for this process to occur. This can be achieved by having the team leaders monitor the team’s progress at regularly scheduled review meetings where any issues or changes in requirements can be assessed and addressed by the next meeting (Grenning, 2001). When the development team is an adaptive organizational unit that is ready to tolerate the ambiguities of the design and development process, it allows for sufficient time to discuss the assumptions, needs, and limitations of approaches (Barnhart and others, 2018). Allowing space for the design and development process to unfold requires leadership and funders to have buy-in in the process (Lavery, 2018).

Work Iteratively

Successful DST development requires iteration (Newman and others, 2000); that is, successive rounds of refining designs based on user feedback. When focusing on iteration with user groups and prototype testing, some USGS tool development teams have enlisted expertise in this area by working with usability experts within USGS or contracting organizations like 18F,⁹ a technology and design consultancy within the U.S. Government.

Respondents reported that without iteration, DST projects could miss the mark. One USGS software developer described the benefits of the Agile method of software design (a method described in Meso and Jain [2006] that includes constant iteration and stakeholder engagement throughout the design process). He stated what happens when iteration is not prioritized:

As the interviewee explains, development processes that lack iteration rarely led to good outcomes. It is important to note that iteration can occur during the development process as well as after the tool is created. Successful tools created by interviewees often resulted from long-term relationships between the tool’s creators and the tool’s users. Some interviewees reported attending meetings or workshops with the user group (sometimes meetings that were already regularly held for other purposes) in order to keep the DST relevant. For example, one interviewee stated:

This quote demonstrates how, over time, the development team worked iteratively with their users to improve the tool after it was created. Iterative prototyping and evolving development processes are key contributors to successful DSTs (Newman and others, 2000).

Obtaining user feedback throughout a tool’s lifetime is an ongoing operational task that improves a tool’s uptake and usability. User feedback can be obtained through both qualitative methods, like emails from users or media coverage, or quantitative methods, such as web analytics. One interviewee explained how user feedback helps him to modify his tool:

This quote explains why user feedback is key to any iterative tool development process and why the pathways for this feedback should be as accessible as possible. User feedback can sometimes also reveal uses for a tool that the tool's designers did not originally intend.

Work with Diverse Teams

Human-centered and user-centered design approaches put a great emphasis on the composition of the development team and their talents, skills, perspectives, and knowledge (Meso and Jain, 2006). Multi-disciplinary teams have been shown to increase the success of DST development (Fleisher and others, 2014). This success may be due in part to the integration of plural perspectives which can create a more holistic process (Vennix, 1995) or due to the way that more diverse team composition enhances creativity (Ulibarri and others, 2019).

USGS respondents who believed their DST(s) were successful all spoke highly of their team members, especially those who brought experience and knowledge to the table that the respondents themselves did not have. One interviewee added that in addition to a mix of roles in a development team, it is good to have a mix of personalities and perspectives: “You need a mix of roles and personalities in a group to make one of these [DSTs] effective * * *. Make sure you don’t have everybody who thinks just like you and is from just your little group on your team” (Interviewee 9).

This interviewee went on to explain that the type of person who excels at the operational side of maintaining the tool and someone who is primarily interested in the initial creation of the tool are very different personality types, and that the development team needs both for the tool to be successful. Multidisciplinary design teams benefit from members that span a range of roles and expertise. Some of the roles and job titles that research participants identified as being valuable included fundraiser, spokesperson, social scientist, usability specialist, graphic designer, writer, coder, and project manager. It is important to note that some of these roles and job titles are scarce in the USGS and comprise skillsets that conventionally trained USGS natural scientists do not necessarily have. For example, out of more than 8,000 USGS employees, only 11 permanent employees hold the title social scientist, although other people with social science training may work under other job titles.¹⁰ This lack of employees with specific expertise generally results in scientists backfilling these roles and shows that more training in those skillsets or hiring more employees who are trained in these skillsets could be beneficial for developing DSTs. As one interviewee explained (when referring to the hiring of usability experts), “There's people who are trained to do these skills. Appreciate them and realize the additional expertise that having someone who is trained and experienced can bring to the table as opposed to having a scientist [take on that task]” (Interviewee 18).

The mention of USGS scientists backfilling tool design roles outside of their expertise was common throughout the interviews. Having team members fulfill roles for which they do not have expertise is a tradeoff that can reduce costs, and sometimes design time, but that can also decrease the quality of aspects of the DST that may be crucial to its success such as graphic design or usability. However, human-centered design is inherently people focused and needed expertise can sometimes be found unexpectedly within teams. For example, some interviewees mentioned working with coders who also enjoyed graphic design. It is beneficial to recognize, use, and nurture the varied talents of all team members while seeking to develop a well-rounded team that can fill the roles required for successful DST development.

Bring Diversity into the Team Early

Managing diverse teams requires an open flow of communication that can benefit from early involvement of all team members. Katzenbach and Smith (1992) found teams performed more effectively when all the team members have a clear understanding of a team’s goal from the beginning. This finding was supported by the interviews on successful DSTs in which the entire design team was brought together early in the design process. The data revealed that teams that did not initially convene with every part of the design team left some roles (developers, usability experts, and stakeholder engagement scientists, in particular) being brought in too late to be fully effective.

Every USGS coder or designer, usability expert, and social scientist who was interviewed preferred to be brought into a DST project early in the process. For example, a USGS employee explained that when the programming team joins a project in its early stages, they can help define what the tool could look like and how it could function:

The benefits of involving the programmer early in the process are twofold: (1) they learn about user needs firsthand, and (2) they can inform the design team of what is and is not possible based on their specific expertise. When interviewing USGS programmers, they confirmed that being involved early in the design process was helpful to them. This holds true for every member of the team essential to the tool’s design. When team members are brought on to a project late in the game, it can be difficult for them to fix the problem that they were brought on to fix. A quote from a project manager explains how a tool would have benefitted from earlier involvement of a UX-UI expert, a term that refers to someone with expertise in user experience (UX) design and user interface (UI) design:

Get Everyone on the Same Page and Speaking the Same Language

When building a DST, having a multidisciplinary team that can bring multiple perspectives to the development process is valuable (Fleisher and others, 2014). However, working with a multidisciplinary team can be challenging because software developers, natural scientists, social scientists, and usability experts all speak slightly different languages. When you add the users, stakeholders, and outside partners to this list, it takes a considerable amount of time and energy for everyone to get on the same page. Having a mutual understanding of the team’s goals is essential for a development team to be effective because its members must be responsive, competent, and collaborative (Cockburn and Highsmith, 2001; Boehm, 2002). As one of the interviewees explained: “You end up with a better team if everybody understands all the parts of the team” (Interviewee 8).

The interviewee went on to describe how it took multiple interagency meetings for everyone involved in the tool’s design to realize that they were talking past each other. For multiple days, these agencies were talking about the importance of delivering the data in “real time” without realizing that they all had different definitions of that term. For this tool, one agency defined “real time” as seconds, the USGS defined “real-time” as four hours, and another partner agency defined “real-time” as four to six hours. Then everyone turned to the primary collaborator and intended user of the tool, and asked what their definition of real time was, and they said that real-time for their organization is seven days. Once everyone realized that they had been talking past each other and that the worry of not being able to deliver data in seconds or four hours was not a key requirement, they could move forward and design a tool that worked for the primary organization’s needs. Although every agency involved in these meetings was approaching the idea of real-time decision making differently, the goals ended up being achievable once they all got on the same page.

Define the Target Users of the Tool

Defining specifically who the target users are for a tool is one of the challenges of the DST development process (Loucks, 1995), and limited involvement of users in the development phase can lead to unsuccessful DSTs (Uran and Janssen, 2003). When users become part of the development process, scientific expertise and local knowledge are combined, maximizing the opportunities and benefits arising from the development of the tool (Oliver and others, 2017; Burnett, 2020), including mutual learning (NRC, 2009).

In the interviews, tools that had low usage were often developed without specific users in mind. Many of the interviewees explained that within USGS unsuccessful DSTs are often built without considering who the target users of the tool will be. One remarked: “That’s my pet peeve of the [U.S. Geological] Survey, is that we do tend to build tools that don’t have a [specific intended] audience… We track it for three years of use and it’s at 200 hits in its lifetime” (Interviewee 15).

The sentiment that some USGS DSTs are built without target users in mind was expressed in many of the interviews. Oftentimes interviewees explained a process in which a group of scientists discovered an intriguing problem that they thought they could solve by building a DST, without thinking about who would actually use the DST or how they would use it. As one interviewee stated:

Creating a tool in a vacuum, without identifying the target users of the tool, has led to the creation of DSTs that are not used over time and do not necessarily support the decisions they were built to support.

Understand the Decision Context

Once the target users of the DST are defined, one of the primary goals of user engagement is to understand the context of the decision(s) that the DST is built to support. Understanding the decision context means learning about how target users accomplish specific tasks, make decisions, and what they are trying to achieve or what problems they are trying to solve. Gathering this information enables the DST development team to meet user expectations and needs. As an interviewee explained:

Understanding the decision context is important because too often the interactions between scientists and users are motivated by the interests and goals of scientists and scientific agencies rather than the needs of their users (Lavery, 2018). This creates a disconnect and can ignore the obstacles faced by users, including diverse data formats and low data literacy (Sandoval-Almazán and others, 2017; Restrepo-Osorio and others, 2022; Stoltz and others, 2023). The context in which a user makes a decision defines a set of constraints that needs to be analyzed and understood early in a tool development process as they are of crucial importance, especially when opinions of the project team differ (Wallach and Scholz, 2012). Structured user engagement is essential for increasing understanding of the complex social, economic, cultural, and political settings in which a decision is made (Röckmann and others, 2012; Carmona and others, 2013) because actual users’ behavior tends to differ from the imagined behavior assumed by DST developers—especially regarding the importance of proposed features (Wallach and Scholz, 2012).

Adapt to Evolving User Needs

Engaging with users is a continuing process that ideally results in sustained relationships with users that begin before DST development and persist after a DST is launched. These relationships are important to maintain as the users’ needs and decision contexts evolve over time. User engagement needs to be strategic and sustained to be successful; Pearman and Cravens (2022) found strong links between user engagement throughout the development process and their perceptions of tools as salient and credible.

Interviewees who described successful tools often described efforts to continue learning from and sharing information with users after the DST was created. These efforts included emailing users who participated in user engagement efforts to let them know the outcome of their involvement, sharing links to the finished tools, sending out newsletters with updates, providing trainings or workshops on how to use the tool(s), and creating new or using existing pathways for user feedback. One interviewee explained their process for maintaining relationships with users:

When asked to reflect what they would have done differently in hindsight, some interviewees spoke about the importance of timing in managing relationships with users. Timing is important, and if user engagement occurs years before a tool is made available, the users’ needs may have evolved, other tools to meet those needs might have become available, or there could be a loss of interest in the tool. Maintaining continuity of participation can also be an issue; after too much time, user interest in tool development may decrease (Newman and others, 2000). One interviewee explained how it can be challenging to get users back on board when they lose interest in a tool before it is complete:

This quote explains how user engagement at the beginning of the development process is not enough to guarantee that the usership will be there once a tool is built. Additionally, sometimes tools built by the USGS are created because they anticipate a need that targeted users have yet to express. This assumption can be problematic for tool development. Users rarely have the time or resources to use new and unfamiliar tools (Barnhart and others, 2018), and tool users report that there can be significant opportunity costs to switching from a familiar tool to a new and untested one (Cravens, 2018), which makes the bar for adopting new tools high. As an interviewee explains:

When developers do not bring their users along with them on the tool creation process, the users do not have investment in the tool being created or the chance to express preferences about how the tool is developed. Involving tool users throughout the development process can increase uptake and use.

Define Metrics of Success

Identifying success metrics at the onset of tool development can help tool developers avoid trying to fulfill too many requirements with a single tool. The scope of an individual DST development effort should be kept manageable (Newman and others, 2000). Stakeholders, technical experts, and scientists need to develop a shared understanding of the decision being supported before creating the DST, and explicit metrics of success should be identified and refined throughout the process (Barnhart and others, 2018).

Evaluating the success of a DST was an aspect of tool deployment that challenged USGS interviewees. Here is how one interviewee explained the challenge:

The challenge described by this interviewee is well documented in the wider literature on decision support and arises from three main sources. First, documenting the causal link between someone’s use of a DST and improvements in their decision-making processes is challenging (Cravens, 2018). Scholars of knowledge use distinguish between conceptual information use (adding to a decision maker’s general knowledge base) and instrumental information use (providing information that feeds into a specific decision) (NRC, 2012). Most DST developers associate instrumental use with success, but studies of how people make decisions indicate that a high percentage of the information that informs a given decision is conceptual rather than instrumental (Amara and others, 2004; van der Molen and others, 2018). Second, despite repeated calls for greater attention to and resources for evaluation, support for DST evaluation remains relatively rare (Moser, 2009; Cravens, 2016; Wong-Parodi and others, 2020). Finally, measuring success requires defining concrete metrics that correspond to developer and user objectives and then collecting empirical data to assess whether and how those objectives are being met (VanderMolen and others, 2019).

Metrics for DST evaluation may be broadly divided into two categories: (1) type and amount of use and (2) how individuals interact with a tool. When interviewees were asked to describe what makes a tool successful, they often said that a tool is successful if people use it. But when we asked whether the interviewees had usage expectations for the tools they had built in the past, the interviewees indicated that they often did not. In general, we found that USGS employees were not defining upfront what they wanted their tools to accomplish nor defining metrics that would let them know their objectives had been achieved. One interviewee expressed why defining success metrics is important:

Evaluating a tool’s performance is difficult without explicit metrics of success.

Analyze Tool Use Trends

One category of success metrics are measurements of tool use across users. This includes aggregated measures of numbers of users across time or space and automated counts of tool use using web analytics. Some interviewees also spoke about assessing success in terms of referrals, mentioning that some collaborators who worked with their design team thought the resulting DST was successful enough to continue making DSTs with them and to refer other groups to their design services.

Interviewees reported they primarily monitor usage using Google Analytics. Google Analytics is a web analytics service offered by Google that tracks and reports website traffic. Web analytics can be configured to track both simple counts (for example, number of total webpage views) as well as count actions that correspond to particular user behaviors that are important to a user’s experience using a given application (for example, submit a form or click on a button). Highly sophisticated automated tracking of user behavior is common in the corporate sector, but such uses of web analytics remain rarer on scientific and government applications due to a combination of privacy regulations, data and software regulations, and likely concerns about public backlash (Cravens, 2014; Pearman and Cravens, 2022). Importantly, the ability for web analytics to measure particular behaviors is intimately related to the design and deployment of the application, which means that defining metrics before an application is built is important to ensuring the software will be designed in such a way that measurement of behaviors of interest is indeed possible (Cravens, 2014).

One interviewee described the nuances of what quantitative analytic metrics can and cannot provide and how analytics relate to tool architecture:

Notably, another interviewee reported that they erroneously believed USGS applications were not allowed to use Google Analytics and described this as a significant (perceived) barrier:

This comment illustrates the diversity of experiences with web analytics across regions and Mission Areas at USGS and, although only reflecting the experience of a single interviewee, may highlight a larger issue with employee understanding of resources available for monitoring tool use.

Understand User Experiences

Another broad category of DST evaluation metrics assesses the experience of users to understand the ways and extent to which a tool supports decision making. This category of metrics is usually assessed using a well-known evaluation technique from the usability discipline called usability testing. Many of the successful tools at USGS involved elements of usability testing, for example, observing target users using early prototypes to carry out actual work. Usability testing enables tool designers to know how well a tool works for the tool’s intended users and, as necessary, make modifications in response to findings (Loucks, 1995).

In this report, we have focused on the usability testing technique because it allows a DST team to understand how a design meets the users’ needs by directly involving the target users. For example, during a usability testing session, the users might be asked to think aloud while performing representative tasks with a DST. In turn, by staying as non-intrusive as possible, the DST team can gain insights into the users’ experiences by listening to the users and observing their interactions with the tools. Jakob Nielsen (1994) defined five key quality components that a design team might learn about through usability testing:

As one tool designer explained: “Early on, I would have a fairly strong vision about what the tool should look like, and then I would realize that the way [the users] see the tool and the way they want to use tool is not necessarily the way I have developed the tool” (Interviewee 7).

This quote shows that one cannot assume how users use a tool, and that this information can only be obtained using empirical data collection methods such as usability testing. Many interviewees expressed how usability testing and sharing early prototypes with users revealed gaps and functionality issues that would not have been discovered otherwise. This process draws on aspects of the first principle (use an adaptive, iterative design process) because it requires space for trial and error. When asked what interviewees would have done differently in hindsight, several indicated that they would have done more usability testing earlier in the tool design process. Interviewees who had access to usability experts within USGS or were able to contract usability experts outside the Bureau reported that the added expertise in usability testing was key to their tool’s success. Importantly, we note that usability testing is a formal discipline with trained professionals who caution against the dangers of “guerilla” usability testing; that is, casual and not rigorous application of their methods. However, rigorous usability testing study design does not imply large sample sizes; testing with as few as 5 to 10 users can often allow a team to identify most issues (Oakley and Daudert, 2016).

Determine the Full Life Span of the Tool

The success of a DST can be constrained by resource limitations; thus, understanding what resources will be needed before embarking on tool development is essential (Dale and English, 1999). DSTs have a life cycle and require maintenance, funding, and staff time to keep them usable during their life span; thus, they can be thought of differently than a traditional research project that may have a distinct beginning and end.

DSTs are developed throughout the USGS to support a variety of decisions, some of which may change or evolve over time, while other decisions may be more stable. Some tools are developed to meet a specific cooperator’s needs with the intention to turn tool maintenance over to the cooperator at a certain point while others may be maintained by USGS personnel over many years. These typologies of maintenance warrant consideration ahead of time to ensure the appropriate level of support and maintenance is provided.

An interviewee explained why sometimes they will not begin a project if there is not enough maintenance support:

This quote describes why the tool’s life span is best defined before tool development takes place. Supporting a tool, including scientifically, technically, or operationally, requires funding and ongoing resources. These resources are easier to acquire when the support requirements and request schedule are included in the planning process rather than retroactively. In addition to data management, technology changes rapidly, and software vulnerabilities occur. Planning in information technology (IT) for life cycle maintenance and support is critical to supporting a DST.

Invest in Maintenance

Some interviewees who experienced working on tools that fulfilled a need but were not maintained cited the inability to provide maintenance to the tool. The lack of maintenance can be due to a lack of funding to support IT staff and a dearth of programming expertise. This section of planning for the tool’s life span focuses on the importance of acknowledging and planning for technical debt. Technical debt is a concept in software development that reflects the implied cost of reworking a tool caused by changes in technology (Tom and others, 2013). Updates may also cause security vulnerabilities. Although technical debt cannot always be avoided, it can be anticipated by staying proactive and diligent about future tool maintenance requirements. During one of the interviews, a USGS developer explained the concept of technical debt:

Many interviewees spoke about technical debt and changing technologies, and the need for time and staff to keep a tool running. One interviewee who lacks in-depth programming experience explained their team’s way of overcoming this technological barrier:

While this quote demonstrates one way of skirting the barrier of technological debt, other interviewees spoke about contracting out programming experience or training in coding software. These solutions require support and resources, and interviewees had many suggestions for resources that would help them in the tool design process. Furthermore, it is important that DST team members work with their supervisors or respective regional or Mission Area administrative leadership and the USGS CIO to ensure their DST uses the correct IT capabilities and addresses Federal requirements. Coordinating early in your planning with those in USGS who set IT policy can avoid issues in the long run.