Inventory of USGS Enterprise IMT Resources

The USGS gained access to many new data-intensive technologies and related resources in the past decade. However, for the typical USGS scientist, accessing such resources is not intuitive because they are spread across different locations, managed by different groups, and connected in different ways. To address these issues, the Data Management and Advanced Technology subgroup worked with the ACIO and CSS to review and document available enterprise IMT resources in the USGS with application to science in the Colorado River Basin. This inventory was initially compiled in a spreadsheet, and the concept eventually evolved into a more user-friendly diagram that illustrates how the different resources relate to one another (fig. 1). The resources are grouped into generalized IMT service categories, and the diagram identifies the “information technology ecosystem” where the resources reside (for example, cloud, hybrid, or on premises). In addition, the diagram places these resources into the USGS Science Data Lifecycle Model that describes the stages of data management and how data flow through a research project (Faundeen and others, 2013). Additional details about the IMT resources identified in figure 1 can be found in appendix 1.

Exploration of Data Management and Principles

Exponential growth in the volume, variety, and complexity of data highlights the need to consider data-management strategies. As such, data-management plans are now required by USGS policy (Fundamental Science Practices Section 502.6: Scientific Data Management Office of Science Quality and Integrity; November 10, 2021, at https://www.usgs.gov/survey-manual/5026-fundamental-science-practices-scientific-data-management). The Fundamental Science Practices provide brief descriptions and resources for generating data-management plans, which include reference to a USGS Data Management Plan website (https://www.usgs.gov/data-management). Data management plans can be structured according to the USGS Science Data Lifecycle Model, which describes the stages of data management and how data flow through a research project from conception through preservation and sharing (Faundeen and others, 2013). The USGS Data Management Plan website provides USGS data-management plan templates and external tools to help establish a data-management plan. The Data Management and Advanced Technology subgroup recommends that a data-management plan be established for all data generated during the project.

USGS data-management practices are guided by the Findability, Accessibility, Interoperability, and Reusability (FAIR) principles that serve to guide data producers and publishers in maximizing the added value gained by formal, scholarly digital publishing of data and algorithms, tools, and workflows that were used to produce data (Wilkinson and others, 2016; Lightsom and others, 2022). A major challenge identified during this assessment is a lack of knowledge on how and where to find selected types of data and models. In addition, differences in the formats and locations of USGS data limit their interoperability and reusability. Specifically, the time, location, and purpose of the data collection affect whether they are stored in enterprise or local systems. The Data Management and Advanced Technology subgroup recommends that future discussions be held regarding how various data repositories supported by the Bureau can be more unified and governed by the FAIR principles, as outlined by Lightsom and others (2022).

The USGS provides data to stakeholders through several mechanisms, and the particular delivery portal used may depend on data type, science discipline, or region in which the data were collected. The Data Management and Advanced Technology subgroup had discussions with USGS managers and database developers regarding several portals to learn more about their uses. ScienceBase was developed by the USGS to enhance and expand information sharing and sound data-management practices. ScienceBase is a USGS trusted digital repository and provides access to aggregated information derived from many data and information domains. Key elements of ScienceBase include the following: (1) data cataloging and collaborative data-management platform, (2) central search and discovery application, (3) web services facilitating other applications, and (4) research community catalogs. The USGS Science Data Catalog integrates with the USGS Thesaurus, which allows users to browse for data using topics identified in the thesaurus. The Science Data Catalog is yet another location to find data and information produced by the USGS. The Science Data Catalog is a search and discovery tool that allows for metadata retrieval, visualization, download, and linkages to data repositories. The Science Data Catalog helps the USGS meet open data reporting required by the White House, offers a single source for the USGS to serve metadata to https://www.doi.gov/data and data.gov, and serves as a member node to the National Science Foundation-sponsored Data Observation Network for Earth, or DataONE, project.

Analysis-ready data are preprocessed in a format that facilitates further analysis. Analysis-ready data, therefore, can increase the speed of returning scientific results to stakeholders and decision makers. The USGS has recognized the importance of analysis-ready data, and substantial work was done to make the Landsat data archive analysis-ready. This allows users to quickly produce Landsat-based maps of land cover and change, and other derived geophysical and biophysical products (Dwyer and others, 2018). The Data Management and Advanced Technology subgroup recognizes the importance of analysis-ready data and recommends providing analysis-ready data in all cases to support integrated applications.

The increase in computing power has also led to an increase in data modeling. The USGS Model Catalog (https://data.usgs.gov/modelcatalog/) is led by the Core Science Systems Mission Area, Science Analytics and Synthesis Program, and Science Data Management Branch. The USGS Model Catalog includes models, tools, frameworks, and testbeds developed by the USGS or developed externally and used in USGS research. This catalog can increase awareness and facilitate model discovery, encourage reproducibility of science, and promote benefits of good model documentation. Compilation of the Model Catalog is ongoing, and at the time of writing, the Model Catalog focused on process-based scientific models rather than statistical, ML, or conceptual models. Entries for each model contain documentation recommended by the Open Modeling Foundation (https://www.openmodelingfoundation.org/standards/documentation/), including model version, related links, open-source license, and programming language.

Big data analytics and ML are applications that highlight the importance of data management, which were discussed by the subgroup and USGS scientists working in these fields. The artificial intelligence for environment and sustainability (ARIES) is an integrated modeling technology that is underlain by AI designed to help computers understand the data they are evaluating (found at https://aries.integratedmodelling.org/). The platform has been in development since 2007 by an international group of scientists. The system uses rules to guide the reasoning process and is highly dependent on semantics (consistent application of terminology) to give rigorous and logically consistent definitions. The Basin Characterization Model, an example of USGS big-data analytics, uses historical climate data, such as snow cover, snow water equivalent, evapotranspiration, soil moisture, and streamflow, to predict future water balances under differing climate and land-cover scenarios (Flint and others, 2013; Flint and others, 2021). The modeling outputs can be semantically described and used to inform other models, such as aquatic and terrestrial ecological models and reservoir management models, in AI applications like ARIES.

Advanced IMT and the Short-Term Colorado River Basin Projects

The ASIST pilot project selected 15 collaborative projects focused on drought to demonstrate their utility to help implement the EarthMAP concept for future science activities in the Colorado River Basin in 2020 (table 1). These projects had short-term deliverables that could be completed in fiscal year 2021 with support from the pilot project. The projects were further reviewed for their proposed and potential applications of advanced IMT to support and facilitate integrated science and deliver actionable information. For a select group of projects, the ACIO provided additional support and coordination to connect the projects with information technology (IT) specialists to discuss and provide recommendations for needed IT capabilities and expertise. Several projects proposed using advanced IMT resources, such as HPC, AI, ML, and analysis and visualization tools, or planned to use these resources in future phases of the projects. Full short-term project descriptions are available in the ASIST science strategy (Dahm and others, 2023).

Science and Technology Collaboration Meetings

In the summer of 2021, the ASIST pilot project organized 12 collaboration meetings that brought together interested USGS staff with diverse perspectives to discuss science and technology challenges, existing capabilities, example applications, knowledge gaps, and actions. The goals of the meetings were to (1) provide opportunities to meet with others working on similar topics in different science centers or programs; (2) identify science and technology applications in the Colorado River Basin, stakeholder science and technology needs, existing USGS expertise and capabilities, and knowledge gaps; and (3) discuss science integration and technology implementation strategies to address knowledge gaps and actions to support implementation of new approaches for the USGS. In nearly all meetings, advanced IMT capabilities were mentioned, highlighting the importance of integrating technologies into future project workflows. The collaboration meetings that specifically focused on advanced IMT included the following topics:

Building EarthMAP Capacity and the Analysis-Ready Data/Dremio Working Group

The Building EarthMAP Capacity project was initiated in July 2020 to support EarthMAP projects and perform a rapid assessment of cyberinfrastructure and integrated modeling capabilities in the USGS. The Building EarthMAP Capacity team established eight working groups to evaluate and provide recommendations for the following thematic areas: analysis-ready data and data lake; analysis and visualization; communities of practice and partnerships; infrastructure as code and development and operations; advanced scientific computing; large data transfer, network, and storage capacity; Model Catalog; and computational testbeds. Identified gaps and recommendations were summarized in an internal USGS report released in November 2020.

In June 2021, members of the ASIST pilot project, ACIO, and CSS established a working group to build on the Building EarthMAP Capacity “analysis-ready data and data lake” recommendations and demonstrate workflows to connect science across disciplines and systems and improve the service of analysis-ready data for direct consumption. Specifically, this project’s participants plan to design, test, and implement a workflow for accessing scientific data held in USGS data repositories (for example, ScienceBase) using the CHS data hub service, Dremio, for analysis and synthesis, with submittal back to the repository for release. Dremio is a Structured Query Language, or SQL, lakehouse platform that leverages existing data repository structure and reduces data complexities and accelerates querying of tabular data to deliver fast and efficient results to end users. A project product will be a data governance strategy to support this data workflow scenario.

Lessons Learned

The USGS initiated the ASIST pilot project to accelerate interdisciplinary science carried out in the Colorado River Basin and apply advanced IMT solutions. The pilot project team was tasked with developing integrated predictive capabilities for decision making through advanced technology. Historically, the USGS developed data workflows within individual disciplines; however, 21st-century predictive science requires tools and technologies that span multiple disciplines and support science projects led by staff across the entire Bureau. The work during the first year of the ASIST pilot project focused on identifying IMT resources; documenting monitoring, analysis, and prediction capacities; and hosting scientific collaboration meetings. The lessons learned through those first-year activities helped focus and define future efforts. The lessons learned from the Data Management and Advanced Technology subgroup include the following: