First Session—Recap of the Main Takeaways from Past Meetings (2017 and 2019)

Christian Crowley (PPA) reviewed the main takeaways from the past two meetings on novel data in recreation monitoring. The main takeaways from the 2017 meeting were as follows: (1) there was variation across traditional approaches that agencies use to estimate recreational visitation and the economic benefits of visitation; (2) there was a desire to increase consistency among agencies; and (3) there was interest in exploring novel data sources, including social media, mobile device data, and remote sensing technologies. The takeaways from the 2019 meeting were as follows: (1) there is a mature research effort underway that is exploring the use of social media data; (2) scientific researchers concluded that user-generated data (for example, social media) are reliably correlated with visitation; and (3) these datasets are worth continuing to explore, and it is helpful to have continued discussions as a community of research and practice.

“State of the Science” Presentation

The 2023 workshop began with a presentation by Spencer Wood (University of Washington), Emily Wilkins (USGS), and Eric White (FS) on the current state of the science regarding the use of novel data for recreation monitoring. The goals of this session were to (1) build on existing knowledge and experience, (2) increase familiarity with traditional and novel data sources, (3) outline known and potential issues with user-generated data, (4) describe empirical evaluations of data and methods, and (5) establish current best practices.

The presentation began with an overview of definitions and concepts and then described the state of the research on using novel data to estimate visitation, visitor characteristics, and information about visits. Definitions, concepts, and examples used during this presentation and workshop include the following.

Novel data.—Examples of novel data include geolocated social media, mobile phone use, and community science submissions. These are sometimes referred to as “volunteered data” or “user-generated data.”

Useful data.—Useful data are data that are accurate, understandable, accessible, ethical, and help answer a relevant question.

Social media.—Social media from internet-based sites that recreation visitors use to publicly post information and images related to recreation visits can be useful for recreation monitoring. Examples of applications include Flickr, Strava, Instagram, AllTrails, and other online trip reporting platforms (for example, Tripadvisor or local online communities). Data from social media applications can include, for example, text content, images, geotagged locations, and time stamps.

Mobile phone location.—Many users enable location services on their mobile phones, and in the process, give their consent for various phone applications to track their locations and sell these data to third parties. Third parties may also process these data and sell them to researchers.

Community science.—Examples of community science include eBird (where users can upload bird sightings) and a community science program that allows people to contribute information about their visit to public lands. Visitors send text messages to a phone number that is connected to a bot that asks questions about their visit.

Traditional (onsite) data sources.—Examples of traditional (onsite) data sources include trail counters, visitor intercept surveys, and trail cameras.

A growing body of research during the last decade has evaluated the use of social media data for estimating visitation rates (for example, Wood and others, 2013; Keeler and others, 2015; Sonter and others, 2016; Tenkanen and others, 2017; Hamstead and others, 2018; for reviews refer to Wilkins, Wood, and Smith, 2021 and Ghermandi, 2022). In many locations, studies show there is a strong correlation between social media posts and traditional data sources, like trail counters. This information about the relative levels of visitation (for example, comparing across time or parks) does not necessarily tell us absolute levels of visitation. Emerging questions include how to estimate absolute numbers of visitors and which data sources are most useful. Two studies have addressed these questions by creating models of visitation using a variety of predictors (Merrill and others, 2020; Wood and others, 2020). These studies took place on public lands in western Washington and New Mexico (Wood and others, 2020) and public beaches in New England (Merrill and others, 2020). Both studies show that visitation models parameterized with traditional data perform better when novel data are included as predictors. Key findings from this research are that visitation models are improved when multiple novel data sources are used in concert and that the models should be calibrated with local, onsite counts.

Some research has also evaluated if novel data can be used to estimate visitor characteristics, including home locations and demographics. Home location can be inferred from social media profiles or frequent posting locations. Using mobile data, home location can be inferred from the phone’s most frequent overnight location, among other methods. Visitor demographics are usually inferred either by using artificial intelligence to evaluate the content of uploaded images (Mashhadi and others, 2021) or by relating home location with census data. Studies evaluating the accuracy of these types of data compared with traditional survey methods include Sessions and others (2016); Heikinheimo and others (2017); Fisher and others (2019); Sinclair and others (2020); and Liang and others (2022). Results indicate that in some situations, home location inferred from social media data may reflect a potential bias depending on who is posting. For instance, international visitors are more likely to post photographs of their trips, whereas local visitors are less likely to share photographs (Sessions and others, 2016; Sinclair and others, 2020; Wilkins and others, 2022). Additionally, although researchers often can get home locations from mobile device data, they may not be able to differentiate between recreational visitors and others in the area not for recreation (Liang and others, 2022). For this and other reasons, the estimated demographics of park visitors from census data may be inaccurate, though to date, there is less research on this (Monz and others, 2021; Liang and others, 2022).

Researchers have also evaluated if novel data can answer questions about a visit, such as what activities visitors engaged in and how satisfied they were with their visit. For inferring activities, researchers have relied on several methods, including image content analysis—either manually or using artificial intelligence tools to process images shared on social media. Available research indicates that using social media images and machine learning (a convolutional neural network) to infer activities can be accurate for some activities, such as hiking and bird watching, but less accurate for other activities, such as backpacking and swimming (as indicated in a test at two national forest sites in Washington) (Winder and others, 2022). Winder and others (2022) note that the specific convolutional neural networks are generally better at recognizing activities that have clear and recognizable objects (for example, bird watching) but are less able to distinguish between activities with similar equipment (for example, hiking and backpacking). There is also likely some bias in which activities are photographed and shared (Winder and others, 2022). Content posted on social media can be used to assess visitor sentiment (refer to Pickering and others, 2020), though this requires a large amount of text processing even for a coarse evaluation of emotion (for example, positive, neutral, or negative).

“State of the Application” “Lightning-Round” Talks

After the “State of the Science” presentation, a representative from each agency gave a 10-minute presentation or update regarding current (as of February 2023) uses of novel data sources, or how novel data might help answer agency questions in the future. The main points by agency representatives are summarized as follows.

NOAA.—The NOAA has used human mobility data for several projects and continues to expand their application in marine areas. For instance, the NOAA used mobile data from a vendor named Unacast to analyze spatial and temporal visitor-use trends within the Middle Peninsula in Virginia with an overall goal of understanding human pressures on reserve sites and potential barriers to access. With West Virginia University, the NOAA completed a pilot project in the Florida Keys National Marine Sanctuary that used Instagram data and survey data to estimate trips for different activities (for example, diving and fishing). They are planning to incorporate multiple datasets (including apps, surveys, and satellite imagery) for visitor monitoring as part of the socioeconomic monitoring plan for the Mission: Iconic Reefs.

BLM.—The BLM has not used novel data sources to date but is starting to explore the potential (for example, using mobile device data to help estimate visitation). They have a key goal of supporting staff whose other duties may preclude monitoring visitation using traditional methods. The BLM manages a vast land area with many low-use or dispersed sites, and there is often little to no visitor data for these types of locations.

FWS.—The FWS has efforts underway to improve approaches to visitor estimation, including a national visitor survey through 5 years at several national wildlife refuges. The FWS also has an ongoing project with Clemson University and the University of Washington to test methods and data sources for estimating visitation. Phase 1 of this project consisted of a literature review and Delphi panel of experts to compare attributes of visitor estimation methods. Phase 2 is ongoing and involves developing a visitation model for refuges using novel data and on-the-ground data to parameterize the model. Research questions include how to scale up a model and if it can be applied nationally to all (or some) refuges.

FS.—The visitor monitoring program for the FS is the National Visitor Use Monitoring (NVUM) program. Through this program, the FS collects onsite data every 5 years to create visitation estimates and describe visitor characteristics. One major benefit of these data is that they are a long time series with more than 20 years of data. Although the FS is interested in novel data, there is some concern about losing the value of many years of consistently collected data and losing the type of information provided by visitor surveys (or other original data collection efforts) if it were replaced by novel data in the future. The FS is forming an advisory group related to the use of novel data.

NPS.—The NPS is testing the use of novel data to determine where and how they can be applied. To date, the NPS has completed studies at five parks to test the use of mobile and connected-vehicle data. One of the main findings is that mobile data are potentially useful but need more validation and cannot replace traditional counts. Additionally, the cost of using mobile data is high and difficult to estimate in advance. At the time of this workshop, the NPS views novel data as a “value-added dataset” for calculating or correcting multipliers or to identify high- or low-use areas that may require more or less survey effort.

EPA.—The EPA initially explored using novel data when visitor surveys were delayed while under Office of Management and Budget review for the Paperwork Reduction Act (Public Law 104–13; 109 Stat. 163). Novel data allowed researchers to get information about visitors without directly interacting with them. Environmental Protection Agency researchers found the data useful for comparing on-the-ground counts with mobile data. Additionally, EPA offices tend to mostly use stated-preference surveys, and the revealed-preference methods introduced by novel data were a useful addition and complement to stated-preference methods.

Examples of Success and Need for Additional Examples (Breakout Session 1)

Questions guiding this discussion were as follows: What are the shining examples of success using novel data for recreation monitoring in recent years (in other words, 2019–23)? Where would we like to see more examples? The following are some of the responses discussed during this breakout session.

Use-Case Presentations

Four presenters gave examples of case studies for evaluating the use and accuracy of novel data. Short summaries with key points from each presentation are as follows.

“Visitation Estimation with Cellular Device Location Datasets” (Nate Merrill with coauthors Wei-Lun Tsai, Anne Neale, Madeline Grupper, Kate Mulvaney, Marisa Mazzotta, and Justin Bousquin).—Nate Merrill (EPA) presented on using human mobility data to predict on-the-ground visitation counts. Specifically, the authors compared mobile device data from AirSage to monthly visitation counts at 38 NPS units. They ran a panel regression with random effects, stratifying by distance to a population center, level of recognition among nonlocal communities (in other words, if a park was considered iconic), park size, and porousness. The authors found generally high coefficient of determination values for the regressions predicting monthly visitation counts using mobile device data. In particular, mobile device location data were a good predictor of visitation for nonurban and iconic parks, large parks, and parks with low porousness. This indicates that for some parks, mobile data could serve as a complement to on-the-ground visitation estimates and may also be useful to show relative changes. However, the authors note that this approach still relies on on-the-ground methods and that cell data providers, inputs, and data processing are continually changing, requiring researchers to frequently update and adapt their methods. Merrill is working on a study to assess the accuracy of origin information from mobile device data. For more information on this research, refer to Tsai and others (2023); for previous related work, refer to Merrill and others (2020).

“Modeling Recreation on Public Lands Using Social Media and Mobile Data” (Sama Winder with coauthors Emmi Lia and Spencer Wood).—Sama Winder (University of Washington) presented a study using multiple novel data sources to estimate visitation on public lands, including trails in Washington and New Mexico and national wildlife refuges. For estimating visitation on trails in Washington and New Mexico, the authors used social media data from Flickr, Twitter, and Instagram. They also included baseline data of holidays, weather, and week of the year in the models. Using these factors, the authors could predict weekly visitors with an adjusted coefficient of determination of 0.74 in Washington. When applying the model to New Mexico, it still worked well but worked best if it could be calibrated with local data. The authors are currently (as of February 2023) working on applying the same methods to estimate visitation at 11 national wildlife refuges but also including AllTrails reports, eBird checklists, and mobile data from AirSage and Spectus to help predict visitation. Initial analyses show that the accuracy of novel datasets varies by data source, by location, and through time. However, combining these data from multiple sources, including social media and mobile device data, in a single model shows promise for predicting monthly visitors to national wildlife refuges. This work highlights the importance of calibrating novel datasets with trustworthy onsite data, combining multiple novel datasets, testing local model performance, and periodically recalibrating models as datasets change through time. The work on national wildlife refuges is still preliminary, but for more information on the models in Washington and New Mexico, refer to Wood and others (2020).

“Location-based Services Data” (Matt Brownlee).—Matt Brownlee (Clemson University) joined remotely to present three case studies using location-based services (LBS) data. The first study focused on spatial and temporal distributions of visitor use in Theodore Roosevelt National Park in North Dakota, the second focused on visitor demographics in Joshua Tree National Park in California, and the third investigated visitors’ dwell time and stops in the North Inlet-Winyah Bay National Estuarine Research Reserve in South Carolina. For the first study on spatial and temporal distributions, Brownlee compared information from global positioning system data loggers to LBS data. The spatial and temporal distributions are highly correlated in these two datasets but do contain some statistical differences. For the second study, when comparing demographics from LBS data to surveys, there was a positive association between age, education, and income categories but a statistically significant difference in averages. For instance, surveys indicated that park visitors were more educated and had higher incomes relative to estimates from LBS data. In study three, Brownlee found that LBS data were useful for determining stops and dwell time. This work is not published as of 2023 but may be published in the future.

“Regional Recreational Transportation Analysis Projects” (Rachel Collins with coauthor Erica Cole).—Rachel Collins (NPS) joined remotely to discuss using mobile device data for planning purposes in NPS contexts. The authors found that mobile device data were useful for a regional transportation system usage analysis for national parks in Colorado, using LBS data from 2018 to 2019. From the mobile device data, they obtained metrics such as weekday compared with weekend visitation, seasonality of use, proportions of resident compared with nonresident visitors, and travel speed. In Mount Rainier National Park, they completed a regional transportation-system usage analysis for 2020–22. Location-based services data confirmed seasonal visitation trends at the park. Overall, LBS data closely match count data from park entrance locations, and they also match home-location data from surveys fairly well. For related work, refer to Baird and others (2022).

Day 1 Wrap-Up

The first day ended with a whole-group discussion inviting questions that had not yet been discussed. Two main questions emerged: Can we use novel data to create forecasting models to predict future visitation? Could these data be used for real-time estimates? For the first question, the group consensus was that forecasting in general is difficult, and there is uncertainty in forecasts compounded by reliance on other forecasts (for example, population growth). Including novel data sources in forecasting models could help improve the accuracy. Participants from the FS and the University of Washington noted that they have reports or papers in press that forecast visitation and recreational activity participation (two have now been published: Goebel and others, 2023 and U.S. Department of Agriculture Forest Service, 2023). For the second question, the group discussed how real-time data differ from visitation data, including which questions these data are used to answer (for example, real-time data are often used to monitor sites for crowding). Obtaining and processing novel data to provide real-time estimates of recreation use (at individual sites or for administrative units) is challenging and likely not currently feasible given the time lags in data presently commercially available (as of this workshop in February 2023) and Federal agency computing realities. Additionally, workshop participants were skeptical that real-time estimates, even if available, would be useful in managing crowding or reducing visitor conflict.

Start of Day 2—Concerns with Using Novel Data (Breakout Session 2)

The second day began with a discussion about concerns with using novel data. Questions guiding this discussion were as follows: What are the main concerns with current applications, current research, and future directions for the use of social media and mobile device data, or other novel data for recreation monitoring (as of the time of this workshop in February 2023)? Topics mentioned during this breakout session are detailed as follows.

Panel on Mobile Device Data

A panel of four people spoke about their experiences working with mobile device data from an agency perspective. The panel consisted of Nate Merrill (EPA), Sarah Cline (FS), Rachel Collins (NPS), and David Pettebone (NPS). Other workshop participants who had experience using mobile device data also shared input. This section consists of a summary of some of the topics discussed and insights gained from the panelists and other participants with experience using these data.

The panelists noted that there are many companies that sell processed mobile device data (for example, AirSage, StreetLight, and Unacast) and that they all are slightly different. The NPS has used a contracting approach where the contractor decides how to purchase and process the data, so the NPS does not own the data. One of the key questions to consider before buying mobile device data is whether to purchase raw data that need processing to be useful or data that have already been processed. Purchasing raw data can be useful because then the researcher will know how the data were processed to generate final visitation estimates, but raw data also present a privacy concern and require someone who is skilled in analyzing this type of data. The NPS also has purchased connected-vehicle data (from Otonomo), which they note have the option to “chain” or combine trips made during a 24-hour period, rather than have trips defined by when a vehicle is turned on and off.

The cost to purchase mobile device data varies based on the vendor. Vendors generally offer data as either subscriptions or as downloads for certain places and times (by a bank of credits). One panelist noted that most purchase agreements from companies selling mobile device data prohibit the sharing of these data, which can be challenging because Federal agencies are required to share research data.

Two significant concerns regarding mobile device data were (1) the methods for processing the data are often a “black box” and (2) the data are often presented by vendors as being off-the-shelf-ready, when in reality, the data often need to be calibrated and validated to ensure quality. The methods being a “black box” are concerning because agency researchers must be able to understand and describe all data processing. Additionally, changes to the methods through time may affect the reliability of these data for visitor monitoring across periods. Panelists also noted that vendors may market mobile data to people who do not have prior experience using this type of data source, and companies sometimes may present this data source as a replacement that is cheaper and easier to use than onsite data sources (for example, traffic counters, trail counters, and surveys). Mobile data currently (as of February 2023) cannot replace traditional data sources on public lands and waters and should be viewed as a complementary data source rather than a replacement. Further, on-the-ground counts are still needed to calibrate and validate mobile device data.

Working Session for a Best Practices Document

Questions guiding this discussion were as follows: Who is the audience for a best practices document? What is the content? This working session began with a review of a document produced from the 2019 workshop: a four-page report that focused on the uses, the limitations, and a case study for using social media for visitor monitoring. Workshop participants agreed that an updated overview document would be very useful, following a similar format (either two or four pages). The new document should address the use of social media and mobile device data. Important points to communicate are that novel data are a complement—not a replacement—to traditional data, and the continued exploration of integrating novel data is important for a robust visitor monitoring program.

Research Agenda and Future Directions (Breakout Session 3)

Questions guiding this discussion were as follows: What are the research needs to inform practice? What new projects could be rolled out now (if funding were unlimited)? An overview of topics discussed during this breakout session is presented as follows.

Workshop Outcomes and Survey Results

Workshop participants agreed that it would be beneficial to start a community of practice where participants and others interested in this line of work can share research and ongoing studies related to using novel data for recreation monitoring. Some workshop participants volunteered to help start this group, and the DOI now hosts a SharePoint site and associated Microsoft Teams channel that is open for anyone to join, ask questions, and share resources related to these topics. The group also plans to hold recurring meetings with updates and research presentations. Participants noted that continuing to meet in person every 2 to 3 years would be valuable.

The workshop planning committee circulated an anonymous followup survey for Federal participants to share their thoughts about the workshop. Of 19 attendees (not including the 5 workshop organizers), we received 11 responses between February 18 and March 3, 2023 (response rate of 58 percent). All 11 respondents attended in person for day 1, and 10 of these respondents attended in person for day 2.

For our question about overall satisfaction with the workshop, nine respondents chose “very satisfied,” one respondent chose “somewhat satisfied,” no respondents chose “somewhat dissatisfied,” and one respondent chose “very dissatisfied” (the respondent who chose “very dissatisfied” indicated they were “very satisfied” or “somewhat satisfied” with every individual session of the workshop, so it is possible this was a mistake due to the ordering of the choices). For the individual sessions, the satisfaction level of participants was as follows:

For our question on new plans for projects as a result of the meeting, we received several positive responses. Some examples are as follows:

We also received several less definite responses, including the following:

We asked respondents to rank five methods in terms of the potential to add value to their work. The results were as follows (in order from highest potential to add value to lowest potential to add value):

Current State of Knowledge and Practice in Using Social Media as Recreation Data

Spencer Wood (University of Washington) gave an introductory presentation that provided an overview of the research using social media to quantify and characterize visitation. Key points from this presentation are as follows.

Estimating visitation.—The presentation summarized the results of a growing body of academic research that shows a correlation between the number of posts on social media platforms and onsite visitation as measured and reported through traditional methods. The presentation discussed the platforms and content types used in this type of analysis, primarily georeferenced photographs on Flickr and Instagram, georeferenced tweets on Twitter, and location-specific posts on participatory trip forums such as Washington Trails Association (wta.org), a local example in Washington.

Technical methods include using social media posts to estimate user-days, which is the unique number of people posting in each area on a given day (Wood and others, 2013). Many of these studies, in diverse locations and settings around the world, have a strong correlation—around 70 percent—between the number of photographs posted online and visitor counts from conventional methods (Wood and others, 2013). These studies typically use a large geographic scale, such as an entire national park, and a long temporal scale, such as an entire year. Additional work is needed for understanding the functional limits of the methods at finer spatial and temporal scales. In addition, correlations are often strongest at popular sites and weaker in locations with fewer visitors or less social media data to analyze. An example of using social media to estimate visitation is a study from western Washington, where researchers have been working in the Mount Baker-Snoqualmie National Forest to collect onsite and social media data to create models of recreation visitation that incorporate a variety of parameters (Fisher and others, 2018). These approaches relied on several social media platforms to obtain the most robust characterization of visitation. This research has since been published; refer to Wood and others (2020).

Visitor locations and demographics.—The home location of visitors who post social media can be inferred using information contained in the user profiles or by analyzing the geographic patterns of their posts. Examples were shared from national parks that were able to provide an estimate of visitors’ county of origin that corresponded to on-the-ground estimates produced by the NPS with some important differences (refer to Sessions and others, 2016).

Visitor preferences.—Social media may help researchers investigate visitor preferences. Using an econometric approach that analyzed revealed preferences, the data can illustrate the place-dependent nature of visitor preferences. For example, a study in Minnesota and Iowa found that recreationists prefer lakes with higher water clarity based on Flickr “photo-user-days” and travel distances of lake users across the two States (refer to Keeler and others, 2015).

Data access and availability.—Posts are often accessed by means of an application programming interface provided by the relevant social media platform. The access policies and format of the data vary across platforms. For example, Flickr, a photograph-sharing platform, makes available the specific latitude and longitude of every photograph. Instagram, in contrast, allows people who share images to tag the images by selecting from a prepopulated set of geographic locations that may not correspond to official park boundaries, although the platform did previously show the precise coordinates of every image. This change in Instagram’s user interface illustrates that platform policies and policy changes have implications for the collection and analysis of social media data. Additionally, the popularity of both platforms has changed through time. Flickr use has declined since around 2016, whereas Instagram use has increased since then.

Content analysis.—Content analysis is usually performed by analyzing the text or images in social media posts. A relevant example for outdoor recreation is using photographs or text to determine visitor activity participation. For example, some researchers have examined user posts to Tripadvisor to understand participation in various coral reef-based activities (refer to Bartelet and others, 2022).

State-of-the-Art Applied Research and Applications

There were five presentations on applications of crowdsourced data for recreation monitoring. Details about the presentations are as follows.

Promising Applications, Concerns, and Limitations (Breakout Session 1)

During the final session of day 1, participants formed five small groups to discuss their reactions to the technical presentations and the upsides and complications of using crowdsourced data. The workshop organizers asked three questions related to the promise of these methods, opportunities, and concerns:

Responses to each of these questions are outlined as follows.

Federal Agency Updates on Recreation Monitoring Programs

Agency recreation leads shared updates on agency recreation monitoring programs and focused on happenings since the 2017 workshop. The following are some key points mentioned by each agency representative during the 2019 workshop.

BLM.—The BLM uses the Recreation Management Information System, a field-driven visitor estimation reporting program. The focus (as of 2019) is having field staff provide a reporting plan and document methodology for generating visitation estimates. The BLM is working to update guidelines for estimation reporting and establish formal geospatial boundaries for reporting units. Estimating visitation in dispersed areas is a challenge that could possibly be improved with emerging methods.

Bureau of Reclamation.—The Bureau of Reclamation relies on managing partners to provide recreation-use estimates and moved from physical form-based systems to an online reporting system for visitation estimates. Local administrators and partners are not required to document their estimation methodology, but that may be a future step.

NOAA.—The NOAA is working to better estimate recreation at national marine sanctuaries. West Virginia University is leading pilot monitoring studies in two locations: Gray’s Reef National Marine Sanctuary (Georgia) and Florida Keys National Marine Sanctuary. They will be looking at a range of potential data sources including smart buoys, satellite data, and social media data. The goal is to develop a low-cost method of understanding visitor use to serve as a basis for understanding use at all 15 sanctuaries and 2 marine national monuments. Some of this research has since been published; refer to Burns and others (2020).

NPS.—The NPS has several efforts exploring the use of new tools and technology in visitor-use statistics. Some examples include the following: testing the use of Bluetooth and Wi-Fi in Zion National Park (Utah) and Yellowstone National Park to understand how visitors move through the park, testing the use of three-dimensional cameras at the Korean War Veterans Memorial to count visitors instead of manual onsite counts, and testing the use of LBS data in northern Arizona. The NPS is also launching a national socioeconomic monitoring program. This will be a survey effort (with sampling at 24 representative parks per year), and the results will also help refine vehicle multipliers for visitor estimation.

USACE.—The USACE completed a monitoring modernization project that began in 2013. They used survey data to develop new parameters for handling recreation traffic counts and converting those counts into visitation estimates. The USACE’s most challenging areas for estimating visitation are rural areas and areas that have a variety of access points.

FWS.—A variety of approaches are used to develop recreation-use estimates at refuges. In 2018, the FWS pilot-tested a new survey system that would sample all refuges every 5 years. They are also pilot-testing a visit estimation system with a focus on hunting and visits to urban refuges. The focus is on traditional approaches, but there is interest in emerging methods.

FS.—There were three changes to the NVUM program implemented in fiscal year 2020. These changes include switching from paper surveys to using tablets, slightly modifying some survey questions to reduce confusion, and no longer requiring 24-hour mechanical traffic counters during sampling dates.

USGS.—The USGS is initiating a project to estimate ecosystem services produced in the Nisqually River delta in western Washington, using social media to estimate the provision of recreation-related ecosystem services.

Opportunities for Integrating New and Emerging Data Sources (Breakout Session 2)

In late morning of day 2, participants broke into four small groups to discuss how the new methods discussed on day 1 could be used or integrated with existing approaches. The groups brainstormed three types of applications: estimating visitation, content analysis (including demographics, preferences, and values), and active crowdsourcing. The following list represents a set of possible studies, administrative actions, or uses of crowdsourced data as of April 2019.

Research and Management Needs Going Forward (Breakout Session 3)

In early afternoon of day 2, participants broke into researcher community and management community groups to identify needs for expanded use of social media and crowdsourced data. Following is a summary of the content recorded by those groups on paper posters.

Workshop Outcomes and Survey Results

After the 2019 workshop, the organizers distributed a survey for participants to give anonymous feedback. Of 29 attendees (not including the 4 organizers), 21 participants responded to the survey. All respondents expressed that they were either satisfied (6 out of 21) or very satisfied (15 out of 21) with the meeting, and no respondents expressed that they were neutral, dissatisfied, or very dissatisfied.

When asked if they made any new plans for projects with partners because of this meeting, 10 respondents indicated they had, and an additional 3 respondents said they were still following up with other participants and may have new projects and partners because of this meeting. When asked if they made any new plans for projects solely within their organization because of this meeting, seven respondents said they had, and another six indicated “potentially” or “not yet” but that there may be projects in the future stemming from this meeting.

The survey also asked participants which method they thought had the most potential for adding value to their work. Ten people indicated “location-based services data (for counting visitors, looking at travel patterns, etc. [sic]),” four people indicated “actively solicited crowd-sourced [sic] data (apps where users can volunteer information, Short Message Service [sic] (SMS) surveys, etc. [sic]),” three people indicated “social media for counting visitors,” one person indicated “social media image content (user experience, demographics, activities, etc. [sic]),” and three people responded “other.” The “other” responses consisted of “paired actively solicited and location-based,” “social media for validating sampling frameworks, especially in dispersed settings,” and “social media content analysis (text as well as images).”

All 21 respondents expressed that this community should meet again. Nine respondents thought the community should meet every 2 years, 11 respondents thought the community should meet every year, and 1 respondent thought there should be meetings more often than once a year. No respondents indicated that meetings should be less often than every 2 years. When asked if the next meeting should be held in Washington, D.C., Colorado, or the West Coast, preferences were somewhat split. There were eight votes for Colorado, six votes for the West Coast, and five votes for Washington, D.C. (and some respondents did not answer).