Semantic mapping in 3D is fundamental to a wide range of geoscientific studies and applications, including geomorphology, hazard assessment, and environmental monitoring. However, automatically segmenting geological features from large-scale photogrammetric datasets remains a significant challenge. We present a methodology to address this gap. Using overlapping images collected over environments of interest, Structure-from-Motion (SfM) produces georeferenced point clouds and estimates camera poses. Existing large vision models, such as Segment Anything Model, segment objects in the images, generating pixel-segmentation associations. To produce pixel-point associations, we project the points back onto the camera image planes. As objects are independently segmented across multiple images with different perspectives, we develop a segmentation mosaicking algorithm to build probabilistic point-segmentation associations that combines the pixel-segmentation associations and pixel-point associations. Our methodology is validated using both synthetic data generated by Kubric and real-world UAV-SfM data. The implementation is designed to be compatible with existing SfM software, including Agisoft and OpenDroneMap, for photogrammetry mapping in geoscience studies. As a case study, we apply our method to the semantic mapping of precariously balanced rocks (PBRs), which provide upper-bound constraints on historical ground motion shaking intensity. To support object-level identification of PBRs, we additionally integrated Grounding DINO, enabling text-prompted segmentation of features of interest within UAV imagery. This case study demonstrates the effectiveness of our method in generating a 3D semantic map of PBRs, enabling spatial distribution of PBR fragility for earthquake hazard analysis.