In an effort to infer compositional information about distant targets based on multispectral imaging data, we investigated methods of relating Mars Exploration Rover (MER) Pancam multispectral remote sensing observations to in situ alpha particle X-ray spectrometer (APXS)-derived elemental abundances and Mössbauer (MB)-derived abundances of Fe-bearing phases at the MER field sites in Gusev crater and Meridiani Planum. The majority of the partial correlation coefficients between these data sets were not statistically significant. Restricting the targets to those that were abraded by the rock abrasion tool (RAT) led to improved Pearson’s correlations, most notably between the red–blue ratio (673 nm/434 nm) and Fe³⁺-bearing phases, but partial correlations were not statistically significant. Partial Least Squares (PLS) calculations relating Pancam 11-color visible to near-IR (VNIR; ∼400–1000 nm) “spectra” to APXS and Mössbauer element or mineral abundances showed generally poor performance, although the presence of compositional outliers led to improved PLS results for data from Meridiani. When the Meridiani PLS model for pyroxene was tested by predicting the pyroxene content of Gusev targets, the results were poor, indicating that the PLS models for Meridiani are not applicable to data from other sites. Soft Independent Modeling of Class Analogy (SIMCA) classification of Gusev crater data showed mixed results. Of the 24 Gusev test regions of interest (ROIs) with known classes, 11 had >30% of the pixels in the ROI classified correctly, while others were mis-classified or unclassified. k-Means clustering of APXS and Mössbauer data was used to assign Meridiani targets to compositional classes. The clustering-derived classes corresponded to meaningful geologic and/or color unit differences, and SIMCA classification using these classes was somewhat successful, with >30% of pixels correctly classified in 9 of the 11 ROIs with known classes. This work shows that the relationship between SWIR multispectral imaging data and APXS- and Mössbauer-derived composition/mineralogy is often weak, a perhaps not entirely unexpected result given the different surface sampling depths of SWIR imaging (uppermost few microns) vs. APXS (tens of μm) and MB measurements (hundreds of μm). Results from the upcoming Mars Science Laboratory (MSL) rover’s ChemCam Laser Induced Breakdown Spectroscopy (LIBS) instrument may show a closer relationship to Mastcam SWIR multispectral observations, however, because the initial laser shots onto a target will analyze only the upper few micrometers of the surface. The clustering and classification methods used in this study can be applied to any data set to formalize the definition of classes and identify targets that do not fit in previously defined classes.