Herbivores alter plant biodiversity (species richness) in many of the world’s ecosystems, but the magnitude and the direction of herbivore effects on biodiversity vary widely within and among ecosystems. One current theory predicts that herbivores enhance plant biodiversity at high productivity but have the opposite effect at low productivity. Yet, empirical support for the importance of site productivity as a mediator of these herbivore impacts is equivocal. Here, we synthesize data from 252 large-herbivore exclusion studies, spanning a 20-fold range in site productivity, to test an alternative hypothesis—that herbivore-induced changes in the competitive environment determine the response of plant biodiversity to herbivory irrespective of productivity. Under this hypothesis, when herbivores reduce the abundance (biomass, cover) of dominant species (for example, because the dominant plant is palatable), additional resources become available to support new species, thereby increasing biodiversity. By contrast, if herbivores promote high dominance by increasing the abundance of herbivory-resistant, unpalatable species, then resource availability for other species decreases reducing biodiversity. We show that herbivore-induced change in dominance, independent of site productivity or precipitation (a proxy for productivity), is the best predictor of herbivore effects on biodiversity in grassland and savannah sites. Given that most herbaceous ecosystems are dominated by one or a few species, altering the competitive environment via herbivores or by other means may be an effective strategy for conserving biodiversity in grasslands and savannahs globally.