Most animal groups vary extensively in size. Because individuals in certain sizes of groups often have higher apparent fitness than those in other groups, why wide group size variation persists in most populations remains unexplained. We used a 30-y mark–recapture study of colonially breeding cliff swallows (Petrochelidon pyrrhonota) to show that the survival advantages of different colony sizes fluctuated among years. Colony size was under both stabilizing and directional selection in different years, and reversals in the sign of directional selection regularly occurred. Directional selection was predicted in part by drought conditions: birds in larger colonies tended to be favored in cooler and wetter years, and birds in smaller colonies in hotter and drier years. Oscillating selection on colony size likely reflected annual differences in food availability and the consequent importance of information transfer, and/or the level of ectoparasitism, with the net benefit of sociality varying under these different conditions. Averaged across years, there was no net directional change in selection on colony size. The wide range in cliff swallow group size is probably maintained by fluctuating survival selection and represents the first case, to our knowledge, in which fitness advantages of different group sizes regularly oscillate over time in a natural vertebrate population.