The population size of southern elephant seals in the southern Indian and Pacific Oceans decreased precipitously between the 1950s and 1990s. To investigate the reasons behind this, we studied the population of southern elephant seals at Heard Island between 1949 and 1954, using data collected by the early Australian National Antarctic Research Expeditions. Seals were marked and measured (lengths) as weaned pups, and resighted at Heard and Marion islands and in the Vestfold Hills, Antarctica in subsequent years. Bayesian state-space mark-recapture models were used to determine post-weaning survival. Yearling survival was consistently lower (ϕy: 0.28–0.40) than sub-adult survival (ϕs: 0.79–0.83). We found evidence for constant sub-adult survival and time-dependent resight probabilities. Weaning length was an important determinate of yearling survival, with the probability of survival increasing with individual length. There was some suggestion that the Southern Annular Mode influenced yearling survival but this evidence was not strong. Nonetheless, our results provide further support showing that size at independence affects yearling survival. Given the known sensitivity of southern elephant seal populations to survival early in life, it is possible that the decline in population size at Heard Island between the 1950s and 1990s like that at Macquarie Island was due to low yearling survival mediated through maternal ability to produce large pups and the dominant environmental conditions mothers experience during pregnancy.