Self-organization is a process of establishing and reinforcing local structures through feedbacks between internal population dynamics and external factors. In reef-building systems, substrate is collectively engineered by individuals that also occupy it and compete for space. Reefs are constrained spatially by the physical environment, and by mortality, which reduces production but exposes substrate for recruits. Reef self-organization therefore depends on efficient balancing of production and occupancy of substrate. To examine this, we develop a three-dimensional individual-based model (IBM) of oyster reef mechanics. Shell substrate is grown by individuals as valves, accumulates at the reef level, and degrades following mortality. Single restoration events and subsequent dynamics are simulated for a case study in South Carolina (USA). Variability in model processes is included on recruitment, spatial environmental constraints, and predation, over multiple independent runs and five predator community scenarios. The main goal for this study is to summarize trends in dynamics that are robust across this uncertainty, and from these generate new hypotheses and predictions for future studies. Simulation results demonstrate three phases following restoration: initial transient dynamics with considerable shell loss, followed by growth and saturation of the live population, and then saturation of settlement habitat several years later. Over half of simulations recoup initial shell losses as populations grow, while others continue in decline. The balance between population density, substrate supporting the reef, and exposed surfaces for settlement is mediated by overall population size and size structure, presence of predators, and relative amounts of live individuals and intact dead shells. The efficiency of settlement substrate production improves through time as population size structure becomes more complex, and the population of dead valves accumulates.