In the Arctic Ocean, the cold and relatively fresh water beneath the sea ice is separated from the underlying warmer and saltier Atlantic Layer by a halocline. Ongoing sea ice loss and warming in the Arctic Ocean have demonstrated the instability of the halocline, with implications for further sea ice loss. The stability of the halocline through past climate variations is unclear. Here we estimate intermediate water temperatures over the past 50,000 years from the Mg/Ca and Sr/Ca values of ostracods from 31 Arctic sediment cores. From about 50 to 11 kyr ago, the central Arctic Basin from 1,000 to 2,500 m was occupied by a water mass we call Glacial Arctic Intermediate Water. This water mass was 1–2 °C warmer than modern Arctic Intermediate Water, with temperatures peaking during or just before millennial-scale Heinrich cold events and the Younger Dryas cold interval. We use numerical modelling to show that the intermediate depth warming could result from the expected decrease in the flux of fresh water to the Arctic Ocean during glacial conditions, which would cause the halocline to deepen and push the warm Atlantic Layer into intermediate depths. Although not modelled, the reduced formation of cold, deep waters due to the exposure of the Arctic continental shelf could also contribute to the intermediate depth warming.