Recent observations have shown the existence of an apparent impenetrable barrier at the inner edge of the ultra-relativistic outer electron radiation belt. This apparent impenetrable barrier has not been explained. However, recent studies have suggested that fast loss, such as associated with scattering into the atmosphere from man-made very-low frequency transmissions, is required to limit the Earthward extent of the belt. Here we show that the steep flux gradient at the implied barrier location is instead explained as a natural consequence of ultra-low frequency wave radial diffusion. Contrary to earlier claims, sharp boundaries in fast loss processes at the barrier are not needed. Moreover, we show that penetration to the barrier can occur on the timescale of days rather than years as previously reported, with the Earthward extent of the belt being limited by the finite duration of strong solar wind driving, which can encompass only a single geomagnetic storm.