We report on new dynamics in vertical-cavity surface-emitting lasers (VCSELs), which is induced by a polarization preserving optical feedback. Our solitary VCSEL, i.e. without feedback, emits in a single linearly polarized (LP) and fundamental transverse mode. Optical feedback induces multiple polarization switchings in the L-I curve. When the injection current is set close to one of the multiple polarization switching points, the laser system exhibits a random anticorrelated hopping of the two orthogonal LP fundamental modes. Of particular interest is the observation of antiphase oscillations in the two LP modes that occur with a period corresponding to the delay time, and which complement the slow mode-hopping. Moreover, the LP-resolved optical spectra show random frequency hops between ECMs. The laser system is therefore not only bistable between the LP modes, but also multistable between ECMs. A theoretical rate equation approach yields a good qualitative agreement with our experiment. We report similar delay-periodic oscillations complementing a mode hopping in a scalar delay differential equation modeling the motion of a particle in a double well potential. Such a model was recently investigated by Tsimring and Pikovsky, in a very general context. Our results in VCSELs yield the first experimental evidence of delay-periodic oscillations in a bistable physical system with noise, and are therefore thought to be of general interest for many other processes in biology, medicine, etc. that exhibit interaction of noise, delay and bistability.