Glass containing spherical silver nanoparticles shows a strong extinction band in the visible range due to the surface plasmon resonance (SPR) of the particles. Irradiating this material with intense, ultrashort laser pulses with a wavelength close to the SPR leads to permanent changes of its optical properties. In particular, using linearly polarized pulses, we observed strong dichroism; the latter is nanoscopically caused by deformation of the particles to ellipsoidal shapes with an additional halo of small silver particles around the central one, with a preferential orientation. In case of a single laser shot of sufficient intensity this orientation is orthogonal to the laser polarization, whereas multi-shot irradiation usually causes preferential orientation along the laser polarization. This effect is quite useful for the production of dichroitic or polarizing microstructures, and optical elements or optoelectronic devices. In this paper we describe the results of a variety of experimental studies (mostly femtosecond laser pump-probe, electron microscopy, photoluminescence) on the understanding of the physical processes, which show clearly that ultrafast ejection of electron and silver ions into the glass matrix is the starting mechanism, whereas in the course of deformation diffusion processes controlled by the local temperature play a decisive role for the final particle shapes (and thus the optical properties after laser treatment).