The aim of this work is the theoretical investigation of instability development of the nanoparticle surface shape during diffuse growth. The unperturbed surface is considered as an ideal sphere. The mass diffusion flux promotes increase of the sphere volume and development of free surface perturbations. The relaxation of these perturbations due to action of capillary forces is also considered along with development of small perturbations of a free surface at the expense of perturbation of a mass diffusion flux. We use the effective coefficient of viscosity μ and surface tension coefficient σ of nanoparticle to describe a surface shape relaxation. As a result there were revealed various modes at which the surface of nanoparticle loses the stability and investigated dynamics of instability development of surface.