Regarding the soil portion below the floating pile as a virtual soil pile, and the soil around the pile as a fractional derivative viscoelastic medium, the vertical vibration of the floating pile was solved. The boundary conditions of the soil and the pile and their continuity conditions were considered. The influence of main mechanical parameters of the pile and the soil on the complex stiffness at the head of the floating pile was investigated. The result indicates that the order of the fractional derivative has some influence on the real part and imaginary part of the complex stiffness. The constraint effect of the soil on the pile will be larger when the length of the pile is longer. The real part of the complex stiffness at the pile head will decrease when the model’s constitutive parameter increases. And the influence of the model’s constitutive parameter on the imaginary part of the complex stiffness is somewhat complicated.