In this paper, we analyze the characteristics of the phase velocity of fundamental and higher mode Rayleigh waves in a six-layer earth model. The results show that fundamental mode is more sensitive to the shear velocities of shallow layers (< 7 m) and concentrated in a very narrow band (around 18 Hz) while higher modes are more sensitive to the parameters of relatively deeper layers and distributed over a wider frequency band. These properties provide a foundation of using a multi-mode joint inversion to define S-wave velocity. Inversion results of both synthetic data and a real-world example demonstrate that joint inversion with the damped least squares method and the SVD (Singular Value Decomposition) technique to invert Rayleigh waves of fundamental and higher modes can effectively reduce the ambiguity and improve the accuracy of inverted S-wave velocities.