Nonlinear dynamics and stability of the rotor bearing seal system are investigated both theoretically and experimentally. An experimental rotor bearing seal system is simplified as the Jeffcott rotor. The nonlinear oil film forces are obtained under the short bearing theory and Muszynska nonlinear seal force model is used. Numerical method is utilized to solve the nonlinear governing equations. Bifurcation diagrams, Poincar maps, spectrum plots and rotor orbits are drawn to analyze various nonlinear phenomena and system unstable processes. Theoretical results from numerical analysis are in good agreement with results from experiments.