An understanding of the factors controlling plant community composition will allow improved prediction of the responses of plant communities to natural and anthropogenic environmental change. Using monitoring data from 1980 to 2009, we quantified the changes in community composition in Leymus chinensis and Stipa grandis dominated grasslands in Inner Mongolia under long-term grazing-exclusion and free-grazing conditions, respectively. We demonstrated that the practice of long-term grazing exclusion has significant effects on the heterogeneity, the dominant species, and the community composition in the two grasslands.