Abstract: Based on the data of CEPS (2014-2015) , this paper takes the HLM Model to explore the restrictive factors on physical exercise of the middle school students from the dimensions of individual characteristics, family and school. The result shows: interest on physical exercise, awareness of health, health class, identity of one-child, academic pressure, parents' levelof education, family economic status, location of school and sports facility have significant and positive effect on physical exercise time of the middle school students, but media use has a negative effect. Gender, the sense of obesity and course duration have no significant effect. Therefore, school and family, on one hand, should conduct more train of physical fitness to cultivate students' physical exercise consciousness, interest and autorhythmicity, and effectively control students' media use; on the other hand, should pay attention to sports activities, create better circumstance of physical exercise, support and urge students to attend physical exercise.