Fencing for grazing exclusion and grazing are common land-use methods in the semi-arid areas of the Loess Plateau in China, which have been widely found to change grassland soil organic carbon (SOC); however empirical studies that evaluated driving factors of soil carbon (C) stocks under the different land use are still weak. In this study, we investigated soil physicochemical and soil respiration (Rs) in the fenced and grazed grassland, to study the soil C stock variations and the main driving mechanism of soil C accumulation. The results showed that bulk density (BD), soil moisture content (SMC), and soil porosity (SP) had no significant difference between fenced and grazed grassland. Fencing increased the SOC, total nitrogen (TN), and C/N ratio, and significantly increased the aboveground biomass (AGB), belowground biomass (BGB), and the amount of soil large macro-aggregates in the topsoil layer (0-10 cm), and the soil stability was improved. Meanwhile, grazing increased soil temperature (ST) and Rs. The soil C stock in the topsoil layer (0-10 cm) of fenced grassland was significantly higher than that of grazed grassland. The soil C/N ratio, BD, and MWD explained large proportions of the variations in soil C stocks. Our results indicate that fencing can improve the stability of soil structure, and reduce Rs, then increase soil C stocks, which is an effective way to improve soil C stocks of grassland ecological in semi-arid areas of northwest China.