Cadmium (Cd) exposure is recognized as an important risk factor for psychological health, but suitable physical activity may relieve depression. However, it remains unknown whether physical activity (PA) can reduce the effect of cadmium exposure on depression. Therefore, a cross-sectional data from National Health and Nutrition Examination Survey (NHANES) 2015–2018 was used. The Nine-item Patient Health Questionnaire (PHQ-9) was used to assess depression among the participants. PA was calculated according to the metabolic equivalent (MET), weekly frequency, and duration of each activity. Logistic regression and restricted cubic spline models were used to examine the associations of Cd and depression. A total of 5560 adults aged 20 years and above were finally included in this study. The results indicated a positive correlation between blood Cd and depression. The multivariate-adjusted ORs (95% CI) of the highest quartile were 2.290 (1.754–2.990) for depression, which was still significant after controlling other heavy metals (P < 0.05). Under Cd exposure, the high intensity of physical activity group had the lowest risk of depression (OR = 2.226, 95%CI: 1.447–3.425), while the group with no physical activity had the highest risk (OR = 2.443, 95%CI: 1.382–4.318). Our results indicate that inner Cd exposure may be a risk factor for depression, and physical activity can moderate this relationship to some degree.