Long-term retention and accumulation of heavy metals in surface water and sediment pose a great threat to the sustainable development of aquatic ecosystems and human health. In this research, macrozoobenthos, and surface water and sediment heavy metal (Mn, Cr, Cu, Ni, Pb, Zn, Cd, and As) samples were collected from 23 sites in the upper and middle reaches of the Heihe River in the summers of 2019–2020. The interrelationships between heavy metals and macrozoobenthos were analyzed by Pearson correlation analysis and redundancy analysis (RDA), and the contamination level and potential ecological risk of the heavy metals in surface water and sediments were assessed by comprehensive pollution index (WQI), single potential ecological risk index ([Formula: see text]), and sediment quality guideline-quotient (SQG-Q), respectively. The results showed that the level of heavy metal pollution in the surface water of the Heihe River was extremely low, but the contents of Mn and Pb exceeded the third class of surface water environmental quality standards (GB 3838–2002). The spatial distribution of heavy metals in surface sediments were significantly different, and the average concentration of 8 heavy metals exceeded the background values of Gansu Province. Meanwhile, the pollution level of heavy metals in surface sediments was high pollution, and the ecological risk level was moderate risk. Combining [Formula: see text] and SQG-Q evaluations showed that Cd, Ni, and Cr were the main ecological risk factors. In addition, the distribution coefficients of Cr and Cd were low, indicating that Cr and Cd were easily released from the surface sediments, causing secondary pollution. In terms of the source of the heavy metals, Cu, As, Zn, Cr, and Ni mainly came from industrial and agricultural wastewaters, Pb was from the mining industry and natural sources, and Mn and Cd mainly came from tailings and their landfill leachate. Basommatophora and Araneae can be used as potential indicator organisms for heavy metal pollution in surface water, and Basommatophora, Coleoptera, Hemiptera, and Araneae can be used as indicator species for heavy metal pollution in surface sediments. The study showed that macroinvertebrate community characteristics had a sensitive response to heavy metals in the surface waters and sediments of the Heihe River, which can be used to evaluate the pollution status of heavy metals in inland rivers.