While forest degradation rates and extent exceed deforestation in the Brazilian Amazon, less attention is given to the factors controlling its spatial distribution. No quantified correlation exists between changes of forest structure due to anthropogenic disturbances and dynamics of land use and cover change occurring at broader spatial levels. This study examines the influence of multi-scale landscape structure factors (i.e. spatial composition, configuration and dynamic of land use/cover) on primary forest’s aboveground biomass (AGB), spanning from low to highly degraded, in Paragominas municipality (Pará state). We used random forest models to identify the most important landscape predictors of degradation and clustering methods to analyze their distribution and interactions. We found that 58% of the variance of AGB could be explained by metrics reflecting land use practices and agricultural dynamics around primary forest patches and that their spatial patterns were not randomly distributed. Forest degradation is mainly driven by fragmentation effects resulting from old deforestation and colonization events linked with cropland expansion (e.g. soybean and maize) coupled with high accessibility to market. To a lesser extent, degradation is driven by recent and ongoing (1985–2015) deforestation and fragmentation in slash-and-burn agricultural areas, characterized by heterogeneous mosaics of pastures and fallow lands combined with high use of fire. Our findings highlight the potential of landscape-level framework and remotely sensed land cover data for a thorough understanding of the distribution of forest degradation across human-modified landscapes. Addressing these spatial determinants by looking at agricultural dynamics beyond forest cover is necessary to improve forest management which has major implications for biodiversity, carbon and other ecosystem services.