Recent deep learning methods have achieved promising results in image shadowremoval. However, their restored images still suffer from unsatisfactoryboundary artifacts, due to the lack of degradation prior embedding and thedeficiency in modeling capacity. Our work addresses these issues by proposing aunified diffusion framework that integrates both the image and degradationpriors for highly effective shadow removal. In detail, we first propose ashadow degradation model, which inspires us to build a novel unrollingdiffusion model, dubbed ShandowDiffusion. It remarkably improves the model'scapacity in shadow removal via progressively refining the desired output withboth degradation prior and diffusive generative prior, which by nature canserve as a new strong baseline for image restoration. Furthermore,ShadowDiffusion progressively refines the estimated shadow mask as an auxiliarytask of the diffusion generator, which leads to more accurate and robustshadow-free image generation. We conduct extensive experiments on three popularpublic datasets, including ISTD, ISTD+, and SRD, to validate our method'seffectiveness. Compared to the state-of-the-art methods, our model achieves asignificant improvement in terms of PSNR, increasing from 31.69dB to 34.73dBover SRD dataset.