Shadow detection is a fundamental and challenging task, since it requires anunderstanding of global image semantics and there are various backgroundsaround shadows. This paper presents a novel network for shadow detection byanalyzing image context in a direction-aware manner. To achieve this, we firstformulate the direction-aware attention mechanism in a spatial recurrent neuralnetwork (RNN) by introducing attention weights when aggregating spatial contextfeatures in the RNN. By learning these weights through training, we can recoverdirection-aware spatial context (DSC) for detecting shadows. This design isdeveloped into the DSC module and embedded in a CNN to learn DSC features atdifferent levels. Moreover, a weighted cross entropy loss is designed to makethe training more effective. We employ two common shadow detection benchmarkdatasets and perform various experiments to evaluate our network. Experimentalresults show that our network outperforms state-of-the-art methods and achieves97% accuracy and 38% reduction on balance error rate.