Many previous methods on text-based person retrieval tasks are devoted tolearning a latent common space mapping, with the purpose of extractingmodality-invariant features from both visual and textual modality.Nevertheless, due to the complexity of high-dimensional data, the unconstrainedmapping paradigms are not able to properly catch discriminative clues about thecorresponding person while drop the misaligned information. Intuitively, theinformation contained in visual data can be divided into person information(PI) and surroundings information (SI), which are mutually exclusive from eachother. To this end, we propose a novel Deep Surroundings-person SeparationLearning (DSSL) model in this paper to effectively extract and match personinformation, and hence achieve a superior retrieval accuracy. Asurroundings-person separation and fusion mechanism plays the key role torealize an accurate and effective surroundings-person separation under amutually exclusion constraint. In order to adequately utilize multi-modal andmulti-granular information for a higher retrieval accuracy, five diversealignment paradigms are adopted. Extensive experiments are carried out toevaluate the proposed DSSL on CUHK-PEDES, which is currently the onlyaccessible dataset for text-base person retrieval task. DSSL achieves thestate-of-the-art performance on CUHK-PEDES. To properly evaluate our proposedDSSL in the real scenarios, a Real Scenarios Text-based Person Reidentification(RSTPReid) dataset is constructed to benefit future research on text-basedperson retrieval, which will be publicly available.