The key challenge of image manipulation detection is how to learngeneralizable features that are sensitive to manipulations in novel data,whilst specific to prevent false alarms on authentic images. Current researchemphasizes the sensitivity, with the specificity overlooked. In this paper weaddress both aspects by multi-view feature learning and multi-scalesupervision. By exploiting noise distribution and boundary artifact surroundingtampered regions, the former aims to learn semantic-agnostic and thus moregeneralizable features. The latter allows us to learn from authentic imageswhich are nontrivial to be taken into account by current semantic segmentationnetwork based methods. Our thoughts are realized by a new network which we termMVSS-Net. Extensive experiments on five benchmark sets justify the viability ofMVSS-Net for both pixel-level and image-level manipulation detection.