We propose a novel data-augmentation technique for neural machine translation based on ROT-$k$k ciphertexts. ROT-$k$k is a simple letter substitution cipher that replaces a letter in the plaintext with the $k$kth letter after it in the alphabet. We first generate multiple ROT-$k$k ciphertexts using different values of $k$k for the plaintext which is the source side of the parallel data. We then leverage this enciphered training data along with the original parallel data via multi-source training to improve neural machine translation. Our method, CipherDAug, uses a co-regularization-inspired training procedure, requires no external data sources other than the original training data, and uses a standard Transformer to outperform strong data augmentation techniques on several datasets by a significant margin. This technique combines easily with existing approaches to data augmentation, and yields particularly strong results in low-resource settings.