The neural network model is having a significant impact on many real-world applications. However, the ever increasing popularity and complexity of these models amplifies their security and privacy challenges, with privacy leakage from training data being one of the most prominent issues. In this context, prior studies proposed to analyze the abstraction behavior of neural network models, e.g., \emph{RNN}, to understand their robustness. However, the existing research rarely addresses privacy breach memorization in neural language models. To fill this gap, we propose a novel approach, \emph{DeepMemory}, that analyzes memorization behavior for a neural language model. We first construct a memorization-analysis oriented model, taking both training data and a neural language model as input. We then build a semantic first-order Markov model to bind the constructed semantic memorization-analysis oriented model to the training data to analyze memorization distribution. Finally, we apply our approach to address data leakage issues associated with memorization and to assist with dememorization. We evaluate our approach on one of the most popular neural language models, the \emph{LSTM}-based language model, with three public datasets, namely, WikiText-103, WMT2017, and IWSLT2016. We find that sentences in the studied datasets with low perplexity are more likely to be memorized. Our approach achieves an average AUC of 0.73 in automatically identifying data leakage issues during assessment. Finally, with the assistance from our approach, the memorization risk from the neural language model can be mitigated by mutating training data without impacting the quality of neural language models.