The exponential growth of social media platforms such as Twitter and Facebook has revolutionized textual communication and textual content publication in human society. However, they have been increasingly exploited to propagate toxic content, such as hate speech, malicious advertisement, and pornography, which can lead to extremely negative impacts (e.g., harmful effects on teen mental health). To address this problem, both researchers and practitioners have been enthusiastically developing and extensively deploying textual content moderation software. However, we find that malicious users can evade moderation by changing only a few words in the toxic content. Moreover, how well modern content moderation software performs against malicious inputs remains underexplored. To this end, we propose MTTM, a Metamorphic Testing framework for Textual content Moderation software. Specifically, we conduct a pilot study on 2,000 text messages collected from real users and summarize eleven metamorphic relations across three perturbation levels: character level, word level, and sentence level. \methodname employs these metamorphic relations on textual toxic contents to generate test cases, which are still toxic yet likely to evade moderation. In our evaluation, we employ \methodname to test three commercial textual content moderation software and two state-of-the-art moderation algorithms against three kinds of toxic content. The results show that MTTM achieves up to 83.9%, 51%, and 82.5% error finding rates (EFR) when testing commercial moderation software provided by Google, Baidu, and Huawei, respectively, and it obtains up to 91.2% EFR when testing the state-of-the-art algorithms from the academy. In addition, we leverage the test cases generated by MTTM to retrain the model we explored, which largely improves model robustness (0% to 5.9% EFR) while obtaining identical accuracy (92.4%) on the original test set.