We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, verifying the correctness of such applications is extremely challenging and would be very expensive if we follow today’s testing methodologies. In this work, we present an articulation of the challenges in verifying ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application using Support Vector Machine and an application using Deep Learning. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.