Testing deep learning (DL) software is crucial and challenging. Recent approaches use differential testing to cross-check pairs of implementations of the same functionality across different libraries. While useful, such approaches require two independent DL libraries implementing the same function, which is often unavailable. In addition, they rely on a high-level library, Keras, that implements missing functions in all supported DL libraries which is prohibitively expensive and thus no longer maintained.

To address this issue, we propose EAGLE, a new technique that uses differential testing at a different dimension, by using equivalent graphs to test a single DL implementation(e.g., a single DL library). Equivalent graphs use different APIs, data types, or optimizations to achieve the same functionality. The rationale is that two equivalent graphs executed on a single DL implementation should produce identical output given the same input. Specifically, we develop 17 new DL equivalence rules, and propose a technique, EAGLE, that (1) uses these equivalence rules to build concrete pairs of equivalent graphs and (2) cross-checks the output of these equivalent graphs to detect inconsistency bugs in a DL library.

Our evaluation on two widely-used DL libraries (TensorFlow and PyTorch) shows that EAGLE detects 20 bugs (12 in TensorFlow and 8 in PyTorch), including 9 previously unknown bugs.