The compiler bug duplication problem (where many test failures are caused by the same compiler bug) can lead to huge waste of time and resource in diagnosing test failures produced by compiler testing. It is particularly challenging with regard to the silent compiler bugs that do not produce any error messages. To address this problem, multiple white-box techniques were proposed, but they are inapplicable in many practical scenarios. Black-box techniques are more practical, but the existing ones are less effective as they often rely on irrelevant syntactic information. To bridge this gap, we propose a novel black-box technique (BLADE), which aims to improve the effectiveness of black-box de-duplication by extracting failure-relevant semantic information from failure-triggering test programs in a black-box manner. It first learns failure-relevant semantic information based on intermediate representation learning by employing the classification of failure-triggering and failure-free test programs as the auxiliary objective, and then extracts such information based on model interpretation. Our experiments on four widely-used datasets (collected from GCC and LLVM) show that BLADE significantly outperforms the two existing black-box techniques with an average improvement of 36% and 12% in identifying unique silent compiler bugs when analyzing the same number of test failures respectively, and achieves competitive effectiveness with the state-of-the-art white-box techniques.