Binary code similarity detection(BCSD) has numerous applications in software security, including malware family detection, vulnerability detection and code plagiarism detection. Recently, deep learning-based BCSD approaches have demonstrated promising performance. However, existing approaches suffer from two important issues which hinder the improvement of detection accuracy. On the one hand, most BCSD approaches using sequence networks (like RNN, Transformer) regard instructions as words, which results in large vocabulary size and severe OOV problem. On the other hand, the nature of graph convolutional network only allows for considering local neighborhood structures, and discards long-distance dependencies between basic blocks.

To address these issues, this paper proposes Syntax Tree-based instruction embedding and introduces the acyclic graph neural network. The former decomposes assembly instructions into fine-grained tokens and utilizes a tree-structured neural network to generate vector representations for instructions. The latter transforms CFGs into directed acyclic graphs based on their reducibility, and further captures the dependency between basic blocks with a directed acyclic graph neural network. Based on these two techniques, we implemented a prototype named RCFG2Vec and conducted comprehensive evaluation on a public dataset. The experiment results demonstrated that RCFG2Vec outperforms almost all baselines and achieves competitive detection performance comparable with a large model-based approach. Meanwhile, when integrated with our proposed techniques, several baseline approaches shown significant improvement in detection performance.