With the increasing popularity of embedded and mobile devices, ARM is becoming the dominant architecture in them. Accordingly, there is a pressing need to perform security assessments to these devices. Due to the fragmentation, it is an ongoing research question to dynamically run the systems of these devices (or the firmware) in an emulated environment. Mainly due to this, the static analysis approach is still a commonly used way. In particular, existing works usually leverage off-the-shelf disassembly tools to disassemble stripped (ARM) binaries, and assume that reliably disassembling them and identifying functions are solved problems. However, whether this assumption holds for real world ARM binaries is unknown.

In this paper, we conduct a comprehensive study on ARM disassembly tools. Specifically, we build 1,896 ARM binaries (including 248 obfuscated ones) with different compilers, compiling options, and obfuscation methods. Using these binaries, we then evaluate eight state-of-the-art ARM disassembly tools (including both commercial and noncommercial ones) on their capabilities to locate instruction and function boundaries. These two primitives are fundamental ones and could be leveraged to build other primitives. Based on our evaluation, we present observations that were not systematically summarized and/or confirmed previously. For instance, we find that the existence of both the ARM and the Thumb instruction sets, and the reuse of the BL instruction for both direct function call and direct branch bring serious challenges to disassembly tools. Our evaluation sheds light on the limitations of the state-of-the-art disassembly tools and points out potential directions to improve them. To engage the community, we will publicly release the compiled ARM binaries, the retrieved ground truth, and the result.