Deep learning (DL) frameworks provide diverse fundamental algorithmic units as operators, which are critical infrastructure for constructing various intelligent software. Due to the fact that open source development paradigm is widely adopted by mainstream frameworks, bugs may repeatedly occur across operators and even frameworks. Although some efforts have been reported that large language models (LLMs) are leveraged to exploit historical issues to generate test cases for cross-framework testing, existing approaches suffer from the following limitations: 1) the generated test cases have low fault detection capability since they mainly rely on inputs or contexts from historical issues without considering the underlying root causes; 2) an effective mechanism is missing to determine the appropriate scope in a targeted framework for applying generated test cases. To overcome these limitations, we propose SpectraDL, a historical issue-driven, test specification-assisted transfer testing approach for DL frameworks. SpectraDL first extracts rigorous test specifications for each operator from official documentation, and then extracts and transforms historical issues and associated pull requests into structured fault representations (i.e., bug patterns). A dual retrieval mechanism based on semantic intent and structural input-space features is designed to transfer these bug patterns for testing related operators across frameworks. We evaluated SpectraDL with four mainstream DL frameworks, and experimental results have shown that SpectraDL successfully detected 125 previously unknown bugs, with 107 confirmed by developers. The results confirm that SpectraDL delivers a promising transfer testing approach for DL frameworks.