The generation of class integration test orders (CITOs) is a pivotal element in integration testing, which focuses on determining the optimal order for integrating classes while testing an object-oriented system. Due to a high number of dependencies and their possible error proneness, some classes are more critical than others in a program. Existing methods for handling these classes only assess risk in terms of their dependencies; they do not consider historical bug information as an additional indicator and mainly work on small programs. To overcome these limitations, this paper introduces Criticality-Driven CITO (CD-CITO) generation, an innovative approach to optimize CITOs by focusing on class criticality. CD-CITO assesses both the importance of a class in terms of its dependencies and the likelihood of defects, based on historical bug data, to determine a criticality score. Then, it reformulates the CITO generation problem as a Reinforcement Learning (RL) task and uses the Advantage Actor-Critic (A2C) algorithm to address it. We propose a novel reward calculation strategy to guide the learning agent, balancing stubbing costs with the criticality values of classes to optimize the test order. To extract fault proneness information and access the approach, the paper uses Defects4J, a data set that contains real bugs and patches of Java programs. The results obtained show that CD-CITO effectively identifies and prioritizes highly critical classes and also minimizes stubbing costs while generating CITOs, which makes it a valuable tool for integration testing.