Refactoring is a widely adopted practice for improving code comprehension and for removing severe structural problems in a project. When refactorings affect the system architecture, they are called architectural refactorings. Unfortunately, developers usually do not know when and how they should apply refactorings to remove architectural problems. Nevertheless, they might be more susceptible to applying architectural refactoring if they rely on code smells and code refactoring – two concepts that they usually deal with through their routine programming activities. To investigate if smells can serve as indicators of architectural refactoring opportunities, we conducted a retrospective study over the commit history of 50 software projects. We analyzed 52,667 refactored elements to investigate if they had architectural problems that could have been indicated by automatically-detected smells. We considered purely structural refactorings to identify elements that were likely to have architectural problems. We found that the proportion of refactored elements without smells is much lower than those refactored with smells. By analyzing the latter, we concluded that smells can be used as indicators of architectural refactoring opportunities when the affected source code is deteriorated, i.e., the code hosting two or more smells. For example, when God Class or Complex Class appear together with other smells, they are indicators of architectural refactoring opportunities. In general, smells that often co-occurred with other smells (67.53%) are indicators of architectural refactoring opportunities in most cases (88.53% of refactored elements). Our study also enables us to derive a catalog with patterns of smells that indicate refactoring opportunities to remove specific types of architectural problems. These patterns can guide developers and make them more susceptible to apply architectural refactorings.