[Context] User Stories (USs) are a popular notation for writing requirements in Agile software development. USs are often stored in Issue Tracking Systems (ITSs) and are a starting point for defining software development tasks. [Problem] While writing high-quality requirements statements is a typical concern when authoring requirements specification documents, this is less the case when writing USs in an ITS. This may also be the attributed to the fact that practitioners are not familiar with techniques for improving the quality of their USs. [Method] As part of previous research in a large organization, we found that practitioners were eager to learn how to write better USs and asked four Agile teams to participate in a study aimed at improving that practice. We conducted canonical action research where these teams were offered a lightweight intervention in the form of guidelines for writing USs—based on the Quality User Story (QUS) framework—, which they could use to reflect upon the quality of their USs. [Findings] The share of atomic and minimal violations decreased through the use of the intervention and, for the former, the positive effects lasted even after the intervention period ended. However, practitioners did not agree with all the guidelines and argued that violating the criteria can sometimes benefit them in terms of clarity and time spent. These results call for better contextualization of research on user story quality, which we initiate by proposing revised formulations of our guidelines.