Although artificial intelligence (AI) is increasingly involved in everyday technologies, AI literacy amongst the general public remains low. Thus many AI education curricula for people without prior AI experience have emerged, often utilizing graphical programming languages for hands-on projects. However, there are no tools that assist educators in evaluating learners’ AI projects or provide learners with contemporaneous feedback on their work. We developed LevelUp, an automatic code analysis tool to support these educators and learners. LevelUp is built into a block-based programming platform and gives users continuous feedback on their text classification projects. We evaluated the tool with a crossover user study where participants completed two text classification projects, once where they could access LevelUp and once when they could not. To measure the tool’s impact on participants’ understanding of text classification, we used pre-post assessments and graded both of their projects against LevelUp’s rubric. We saw a significant improvement in the quality of participants’ projects after they used the tool. We also used questionnaires to solicit participants’ feedback. Overall, participants said that LevelUp was useful and intuitive. Our investigation of this novel automatic assessment tool can inform the design of future code analysis tools for AI education.