High-quality code summaries help developers understand code, but they are often missing or require significant effort to write. Automatic code summarization can generate summaries that resemble human-written ones, with Large Language Models (LLMs) achieving state-of-the-art performance. Yet, they often fail and produce summaries unlike what a human would say. Can we gauge the likelihood that a LLM-generated summary is sufficiently similar to what a human would write? We examine whether the token-level confidences of several LLMs indicate this notion of summary quality. We suggest a method to produce well-calibrated likelihoods that a LLM-generated summary is similar to a human-written one.