While public services should be easy to execute, the procedures are complex and cumbersome for citizens. In Germany for example, to apply for a financial support in healthcare usually requires filling lengthy forms such as G100 or G134, which are not designed for the citizen, but for the public authority (like the Deutsche Rentenversicherung) to gather all potentially relevant data for the application. In our contribution, we present the design and development of a conversational agent to support patients and employees of technical healthcare support providers in administrative form-filling based on a case from an orthopedic technology provider. We elaborate on the architectural design considerations necessary for creating such a specialized conversational agent. This includes addressing the integration of domain-specific knowledge with core LLM capabilities and designing effective interaction flows, offering a perspective on building tailored solutions for government services rather than relying solely on general-purpose interfaces. Furthermore, we highlight several critical challenges pertinent to deploying such conversational agents within public sector contexts such in healthcare.