Large language models (LLMs) can perform a variety of tasks given an input prompt that contains a description of the task. In an attempt to enhance the performance and capabilities of LLMs, recent research has focused on augmenting LLMs with external tools, such as Python functions, REST APIs, and other deep learning models. While much of the research on tool-augmented LLMs (TaLLMs) has been focused on improving their capabilities, research on understanding and characterizing the kinds of failures that can occur in these systems is lacking. To address this gap, this paper proposes a taxonomy of failures in TaLLMs and their root causes, details an analysis of the failures that occur in two published TaLLMs (Gorilla and Chameleon), and provides recommendations on fault localization and repair of TaLLMs.