Classifying failures for automated diagnosis is critical for maintaining the reliability of ever-increasing microservice systems. Existing failure classification methods focus solely on analyzing one specific data modality (e.g., metrics) or feeding multi-modal data (e.g., logs and traces) in a one-fit-all model. This would lead to biased inference as different modalities have differentiated proportions and manifest distinct sensitivities and patterns in failures of varied runtime contexts. This work proposes a novel failure classification framework for microservices, named AgileMFC, with the basic idea of flexibly learning and assembling the expertise of multi-modal monitoring data. The flexibility is attained with the split-and-merge design of data representation, where it assigns each modality a different gate that routes it to dedicated expert networks and uses a sharing gate to facilitate controllable knowledge interaction among modalities. Meanwhile, AgileMFC trains the representation networks and a classifier in consecutive stages with concatenated and weighted feature fusion, thus decoupling task-agnostic feature extraction with task-specific classification. Experiments on three datasets show that AgileMFC outperforms four baselines w.r.t., precision, recall, and F1-score on failure classification.