As distributed AI workloads scale in complexity, GPU-based collective communication libraries (xCCLs) such as NCCL are becoming increasingly critical for efficient data movement across heterogeneous hardware. Characterizing and optimizing the interactions between hardware architectures, communication algorithm designs, and low-level primitive configurations is essential to fully utilize GPU and interconnect resources while ensuring scalability. However, existing analysis methods often lack depth, limiting optimization insights, leaving significant performance gains untapped, or lack generality. To address these limitations, we propose a novel primitive-based profiling and analysis method that integrates architectural, algorithmic, and primitive-level insights to comprehensively qualify and quantify performance-critical parameters in GPU-based CCLs that we implement and test in our tool CCLInsight. Evaluating three large-scale GPU clusters with diverse GPU architectures featuring up to 64 H100s, 64 RTX 5000s, and 256 A100s, CCLInsight uncovers critical parameter influences on communication efficiency and scalability, as well as exposing a new CCL performance scaling law. Adjusting parameters based on this analysis, we demonstrate a 43.90% improvement in NCCL and a 40.64× speedup in MSCCL over default configurations in the collective communication microbenchmark, while also improving NCCL and MSCCL performance on LLM training workloads (GPT- 3, LLaMA2, DeepSeek-R1) by up to 2.27× and 3.00×, respectively, on a 256-GPU H200 cluster. These contributions establish CCLInsight as a useful tool for xCCL characterization and optimization in distributed AI environments.